←BaldanAll skillsLet's talk
Machine Learning lab

Classification

Predicting machine failures when only 3 in 100 runs fail, with rules you can read.

0.891

F1 (failures)

84%

Recall

95%

Precision

3.4%

Failure rate

Overview

In a factory, a missed failure means downtime and scrap, while a false alarm means an unnecessary stop. With only about 3% of runs failing, accuracy is meaningless: a model that always says “fine” is 97% accurate.

Instead of reaching for a bigger model, this project adds three features straight from physics: power, the temperature gap and tool wear × torque. A small decision tree with those features matches gradient boosting and can be drawn on one screen.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    10,000 runs

    14 columns, 339 failures

  2. 02

    Leak check

    −5 flags

    Failure-mode flags removed: they reveal the answer

  3. 03

    Physics

    +3 features

    Power (W), temperature gap (K), wear × torque

  4. 04

    Split

    8,000 / 2,000

    Stratified, 68 failures in the test set

  5. 05

    Compare

    3 models × 2

    Logistic, tree, boosting · raw vs physics

  6. 06

    Explain

    10 leaves

    Depth-5 tree, every decision readable

Inside the model

One machine, one tree.

A decision tree is a flowchart learned from data: 10 leaves, 5 questions at most. Scroll to drop one real machine reading through it.

machine

M19202

Type M

Speed 1,284 rpm

Air 301.7 K

Torque 68.2 Nm

Process 309.8 K

Tool wear 111 min

Temp. diff 8.1 K

Power 9,170 W

Wear × torque 7,570

The data

10,000 machine runs, 339 failures.

Only 3.4% of runs fail. A model that always says “fine” would be 96.6% accurate and useless.

AI4I 2020 Predictive Maintenance (UCI)

A synthetic dataset that mirrors real predictive-maintenance data, published by Stephan Matzka (HTW Berlin) in 2020. Its documentation explains how each failure mode was generated, which lets us check the model against the physics.

10,000

runs

3.4%

failure rate

5

failure modes

3

physics features

Dataset on UCI

Column dictionary

● used by the model

ColumnTypeWhat it means
UDIidRow number
Product IDidSerial number with the quality variant letter
TypecategoryProduct quality: L (60%), M (30%), H (10%)
Air temperature [K]numberAmbient temperature
Process temperature [K]numberTemperature of the process
Rotational speed [rpm]numberSpindle speed
Torque [Nm]numberLoad on the tool
Tool wear [min]numberMinutes the tool has been used
Machine failuretargetRun ended in a failure: the thing we predict
TWF · HDF · PWF · OSF · RNFflagsWhich failure mode happened. Left out: they reveal the answer

Explore the data

Failures follow physics.

The documentation says how each failure mode happens. The data agrees almost perfectly.

Speed × torque, coloured by failure mode

Power failures sit outside the two curves (3,500 W and 9,000 W). Normal runs (grey) live between them.

PowerHeat dissipationOverstrainTool wearRandomFailure, no mode flagNo failure
0204060801,2001,5001,8002,1002,4002,700Rotational speed (rpm)Torque (Nm)3,500 W9,000 W
Documented rule vs the data

How many failures of each mode satisfy the rule in the dataset's documentation.

ModeDocumented ruleFailuresMatch the rule
HDF · Heat dissipationtemp difference < 8.6 K and speed < 1380 rpm115115/115 (100%)
PWF · Powerpower < 3500 W or > 9000 W9595/95 (100%)
OSF · Overstraintool wear × torque > 11,000 (L) / 12,000 (M) / 13,000 (H) min·Nm9898/98 (100%)
TWF · Tool weartool replaced or fails at a random wear time between 200 and 240 min4643/46 (93%)
RNF · Randomrandom 0.1% failure, no pattern19no pattern (random)
Failures by mode

Heat dissipation is the most common (115). A run can have more than one mode.

0100200115Heat dissipat…OverstrainPowerTool wear19Random
Failure rate by product quality

L 3.9% · M 2.8% · H 2.1%. Lower quality fails a bit more, partly through overstrain.

0.0%2.5%5.0%3.9%Type LType M2.1%Type H
Torque: OK vs failure

At 45 Nm or more: 78% of failures vs 30% of normal runs, plus a small cluster below 15 Nm.

OKFailure
0%25%50%01530456075
Tool wear: OK vs failure

Failures climb once a tool passes about 200 minutes.

OKFailure
0%25%50%060120180240

Try it

Load a run, push the machine.

Runs from the test set, one per failure mode where possible. Move a slider and watch the tree take a different path.

Real rows from the dataset

click a row to try it in the demo ↓

RunTypeAir KProcess KrpmNmWear minModel saysOutcome
M19202M301.7309.81,28468.2111100%failure · HDF
M23105M299310.31,30368.6111100%failure · PWF
L56844L299.1310.21,31754.8231100%failure · OSF
M16856M298.43081,41638.21980%failure · TWF
L51260L302310.41,3575810889%failure · HDF
L55686L298.4309.61,71027.31630%failure · ?
M18084M300.6309.51,55733.8840%OK
M20387M302.5311.91,53238.8960%OK
M16721M297.9307.61,60930.3640%OK
L55052L300.6312.31,53237.2540%OK
L54040L301.1311.21,91020.2410%OK

Live demo

runs in your browser

Machine sensors

Product quality (L = low, M = medium, H = high)

Physics features (computed live)

Temperature difference10 K

heat risk below 8.6 K at < 1380 rpm

Power6,311 W

safe band 3,500–9,000 W

Tool wear × torque4,331 min·Nm

overstrain above 11,000 for type L

Tool wear108 min

tools fail between 200 and 240 min

Chance this run ends in a failure

0%Low risk
0%30%60%100%

Share of failures among the 6,809 training runs that landed in the same leaf.

How the tree decided

  1. Tool wear × torque 4,331 min·Nm ≤ 10,999 min·Nm
  2. Power 6,311 W ≤ 9,004 W
  3. Power 6,311 W > 3,496 W
  4. Temperature difference 10 K > 8.7 K
  5. Tool wear 108 min ≤ 202.5 min
  6. Likely OK · 0.1% of similar runs failed

How well it works

Three features did more than a bigger model.

On 2,000 unseen runs (68 failures). A depth-5 tree goes from F1 0.43 to 0.89 once it gets the physics features, level with boosting (0.89).

F1 score: raw sensors vs physics features

Same models, same split. Only the features change.

Raw sensors+ physics features
0.00.51.00.180.27Logistic Regression0.430.89Decision tree0.760.89Gradient Boosting
Precision–recall curves

Higher and further right is better. Random guessing sits at 3.4% precision.

Tree · physicsBoosting · physicsTree · raw
0.00.51.00.000.250.500.751.00RecallPrecision
Confusion matrix (tree, 50% threshold)

Catches 57 of 68 failures with 3 false alarms.

Predicted okPredicted failureActually ok
Correct1,929
False alarm3
Actually failure
Missed11
Caught57
What the tree uses

The three engineered features carry 66% of the importance.

0%25%50%31%strainpowerspeedtemp difftype L2%torque
The whole model, drawn: depth 5, 10 leaves

The first split, wear × torque ≤ 10,999, rediscovers the documented 11,000 overstrain limit on its own. Leaves are coloured by failure share; hover any dot.

wear×torque ≤ 10,999power ≤ 9,004 Wtype L? (right = yes)

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

ai4i() from ml/train.py + 4 helpers (r, save, hist_by_group, rate_by)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load the machine data

    Reads the AI4I 2020 file, gives the columns short names (air, process, speed, torque, wear) and uses the machine-failure flag as the target.

    ml/train.py:640 · 8 lines

    from sklearn.metrics import average_precision_score, precision_recall_curve
    from sklearn.tree import DecisionTreeClassifier
    
    print("manufacturing-defects")
    raw = pd.read_csv(RAW / "ai4i" / "ai4i2020.csv", encoding="utf-8-sig")
    df = raw.copy()
    df.columns = ["udi", "product_id", "type", "air", "process", "speed", "torque", "wear", "fail", "TWF", "HDF", "PWF", "OSF", "RNF"]
    y = df["fail"].to_numpy()
  2. 2

    Add physics features, check rules

    Adds temperature difference, power and tool strain, one-hot codes the product type, then counts how often each failure mode matches its documented rule.

    ml/train.py:650 · 22 lines

    # physics features
    df["temp_diff"] = df["process"] - df["air"]  # K
    df["power"] = df["torque"] * df["speed"] * 2 * np.pi / 60  # W
    df["strain"] = df["wear"] * df["torque"]  # min·Nm
    for t in ["L", "M", "H"]:
        df[f"type_{t}"] = (df["type"] == t).astype(int)
    
    # check the documented failure-mode rules against the data itself
    osf_limit = df["type"].map({"L": 11000, "M": 12000, "H": 13000})
    rule_checks = {
        "HDF": {"rule": "temp difference < 8.6 K and speed < 1380 rpm", "rows": int(df.HDF.sum()),
                "match": int(((df.temp_diff < 8.6) & (df.speed < 1380))[df.HDF == 1].sum())},
        "PWF": {"rule": "power < 3500 W or > 9000 W", "rows": int(df.PWF.sum()),
                "match": int(((df.power < 3500) | (df.power > 9000))[df.PWF == 1].sum())},
        "OSF": {"rule": "tool wear × torque > 11,000 (L) / 12,000 (M) / 13,000 (H) min·Nm", "rows": int(df.OSF.sum()),
                "match": int((df.strain > osf_limit)[df.OSF == 1].sum())},
  3. 3

    Hold out a test set

    Two feature lists (raw readings, and raw plus physics) and a 20% test split, stratified so both parts keep the same failure rate.

    ml/train.py:674 · 4 lines

    rawf = ["air", "process", "speed", "torque", "wear", "type_L", "type_M", "type_H"]
    engf = rawf + ["temp_diff", "power", "strain"]
    idx_tr, idx_te = train_test_split(np.arange(len(df)), test_size=0.2, stratify=y, random_state=SEED)
    ytr, yte = y[idx_tr], y[idx_te]
  4. 4

    Train three models twice

    Logistic regression, a depth-5 tree and gradient boosting each run on raw and on physics features, scored on the test set by average precision, F1 and recall.

    ml/train.py:680 · 33 lines

    def fit_eval(name, model, feats):
        model.fit(df.loc[idx_tr, feats], ytr)
        proba = model.predict_proba(df.loc[idx_te, feats])[:, 1]
        pred = (proba >= 0.5).astype(int)
        res = {
            "model": name,
            "features": "physics" if feats is engf else "raw",
            "ap": r(average_precision_score(yte, proba)),
            "f1": r(f1_score(yte, pred)),
            "recall": r(recall_score(yte, pred)),
            "precision": r(precision_score(yte, pred, zero_division=0)),
        }
        print(f"  {name:28s} {res['features']:8s} AP={res['ap']:.3f} F1={res['f1']:.3f} R={res['recall']:.3f} P={res['precision']:.3f}")
        return res, proba, model
    
    lr = lambda: Pipeline([("sc", StandardScaler()), ("m", LogisticRegression(max_iter=3000))])
  5. 5

    Keep the tree, flatten it

    The depth-5 tree on physics features is kept and turned into plain nodes the browser can walk, with feature importances and recall per failure mode.

    ml/train.py:715 · 27 lines

    # export the tree as plain nodes for the browser
    t = tree.tree_
    nodes = []
    for i in range(t.node_count):
        counts = t.value[i][0]
        nodes.append(
            {
                "f": engf[t.feature[i]] if t.children_left[i] != -1 else None,
                "t": r(t.threshold[i], 3) if t.children_left[i] != -1 else None,
                "l": int(t.children_left[i]),
                "r": int(t.children_right[i]),
                "p": r(counts[1] / counts.sum(), 4),
                "n": int(t.n_node_samples[i]),
            }
        )
    importance = sorted(
  6. 6

    Pick demo test runs

    One test-set failure per mode where possible, up to 2 more failures and 5 normal runs, capped at 12, each with the tree's predicted risk.

    ml/train.py:744 · 28 lines

    # samples from the TEST set: one of each failure mode where possible + normal runs
    te = df.loc[idx_te].copy()
    te["proba"] = tree_proba
    picks = []
    for mode in ["HDF", "PWF", "OSF", "TWF", "RNF"]:
        sub = te[(te[mode] == 1) & (te.fail == 1)]
        if len(sub):
            picks.append(sub.sample(1, random_state=SEED))
    picks.append(te[te.fail == 1].drop(pd.concat(picks).index, errors="ignore").sample(min(2, int(te.fail.sum())), random_state=SEED))
    picks.append(te[te.fail == 0].sample(5, random_state=SEED))
    samp = pd.concat(picks).drop_duplicates(subset="udi").head(12)
    samples = []
    for _, row in samp.iterrows():
        mode = next((m for m in ["HDF", "PWF", "OSF", "TWF", "RNF"] if row[m] == 1), None)
        samples.append(
            {
  7. 7

    Export everything the page needs

    A speed-torque scatter (all failures plus 1200 normal runs), rule checks, the tree, EDA and PR curves, written to src/data/models/manufacturing-defects.json.

    ml/train.py:774 · 48 lines

    # scatter: all failures + a sample of normal runs
    sc = pd.concat([df[df.fail == 1], df[df.fail == 0].sample(1200, random_state=SEED)])
    scatter = [
        [int(r_.speed), r(r_.torque, 1), next((m for m in ["HDF", "PWF", "OSF", "TWF", "RNF"] if r_[m] == 1), "OK" if r_.fail == 0 else "Other")]
        for _, r_ in sc.iterrows()
    ]
    ys = pd.Series(y)
    save(
        "manufacturing-defects",
        {
            "dataset": {
                "name": "AI4I 2020 Predictive Maintenance (UCI)",
                "source": "https://archive.ics.uci.edu/dataset/601/ai4i+2020+predictive+maintenance+dataset",
                "rows": int(len(df)),
                "raw_cols": int(raw.shape[1]),
                "failure_rate": r(y.mean(), 4),

In your browser · TypeScript

  1. 8

    Walk the tree in the browser

    The exported decision tree is a list of nodes; the demo follows the left/right branches for the machine you set up and reports the path.

    src/components/demos/machine-demo.tsx:27 · 14 lines

    /** Walk the exported scikit-learn tree: left when x[f] <= threshold. */
    function walk(x: Record<string, number>) {
      const nodes = model.tree.nodes as Node[];
      const path: { f: string; t: number; v: number; left: boolean }[] = [];
      let i = 0;
      while (nodes[i].l !== -1) {
        const n = nodes[i];
        const v = x[n.f as string];
        const left = v <= (n.t as number);
        path.push({ f: n.f as string, t: n.t as number, v, left });
        i = left ? n.l : n.r;
      }
      return { p: nodes[i].p, n: nodes[i].n, path };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Removed the five failure-mode flags first: they are the answer in disguise.

  2. 2

    Checked the dataset's documented failure rules against the data before modelling.

  3. 3

    Added three physics features: power from torque and speed, the process-air temperature gap, and tool wear × torque.

  4. 4

    Compared three models with and without them. The depth-5 tree jumped from F1 0.43 to 0.89, level with boosting (0.89), so the readable tree was kept.

F1 on the test set (failures)

Logistic Regression · raw0.177
Decision tree (depth 5) · raw0.425
Gradient Boosting · raw0.756
Logistic Regression · physics0.270
Decision tree (depth 5) · physics0.891chosen
Gradient Boosting · physics0.885

Honest note

The tree catches every heat (29/29), power (13/13) and overstrain (16/16) failure in the test set, but only 1 of 10 tool-wear failures: those happen at a random point between 200 and 240 minutes by design. The data is synthetic, so real machines will be noisier.

Reproduce: ml/train.py → ai4i()

dataset source

Next project

Student Performance