←BaldanAll skillsLet's talk
Machine Learning lab

Classification

Scoring auto claims for review, and refusing the signal that looks too good to be true.

0.770

ROC-AUC (claim facts)

48%

Fraud in top 20%

0.766

Severity alone

1,000

Claims

Overview

An insurer can't investigate every claim, so it needs a score that ranks claims for review and reasons an investigator can check. With a quarter of these claims labelled fraud, the risk is less a weak model and more a model that learns the wrong thing.

The rebuild uses only facts about the claim and the incident, gives investigators a three-question checklist from a small tree, and shows openly what happens with a field that should never be used: the policy holder's hobby.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    1,000 claims

    39 columns, 25% fraud

  2. 02

    Clean

    ? → Unknown

    Missing answers kept as their own category

  3. 03

    Fairness

    −24 columns

    Profile, IDs, dates, places and hobbies out

  4. 04

    Features

    14 claim facts

    Severity, incident, authorities, amounts, witnesses

  5. 05

    Validate

    5-fold OOF

    Each claim scored by a model that never saw it

  6. 06

    Explain

    3 questions

    Depth-3 tree as an investigator checklist

Inside the model

One claim, one neuron.

Logistic regression is a single artificial neuron. Scroll to watch it score one real insurance claim with the model's real weights.

claim
incident severityTotal Lossincident typeMulti-vehicle Col…property damageNOcollision typeRear CollisionInjury claim ($)13,100witnesses38 more inputssmall pushesBIAS (STARTING POINT)−1.52ΣNEURON0%50%100%9.8%chance of fraudSIGMOID: SUM → 0–100%SUMWEIGHTS GROWING FROM 0ΣΣΣΣΣΣLAYERS OF NEURONS = A NEURAL NETWORK

Inside the model

Then a three-question checklist.

Next to the score, a small decision tree gives investigators a checklist they can follow by hand and explain to a customer.

claim

CLM-315041

Severity Total Loss

Collision Rear Collision

Authorities Ambulance

Vehicle claim 45,850

Property claim 19,650

Witnesses 3

The data

1,000 auto claims, 25% flagged as fraud.

39 columns about the policy, the person, the incident and the money. Most of them shouldn't go near a fraud model.

Auto insurance claims (insurance_claims.csv)

Auto-insurance claims from three US states with a fraud label, published on Mendeley Data under CC BY 4.0 and widely mirrored on GitHub.

1,000

claims

24.7%

fraud rate

14

features used

24

columns left out

Dataset on Mendeley Data

Column dictionary

● used by the model

ColumnTypeWhat it means
months_as_customernumberHow long the customer has held a policy
age · insured_sex · insured_zipprofileLeft out (fairness)
insured_education_level · insured_occupation · insured_relationshipprofileLeft out (fairness)
insured_hobbiesprofileLeft out: a suspiciously strong artifact, tested separately below
policy_number · policy_bind_date · policy_statepolicyIdentifiers and dates · not used
policy_csl · policy_deductable · policy_annual_premiumpolicyCoverage terms · not used in the final model
umbrella_limitnumberExtra liability cover (one row has an impossible −1,000,000)
incident_type · collision_typecategoryWhat happened ('?' kept as Unknown)
incident_severitycategoryTrivial, minor, major damage or total loss
authorities_contactedcategoryPolice, fire, ambulance, other or none
incident_state · city · location · hourwhere/whenNot used
number_of_vehicles_involved · bodily_injuries · witnessesnumberIncident facts
property_damage · police_report_availableyes/no/?'?' kept as Unknown
injury_claim · property_claim · vehicle_claimnumber ($)Claim amounts (total_claim_amount is their sum, so not used)
auto_make · auto_model · auto_yearvehicleNot used
fraud_reportedtargetY / N: the thing we predict

Explore the data

One field shouts. One field lies.

Fraud rate by incident severity

“Major damage” claims are 61% fraud. Every other level sits between 7% and 13%.

0%50%100%61%MajorMinorTotal Loss7%Trivial
Fraud rate by insured's hobby (not used)

Chess 83%, cross-fit 74%, everything else under 31%. Too strong to be real: almost certainly an artifact of how the data was made.

0%50%100%83%chesscross-fityachtingboard-gamespoloreadingbase-jumping23%hiking
Fraud rate by authorities contacted

When nobody was called, only 7% are fraud.

0%25%50%AmbulanceFire7%None32%OtherPolice
Police report available?

Barely matters: 25%, 26%, 23% across no / unknown / yes.

0%25%50%No26%Unknown23%Yes
Total claim amount: legit vs fraud

Below $10k: 21% of legit claims vs 6% of fraud. Fraud leans to 50–80k.

LegitFraud
0%25%50%$0k$20k$40k$60k$80k$100k

Try it

Load a claim, then argue with the model.

Five fraud and five legitimate claims. Scores in the table come from models that never saw the claim.

Real rows from the dataset

click a row to try it in the demo ↓

ClaimSeverityTypeAuthoritiesVehicle $ScoreOutcome
CLM-283414Major DamageMulti-vehicle CollisionAmbulance51,94042%Legit
CLM-285496Major DamageMulti-vehicle CollisionAmbulance47,68038%Fraud
CLM-354455Minor DamageSingle Vehicle CollisionAmbulance35,14014%Legit
CLM-648397Total LossMulti-vehicle CollisionOther33,39018%Fraud
CLM-555374Minor DamageSingle Vehicle CollisionPolice38,22016%Legit
CLM-315041Total LossMulti-vehicle CollisionAmbulance45,8506%Fraud
CLM-655787Minor DamageMulti-vehicle CollisionPolice53,9709%Legit
CLM-110084Total LossSingle Vehicle CollisionOther35,00010%Fraud
CLM-931625Total LossMulti-vehicle CollisionFire47,28011%Fraud
CLM-979963Total LossSingle Vehicle CollisionFire52,7809%Legit

Score is out-of-fold: it comes from a model that never saw this claim. The demo runs the final model trained on every claim, so its number can differ by a few points.

Live demo

runs in your browser

The claim

Incident severity

Property damage

Police report

Fraud score

14%Low risk
0%30%60%100%

Investigator checklist (3 questions)

  1. Incident severity is not “Major Damage”
  2. Vehicle claim ≤ 60,870
  3. Collision type is “Rear Collision”
  4. Fast-track · 16% of 180 similar claims were fraud

Why

lowers risk raises risk
  • Incident severityMinor Damage
  • Collision typeRear Collision
  • Property damageUnknown
  • Umbrella limit0
  • Authorities contactedPolice

How well it works

Honest signal: AUC 0.77. Tempting signal: 0.87.

Severity alone scores 0.766; all the claim facts together, 0.770. Adding hobbies jumps to 0.870, and that's exactly why it stays out.

ROC curves

The hobby curve looks better. It wouldn't hold up in front of a customer, a regulator or new data.

Claim facts (0.77)Severity only (0.77)+ hobbies (0.87)
0.00.51.00.000.250.500.751.00False positive rateTrue positive rate
Investigation budget

Investigating the top 20% of claims by score catches 48% of all fraud.

0%50%100%0%25%50%75%100%Share of claims investigated, highest score first20%
Confusion matrix at a 50% score

134 of 247 fraud claims flagged, 90 legitimate claims sent for review.

Predicted legitPredicted fraudActually legit
Correct663
False alarm90
Actually fraud
Missed113
Caught134
The checklist, written out

Every path through the depth-3 tree, riskiest first. This is what an investigator would actually read.

  • Incident severity is “Major Damage” · Property claim > 18,550 · Vehicle claim > 36,960100%8 claims
  • Incident severity is “Major Damage” · Property claim ≤ 18,550 · Authorities contacted is “Other”69%65 claims
  • Incident severity is not “Major Damage” · Vehicle claim > 60,870 · Witnesses > 2.563%8 claims
  • Incident severity is “Major Damage” · Property claim ≤ 18,550 · Authorities contacted is not “Other”56%202 claims
  • Incident severity is not “Major Damage” · Vehicle claim ≤ 60,870 · Collision type is “Rear Collision”16%180 claims
  • Incident severity is not “Major Damage” · Vehicle claim > 60,870 · Witnesses ≤ 2.514%37 claims
  • Incident severity is not “Major Damage” · Vehicle claim ≤ 60,870 · Collision type is not “Rear Collision”8%499 claims
  • Incident severity is “Major Damage” · Property claim > 18,550 · Vehicle claim ≤ 36,9600%1 claims
All models

Out-of-fold scores. Average precision (AP) and the share of fraud caught in the top 20% matter most for a review queue.

ModelFeaturesROC-AUCAvg precisionFraud in top 20%
Logistic Regressionclaim facts0.7700.50248.2%
Decision tree (depth 3)claim facts0.7530.48344.9%
Gradient Boostingclaim facts0.7460.50445.7%
Severity onlyone column0.7660.50246.6%
Logistic Regression+ hobbies0.8700.62155.9%
Gradient Boosting+ hobbies0.8600.65652.6%

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

insurance() from ml/train.py + 8 helpers (r, save, clf_metrics, roc_points, hist_by_group, export_logreg, make_pipe, rate_by)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load and clean the claims

    Reads insurance_claims.csv, turns fraud_reported into a 0/1 target and replaces "?" with Unknown and a missing authorities_contacted with None.

    ml/train.py:1378 · 11 lines

    from sklearn.metrics import average_precision_score
    from sklearn.tree import DecisionTreeClassifier
    
    print("insurance-fraud")
    raw = pd.read_csv(RAW / "insurance" / "insurance_claims.csv").dropna(axis=1, how="all")
    df = raw.copy()
    y = (df["fraud_reported"] == "Y").astype(int).to_numpy()
    ys = pd.Series(y)
    for c in ["collision_type", "property_damage", "police_report_available"]:
        df[c] = df[c].replace("?", "Unknown")
    df["authorities_contacted"] = df["authorities_contacted"].fillna("None")
  2. 2

    Choose claim-fact features

    Eight numeric and six categorical facts about the claim; identity and profile fields like sex, age and hobbies are left out, with hobbies tested separately.

    ml/train.py:1391 · 4 lines

    # Left out: identity and profile fields (sex, age, zip, education, occupation, relationship, hobbies),
    # IDs, dates and locations. Hobbies are tested separately to show why.
    num = ["months_as_customer", "umbrella_limit", "injury_claim", "property_claim", "vehicle_claim", "number_of_vehicles_involved", "bodily_injuries", "witnesses"]
    cat = ["incident_severity", "incident_type", "collision_type", "authorities_contacted", "property_damage", "police_report_available"]
  3. 3

    Compare six variants by cross-validation

    Out-of-fold scores (stratified 5-fold) for logistic, a depth-3 tree, boosting, severity alone and two hobby variants, scored by AUC, average precision and top-20% capture.

    ml/train.py:1397 · 31 lines

    cv = StratifiedKFold(5, shuffle=True, random_state=SEED)
    
    def oof(model, n, c):
        return cross_val_predict(make_pipe(model, n, c), df[n + c], y, cv=cv, method="predict_proba")[:, 1]
    
    def top_capture(p, frac):
        o = np.argsort(-p)
        return float(y[o[: int(round(len(p) * frac))]].sum() / y.sum())
    
    variants = [
        ("Logistic Regression", "claim facts", LogisticRegression(max_iter=3000), num, cat),
        ("Decision tree (depth 3)", "claim facts", DecisionTreeClassifier(max_depth=3, random_state=SEED), num, cat),
        ("Gradient Boosting", "claim facts", GradientBoostingClassifier(random_state=SEED), num, cat),
        ("Severity only", "one column", DecisionTreeClassifier(max_depth=3, random_state=SEED), [], ["incident_severity"]),
        ("Logistic Regression", "+ hobbies", LogisticRegression(max_iter=3000), num, cat + ["insured_hobbies"]),
        ("Gradient Boosting", "+ hobbies", GradientBoostingClassifier(random_state=SEED), num, cat + ["insured_hobbies"]),
  4. 4

    Fit final model and tree

    Logistic regression on claim facts is refit on all claims for the demo, and a depth-3 tree is fit too, with its splits turned back into raw values as a checklist.

    ml/train.py:1430 · 31 lines

    p_lr = preds[("Logistic Regression", "claim facts")]
    final = make_pipe(LogisticRegression(max_iter=3000), num, cat).fit(df[num + cat], y)
    
    # readable checklist: a depth-3 tree on the claim facts, trained on all data
    tree_pipe = make_pipe(DecisionTreeClassifier(max_depth=3, random_state=SEED), num, cat).fit(df[num + cat], y)
    names = list(tree_pipe.named_steps["pre"].get_feature_names_out())
    t = tree_pipe.named_steps["model"].tree_
    pre = tree_pipe.named_steps["pre"]
    sc: StandardScaler = pre.named_transformers_["num"]
    
    def describe(i):
        f = names[t.feature[i]]
        thr = t.threshold[i]
        if f.startswith("num__"):
            col = f[5:]
            j = num.index(col)
  5. 5

    Pick demo claims

    Ten real claims (5 fraud, 5 legit) with their out-of-fold logistic scores become the demo presets.

    ml/train.py:1463 · 11 lines

    pick = pd.concat([df[y == 1].sample(5, random_state=SEED), df[y == 0].sample(5, random_state=SEED)]).sample(frac=1, random_state=SEED)
    samples = [
        {
            "id": f"CLM-{int(row.policy_number)}",
            **{c: int(row[c]) for c in num},
            **{c: row[c] for c in cat},
            "actual": "Fraud" if row.fraud_reported == "Y" else "Legit",
            "predicted": r(p_lr[i], 3),
        }
        for i, row in pick.iterrows()
    ]
  6. 6

    Export the JSON

    Fraud rate by hobby, a gains curve, ROC curves for three variants, the model and the tree nodes are written to insurance-fraud.json.

    ml/train.py:1476 · 40 lines

    hobby = df.assign(f=y).groupby("insured_hobbies")["f"].agg(["mean", "count"]).sort_values("mean", ascending=False)
    o = np.argsort(-p_lr)
    gains = [{"pct": k, "captured": r(y[o[: int(round(len(y) * k / 100))]].sum() / y.sum(), 4)} for k in range(1, 101)]
    save(
        "insurance-fraud",
        {
            "dataset": {
                "name": "Auto insurance claims (insurance_claims.csv)",
                "source": "https://data.mendeley.com/datasets/992mh7dk9y/2",
                "mirror": "https://github.com/mwitiderrick/insurancedata",
                "rows": int(len(df)),
                "raw_cols": int(raw.shape[1]),
                "fraud_rate": r(y.mean(), 3),
                "features_used": num + cat,
                "left_out": ["insured_sex", "age", "insured_zip", "insured_education_level", "insured_occupation", "insured_relationship", "insured_hobbies", "policy_number", "dates", "locations", "auto make/model"],
            },

In your browser · TypeScript

  1. 7

    Score it in the browser

    The exported coefficients are plain numbers, so the demo scales the inputs, adds up coefficient × value and applies the sigmoid. No server, no Python.

    src/lib/logreg.ts:11 · 20 lines

    export function predict(model: LogRegModel, input: Record<string, number | string>) {
      const contributions: Contribution[] = [];
      let logit = model.intercept;
    
      for (const [f, p] of Object.entries(model.numeric)) {
        const x = Number(input[f]);
        const effect = p.coef * ((x - p.mean) / p.scale);
        logit += effect;
        contributions.push({ feature: f, value: String(x), effect });
      }
      for (const [f, cats] of Object.entries(model.categorical)) {
        const v = String(input[f]);
        const effect = cats[v] ?? 0;
        logit += effect;
        contributions.push({ feature: f, value: v, effect });
      }
    
      const probability = 1 / (1 + Math.exp(-logit));
      return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Kept missing answers ('?') as an explicit Unknown category instead of guessing.

  2. 2

    Left out every profile field (sex, age, zip, education, occupation, relationship, hobbies) plus IDs, dates and places.

  3. 3

    Compared logistic regression, a depth-3 tree and boosting on the claim facts, plus severity alone as a sanity check.

  4. 4

    Tested hobbies separately: AUC jumps from 0.77 to 0.87, driven by chess and cross-fit. Kept out as an artifact.

Out-of-fold ROC-AUC

Logistic Regression · claim facts0.770chosen
Decision tree (depth 3) · claim facts0.753
Gradient Boosting · claim facts0.746
Severity only · one column0.766
Logistic Regression · + hobbies0.870rejected
Gradient Boosting · + hobbies0.860rejected

Honest note

Most of the signal is one field: incident severity alone scores 0.766 against 0.770 for all claim facts. Only 1,000 claims and no dates to test on later claims, so the score is a triage aid, not a verdict.

Reproduce: ml/train.py → insurance()

dataset source

Next project

Card Fraud Review Queue