Classification
Scoring auto claims for review, and refusing the signal that looks too good to be true.
0.770
ROC-AUC (claim facts)
48%
Fraud in top 20%
0.766
Severity alone
1,000
Claims
Overview
An insurer can't investigate every claim, so it needs a score that ranks claims for review and reasons an investigator can check. With a quarter of these claims labelled fraud, the risk is less a weak model and more a model that learns the wrong thing.
The rebuild uses only facts about the claim and the incident, gives investigators a three-question checklist from a small tree, and shows openly what happens with a field that should never be used: the policy holder's hobby.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
1,000 claims
39 columns, 25% fraud
- 02
Clean
? → Unknown
Missing answers kept as their own category
- 03
Fairness
−24 columns
Profile, IDs, dates, places and hobbies out
- 04
Features
14 claim facts
Severity, incident, authorities, amounts, witnesses
- 05
Validate
5-fold OOF
Each claim scored by a model that never saw it
- 06
Explain
3 questions
Depth-3 tree as an investigator checklist
Inside the model
One claim, one neuron.
Logistic regression is a single artificial neuron. Scroll to watch it score one real insurance claim with the model's real weights.
Inside the model
Then a three-question checklist.
Next to the score, a small decision tree gives investigators a checklist they can follow by hand and explain to a customer.
CLM-315041
Severity Total Loss
Collision Rear Collision
Authorities Ambulance
Vehicle claim 45,850
Property claim 19,650
Witnesses 3
The data
1,000 auto claims, 25% flagged as fraud.
39 columns about the policy, the person, the incident and the money. Most of them shouldn't go near a fraud model.
Auto insurance claims (insurance_claims.csv)
Auto-insurance claims from three US states with a fraud label, published on Mendeley Data under CC BY 4.0 and widely mirrored on GitHub.
1,000
claims
24.7%
fraud rate
14
features used
24
columns left out
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| months_as_customer | number | How long the customer has held a policy |
| age · insured_sex · insured_zip | profile | Left out (fairness) |
| insured_education_level · insured_occupation · insured_relationship | profile | Left out (fairness) |
| insured_hobbies | profile | Left out: a suspiciously strong artifact, tested separately below |
| policy_number · policy_bind_date · policy_state | policy | Identifiers and dates · not used |
| policy_csl · policy_deductable · policy_annual_premium | policy | Coverage terms · not used in the final model |
| umbrella_limit | number | Extra liability cover (one row has an impossible −1,000,000) |
| incident_type · collision_type | category | What happened ('?' kept as Unknown) |
| incident_severity | category | Trivial, minor, major damage or total loss |
| authorities_contacted | category | Police, fire, ambulance, other or none |
| incident_state · city · location · hour | where/when | Not used |
| number_of_vehicles_involved · bodily_injuries · witnesses | number | Incident facts |
| property_damage · police_report_available | yes/no/? | '?' kept as Unknown |
| injury_claim · property_claim · vehicle_claim | number ($) | Claim amounts (total_claim_amount is their sum, so not used) |
| auto_make · auto_model · auto_year | vehicle | Not used |
| fraud_reported | target | Y / N: the thing we predict |
Explore the data
One field shouts. One field lies.
“Major damage” claims are 61% fraud. Every other level sits between 7% and 13%.
Chess 83%, cross-fit 74%, everything else under 31%. Too strong to be real: almost certainly an artifact of how the data was made.
When nobody was called, only 7% are fraud.
Barely matters: 25%, 26%, 23% across no / unknown / yes.
Below $10k: 21% of legit claims vs 6% of fraud. Fraud leans to 50–80k.
Try it
Load a claim, then argue with the model.
Five fraud and five legitimate claims. Scores in the table come from models that never saw the claim.
Real rows from the dataset
click a row to try it in the demo ↓
| Claim | Severity | Type | Authorities | Vehicle $ | Score | Outcome |
|---|---|---|---|---|---|---|
| CLM-283414 | Major Damage | Multi-vehicle Collision | Ambulance | 51,940 | 42% | Legit |
| CLM-285496 | Major Damage | Multi-vehicle Collision | Ambulance | 47,680 | 38% | Fraud |
| CLM-354455 | Minor Damage | Single Vehicle Collision | Ambulance | 35,140 | 14% | Legit |
| CLM-648397 | Total Loss | Multi-vehicle Collision | Other | 33,390 | 18% | Fraud |
| CLM-555374 | Minor Damage | Single Vehicle Collision | Police | 38,220 | 16% | Legit |
| CLM-315041 | Total Loss | Multi-vehicle Collision | Ambulance | 45,850 | 6% | Fraud |
| CLM-655787 | Minor Damage | Multi-vehicle Collision | Police | 53,970 | 9% | Legit |
| CLM-110084 | Total Loss | Single Vehicle Collision | Other | 35,000 | 10% | Fraud |
| CLM-931625 | Total Loss | Multi-vehicle Collision | Fire | 47,280 | 11% | Fraud |
| CLM-979963 | Total Loss | Single Vehicle Collision | Fire | 52,780 | 9% | Legit |
Score is out-of-fold: it comes from a model that never saw this claim. The demo runs the final model trained on every claim, so its number can differ by a few points.
Live demo
runs in your browser
The claim
Incident severity
Property damage
Police report
Fraud score
Investigator checklist (3 questions)
- Incident severity is not “Major Damage”
- Vehicle claim ≤ 60,870
- Collision type is “Rear Collision”
- Fast-track · 16% of 180 similar claims were fraud
Why
- Incident severityMinor Damage
- Collision typeRear Collision
- Property damageUnknown
- Umbrella limit0
- Authorities contactedPolice
How well it works
Honest signal: AUC 0.77. Tempting signal: 0.87.
Severity alone scores 0.766; all the claim facts together, 0.770. Adding hobbies jumps to 0.870, and that's exactly why it stays out.
The hobby curve looks better. It wouldn't hold up in front of a customer, a regulator or new data.
Investigating the top 20% of claims by score catches 48% of all fraud.
134 of 247 fraud claims flagged, 90 legitimate claims sent for review.
Every path through the depth-3 tree, riskiest first. This is what an investigator would actually read.
- Incident severity is “Major Damage” · Property claim > 18,550 · Vehicle claim > 36,960100%8 claims
- Incident severity is “Major Damage” · Property claim ≤ 18,550 · Authorities contacted is “Other”69%65 claims
- Incident severity is not “Major Damage” · Vehicle claim > 60,870 · Witnesses > 2.563%8 claims
- Incident severity is “Major Damage” · Property claim ≤ 18,550 · Authorities contacted is not “Other”56%202 claims
- Incident severity is not “Major Damage” · Vehicle claim ≤ 60,870 · Collision type is “Rear Collision”16%180 claims
- Incident severity is not “Major Damage” · Vehicle claim > 60,870 · Witnesses ≤ 2.514%37 claims
- Incident severity is not “Major Damage” · Vehicle claim ≤ 60,870 · Collision type is not “Rear Collision”8%499 claims
- Incident severity is “Major Damage” · Property claim > 18,550 · Vehicle claim ≤ 36,9600%1 claims
Out-of-fold scores. Average precision (AP) and the share of fraud caught in the top 20% matter most for a review queue.
| Model | Features | ROC-AUC | Avg precision | Fraud in top 20% |
|---|---|---|---|---|
| Logistic Regression | claim facts | 0.770 | 0.502 | 48.2% |
| Decision tree (depth 3) | claim facts | 0.753 | 0.483 | 44.9% |
| Gradient Boosting | claim facts | 0.746 | 0.504 | 45.7% |
| Severity only | one column | 0.766 | 0.502 | 46.6% |
| Logistic Regression | + hobbies | 0.870 | 0.621 | 55.9% |
| Gradient Boosting | + hobbies | 0.860 | 0.656 | 52.6% |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
insurance() from ml/train.py + 8 helpers (r, save, clf_metrics, roc_points, hist_by_group, export_logreg, make_pipe, rate_by)
pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load and clean the claims
Reads insurance_claims.csv, turns fraud_reported into a 0/1 target and replaces "?" with Unknown and a missing authorities_contacted with None.
ml/train.py:1378 · 11 lines
from sklearn.metrics import average_precision_score from sklearn.tree import DecisionTreeClassifier print("insurance-fraud") raw = pd.read_csv(RAW / "insurance" / "insurance_claims.csv").dropna(axis=1, how="all") df = raw.copy() y = (df["fraud_reported"] == "Y").astype(int).to_numpy() ys = pd.Series(y) for c in ["collision_type", "property_damage", "police_report_available"]: df[c] = df[c].replace("?", "Unknown") df["authorities_contacted"] = df["authorities_contacted"].fillna("None") - 2
Choose claim-fact features
Eight numeric and six categorical facts about the claim; identity and profile fields like sex, age and hobbies are left out, with hobbies tested separately.
ml/train.py:1391 · 4 lines
# Left out: identity and profile fields (sex, age, zip, education, occupation, relationship, hobbies), # IDs, dates and locations. Hobbies are tested separately to show why. num = ["months_as_customer", "umbrella_limit", "injury_claim", "property_claim", "vehicle_claim", "number_of_vehicles_involved", "bodily_injuries", "witnesses"] cat = ["incident_severity", "incident_type", "collision_type", "authorities_contacted", "property_damage", "police_report_available"] - 3
Compare six variants by cross-validation
Out-of-fold scores (stratified 5-fold) for logistic, a depth-3 tree, boosting, severity alone and two hobby variants, scored by AUC, average precision and top-20% capture.
ml/train.py:1397 · 31 lines
cv = StratifiedKFold(5, shuffle=True, random_state=SEED) def oof(model, n, c): return cross_val_predict(make_pipe(model, n, c), df[n + c], y, cv=cv, method="predict_proba")[:, 1] def top_capture(p, frac): o = np.argsort(-p) return float(y[o[: int(round(len(p) * frac))]].sum() / y.sum()) variants = [ ("Logistic Regression", "claim facts", LogisticRegression(max_iter=3000), num, cat), ("Decision tree (depth 3)", "claim facts", DecisionTreeClassifier(max_depth=3, random_state=SEED), num, cat), ("Gradient Boosting", "claim facts", GradientBoostingClassifier(random_state=SEED), num, cat), ("Severity only", "one column", DecisionTreeClassifier(max_depth=3, random_state=SEED), [], ["incident_severity"]), ("Logistic Regression", "+ hobbies", LogisticRegression(max_iter=3000), num, cat + ["insured_hobbies"]), ("Gradient Boosting", "+ hobbies", GradientBoostingClassifier(random_state=SEED), num, cat + ["insured_hobbies"]), - 4
Fit final model and tree
Logistic regression on claim facts is refit on all claims for the demo, and a depth-3 tree is fit too, with its splits turned back into raw values as a checklist.
ml/train.py:1430 · 31 lines
p_lr = preds[("Logistic Regression", "claim facts")] final = make_pipe(LogisticRegression(max_iter=3000), num, cat).fit(df[num + cat], y) # readable checklist: a depth-3 tree on the claim facts, trained on all data tree_pipe = make_pipe(DecisionTreeClassifier(max_depth=3, random_state=SEED), num, cat).fit(df[num + cat], y) names = list(tree_pipe.named_steps["pre"].get_feature_names_out()) t = tree_pipe.named_steps["model"].tree_ pre = tree_pipe.named_steps["pre"] sc: StandardScaler = pre.named_transformers_["num"] def describe(i): f = names[t.feature[i]] thr = t.threshold[i] if f.startswith("num__"): col = f[5:] j = num.index(col) - 5
Pick demo claims
Ten real claims (5 fraud, 5 legit) with their out-of-fold logistic scores become the demo presets.
ml/train.py:1463 · 11 lines
pick = pd.concat([df[y == 1].sample(5, random_state=SEED), df[y == 0].sample(5, random_state=SEED)]).sample(frac=1, random_state=SEED) samples = [ { "id": f"CLM-{int(row.policy_number)}", **{c: int(row[c]) for c in num}, **{c: row[c] for c in cat}, "actual": "Fraud" if row.fraud_reported == "Y" else "Legit", "predicted": r(p_lr[i], 3), } for i, row in pick.iterrows() ] - 6
Export the JSON
Fraud rate by hobby, a gains curve, ROC curves for three variants, the model and the tree nodes are written to insurance-fraud.json.
ml/train.py:1476 · 40 lines
hobby = df.assign(f=y).groupby("insured_hobbies")["f"].agg(["mean", "count"]).sort_values("mean", ascending=False) o = np.argsort(-p_lr) gains = [{"pct": k, "captured": r(y[o[: int(round(len(y) * k / 100))]].sum() / y.sum(), 4)} for k in range(1, 101)] save( "insurance-fraud", { "dataset": { "name": "Auto insurance claims (insurance_claims.csv)", "source": "https://data.mendeley.com/datasets/992mh7dk9y/2", "mirror": "https://github.com/mwitiderrick/insurancedata", "rows": int(len(df)), "raw_cols": int(raw.shape[1]), "fraud_rate": r(y.mean(), 3), "features_used": num + cat, "left_out": ["insured_sex", "age", "insured_zip", "insured_education_level", "insured_occupation", "insured_relationship", "insured_hobbies", "policy_number", "dates", "locations", "auto make/model"], },
In your browser · TypeScript
- 7
Score it in the browser
The exported coefficients are plain numbers, so the demo scales the inputs, adds up coefficient × value and applies the sigmoid. No server, no Python.
src/lib/logreg.ts:11 · 20 lines
export function predict(model: LogRegModel, input: Record<string, number | string>) { const contributions: Contribution[] = []; let logit = model.intercept; for (const [f, p] of Object.entries(model.numeric)) { const x = Number(input[f]); const effect = p.coef * ((x - p.mean) / p.scale); logit += effect; contributions.push({ feature: f, value: String(x), effect }); } for (const [f, cats] of Object.entries(model.categorical)) { const v = String(input[f]); const effect = cats[v] ?? 0; logit += effect; contributions.push({ feature: f, value: v, effect }); } const probability = 1 / (1 + Math.exp(-logit)); return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) }; }
Notes
The decisions behind it.
Approach
- 1
Kept missing answers ('?') as an explicit Unknown category instead of guessing.
- 2
Left out every profile field (sex, age, zip, education, occupation, relationship, hobbies) plus IDs, dates and places.
- 3
Compared logistic regression, a depth-3 tree and boosting on the claim facts, plus severity alone as a sanity check.
- 4
Tested hobbies separately: AUC jumps from 0.77 to 0.87, driven by chess and cross-fit. Kept out as an artifact.
Out-of-fold ROC-AUC
Honest note
Most of the signal is one field: incident severity alone scores 0.766 against 0.770 for all claim facts. Only 1,000 claims and no dates to test on later claims, so the score is a triage aid, not a verdict.
Reproduce: ml/train.py → insurance()
dataset sourceNext project
Card Fraud Review Queue