←BaldanAll skillsLet's talk
Machine Learning lab

Classification

Approve or decline a loan, when one mistake costs five times more than the other.

−39%

Cost vs 50% rule

0.779

ROC-AUC (CV)

1,000

Loans

30%

Default rate

Overview

A bank loses far more when it approves a borrower who defaults than when it turns away one who would have repaid. This dataset even says how much more: five to one.

So the goal isn't the highest accuracy. It's the cheapest decisions. The model scores default risk, and the decision threshold comes straight from the cost matrix instead of the usual 50%. Sensitive attributes such as sex, marital status, age and foreign-worker status are left out on purpose.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    1,000 loans

    20 coded attributes + outcome

  2. 02

    Decode

    A11 → words

    Every code mapped back to its meaning

  3. 03

    Fairness

    −3 attributes

    Sex & marital status, age, foreign worker left out

  4. 04

    Features

    11 → 42 columns

    3 numbers scaled, 8 categories one-hot

  5. 05

    Validate

    25 folds

    3 models, 5-fold CV × 5 repeats

  6. 06

    Decide

    threshold 1/6

    Where 5 × risk = 1 × (1 − risk)

Inside the model

One loan, one neuron.

Logistic regression is a single artificial neuron. The decision line sits low on purpose: approving a bad loan costs five times more than turning down a good one.

loan
Checking accountNo accountCredit historyCritical / credit…employment4–7 yearspurposeRepairssavingsUnknown / noneOther instalmentsNone5 more inputssmall pushesBIAS (STARTING POINT)−0.87ΣNEURON0%50%100%1.8%chance of defaultSIGMOID: SUM → 0–100%SUMWEIGHTS GROWING FROM 0ΣΣΣΣΣΣLAYERS OF NEURONS = A NEURAL NETWORK

The data

1,000 loans, 20 attributes.

Every column arrives as a code (A11, A34…). Step one was turning codes back into words.

Statlog German Credit (UCI)

Loan applications from a German bank, published by Prof. Hans Hofmann on the UCI repository. It ships with its own cost matrix: approving a bad borrower costs 5, declining a good one costs 1.

1,000

applications

30.0%

default rate

11

features used

3

left out (fairness)

Dataset on UCI

Column dictionary

● used by the model

ColumnTypeWhat it means
checkingcategoryBalance of the current account (DM)
durationnumberLoan length in months
historycategoryHow past credits were repaid
purposecategoryWhat the loan is for
amountnumberLoan amount (Deutsche Mark)
savingscategorySavings account or bonds
employmentcategoryYears at the current job
installment_ratenumberInstallment as % of disposable income (1–4)
personal_status_sexcategorySex and marital status · left out (fairness)
other_debtorscategoryCo-applicant or guarantor
residence_sincenumberYears at the current address
propertycategoryMost valuable property
agenumberAge in years · left out (fairness)
other_planscategoryOther installment plans (bank, stores)
housingcategoryRent, own or live for free
existing_creditsnumberCredits already at this bank
jobcategorySkill level of the job
dependentsnumberPeople the applicant supports
telephoneyes/noRegistered phone line
foreign_workeryes/noForeign worker · left out (fairness)
targettargetRepaid (1) or defaulted (2): the thing we predict

Explore the data

The account says a lot.

Default rate cut four ways. Hover the bars, or flip any chart to a table.

Default rate by checking account

Overdrawn accounts default at 49%; applicants without a checking account at 12%.

0%50%100%49%< 0 DM0–200 DM≥ 200 DM12%No account
Default rate by loan duration

Rises from 21% for loans up to a year to 52% beyond three years.

0%50%100%21%≤ 1213–2425–3652%37–72
Default rate by credit history

Counter-intuitive: “critical” histories default least (17%), “no credits / all repaid” most (63%). Likely how the loans were sampled: a reason to question data, not trust it.

0%50%100%63%No creditsAll repaid at…On timeDelays in the…17%Critical
Loan amount: repaid vs defaulted

Above 6,000 DM: 23% of defaulted loans vs 12% of repaid ones.

RepaidDefaulted
0%25%50%0k3k6k9k12k

Try it

Load an application, then decide.

Five applications that defaulted, five that repaid. Scores in the table come from models that never saw that loan.

Real rows from the dataset

click a row to try it in the demo ↓

LoanMonthsAmount DMCheckingPurposeSavingsDefault riskOutcome
L021669320–200 DMRetrainingUnknown / none2%Repaid
L08884815,6720–200 DMBusiness< 100 DM66%Defaulted
L0219243,021< 0 DMFurniture< 100 DM43%Repaid
L066212900< 0 DMCar (new)Unknown / none30%Defaulted
L055112996No accountRepairsUnknown / none2%Repaid
L0546241,333< 0 DMCar (new)< 100 DM55%Defaulted
L0442121,620< 0 DMFurniture< 100 DM28%Repaid
L0781394,9330–200 DMRadio / TV< 100 DM11%Defaulted
L0036454,7460–200 DMRadio / TV< 100 DM46%Defaulted
L0713212,476No accountCar (used)Unknown / none2%Repaid

Live demo

runs in your browser

Loan application

Installment as % of disposable income (1 = low, 4 = high)

Chance this loan defaults

33%Medium risk
0%17%50%100%

Decision policy

Declinerisk 33% ≥ 16.7%

Expected cost of each choice

Approve · 5 × risk1.67
Decline · 1 × (1 − risk)0.67

The cost-aware policy simply picks the cheaper bar.

Why

lowers risk raises risk
  • Savings< 100 DM
  • Checking account0–200 DM
  • Other installment plansNone
  • PurposeRadio / TV
  • HousingOwn
  • Employed for1–4 years

How well it works

Same model, 39% lower cost.

Ranking quality is similar across models. The real win comes from choosing the threshold the costs call for.

Total cost by decision threshold

At 50% the errors cost 906. At 1/6, where 5 × risk equals 1 × (1 − risk), they cost 549.

075015000.100.300.500.700.90Decline when risk ≥ threshold1/6 (chosen)
Errors at a naive 50% threshold

163 bad loans approved (×5), 91 good ones declined (×1).

Predicted repayPredicted defaultActually repay
Correct609
False alarm91
Actually default
Missed163
Caught137
Errors at the cost-aware 1/6 threshold

Only 41 bad loans slip through, but 344 good ones are declined.

Predicted repayPredicted defaultActually repay
Correct356
False alarm344
Actually default
Missed41
Caught259
ROC-AUC on every fold

25 folds each. Best mean: random forest 0.786; logistic 0.779, well inside the spread.

Logistic regressionRandom forestGradient boosting
0.600.690.770.860.95Logistic regression0.78Random forest0.79Gradient boosting0.78

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

credit() from ml/train.py + 12 helpers (r, save, clf_metrics, roc_points, hist_by_group, export_logreg, make_pipe, rate_by, CREDIT_COLS, CREDIT_CODES, COST_FN, COST_FP)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load and decode the codes

    Reads the raw german.data file, maps codes like A11 to readable labels, and marks bad credit (target 2) as 1.

    ml/train.py:528 · 7 lines

    print("credit-risk")
    raw = pd.read_csv(RAW / "credit" / "german.data", sep=r"\s+", header=None, names=CREDIT_COLS)
    df = raw.copy()
    for c, m in CREDIT_CODES.items():
        df[c] = df[c].map(m)
    y = (df["target"] == 2).astype(int).to_numpy()  # 1 = bad credit (defaulted)
    ys = pd.Series(y)
  2. 2

    Choose the features

    Three numbers and eight categories; personal status/sex, age and foreign-worker status are left out for fairness, telephone as irrelevant.

    ml/train.py:537 · 5 lines

    # Left out on purpose: personal status/sex, age and foreign-worker status (fairness),
    # telephone (irrelevant), plus a few weak extras to keep the form short.
    numeric = ["duration", "amount", "installment_rate"]
    categorical = ["checking", "history", "purpose", "savings", "employment", "property", "other_plans", "housing"]
    X = df[numeric + categorical]
  3. 3

    Cross-validate three models

    Logistic regression, a random forest and gradient boosting are scored by ROC AUC over stratified 5-fold cross-validation repeated 5 times.

    ml/train.py:544 · 12 lines

    cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=5, random_state=SEED)
    models = {
        "Logistic Regression": LogisticRegression(max_iter=3000),
        "Random Forest": RandomForestClassifier(n_estimators=500, min_samples_leaf=3, random_state=SEED, n_jobs=-1),
        "Gradient Boosting": GradientBoostingClassifier(random_state=SEED),
    }
    comparison, folds = [], {}
    for name, model in models.items():
        scores = cross_val_score(make_pipe(model, numeric, categorical), X, y, cv=cv, scoring="roc_auc")
        comparison.append({"model": name, "roc_auc_mean": r(scores.mean()), "roc_auc_std": r(scores.std())})
        folds[name] = [r(s, 3) for s in scores]
        print(f"  {name:22s} AUC={scores.mean():.3f} ± {scores.std():.3f}")
  4. 4

    Price every decision threshold

    Using out-of-fold risks, each threshold from 0.05 to 0.95 is costed (approving a defaulter costs 5, declining a good payer 1) and compared with 0.5 and 1/6.

    ml/train.py:558 · 15 lines

    oof = cross_val_predict(make_pipe(models["Logistic Regression"], numeric, categorical), X, y, cv=StratifiedKFold(5, shuffle=True, random_state=SEED), method="predict_proba")[:, 1]
    
    # Cost of each decision threshold: decline when P(default) >= t
    def cost_at(t: float) -> dict:
        decline = oof >= t
        fn = int(((~decline) & (y == 1)).sum())  # approved, then defaulted
        fp = int((decline & (y == 0)).sum())  # declined a good customer
        return {"t": r(t, 2), "cost": fn * COST_FN + fp * COST_FP, "approved_bad": fn, "declined_good": fp, "declined": int(decline.sum())}
    
    curve = [cost_at(t) for t in np.round(np.arange(0.05, 0.96, 0.05), 2)]
    best = min(curve, key=lambda c: c["cost"])
    bayes_t = COST_FP / (COST_FP + COST_FN)  # 1/6: where the two expected costs are equal
    at_half = cost_at(0.5)
    at_bayes = cost_at(bayes_t)
    print(f"  cost @0.50={at_half['cost']}  @{bayes_t:.3f}={at_bayes['cost']}  best grid t={best['t']} cost={best['cost']}")
  5. 5

    Final model and demo loans

    Logistic regression is refit on every applicant for the demo, and ten real loans (5 defaulted, 5 repaid) become presets with their out-of-fold risk.

    ml/train.py:575 · 14 lines

    final = make_pipe(LogisticRegression(max_iter=3000), numeric, categorical).fit(X, y)
    encoded_cols = int(final.named_steps["pre"].transform(X.head(1)).shape[1])
    
    pick = pd.concat([df[y == 1].sample(5, random_state=SEED), df[y == 0].sample(5, random_state=SEED)]).sample(frac=1, random_state=SEED)
    samples = [
        {
            "id": f"L{int(i) + 1:04d}",
            **{c: int(row[c]) for c in numeric},
            **{c: row[c] for c in categorical},
            "actual": "Defaulted" if row.target == 2 else "Repaid",
            "predicted": r(oof[i], 3),
        }
        for i, row in pick.iterrows()
    ]
  6. 6

    Export everything the page needs

    Costs, the logistic model as plain numbers, EDA rates, the cost curve and confusion matrices at 0.5 and 1/6, written to src/data/models/credit-risk.json.

    ml/train.py:591 · 42 lines

    duration_band = pd.cut(df["duration"], bins=[0, 12, 24, 36, 72], labels=["≤ 12", "13–24", "25–36", "37–72"])
    save(
        "credit-risk",
        {
            "dataset": {
                "name": "Statlog German Credit (UCI)",
                "source": "https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data",
                "rows": int(len(df)),
                "raw_cols": int(raw.shape[1]),
                "default_rate": r(y.mean(), 3),
                "features_used": numeric + categorical,
                "left_out": ["personal_status_sex", "age", "foreign_worker", "telephone", "other_debtors", "residence_since", "existing_credits", "job", "dependents"],
                "encoded_cols": encoded_cols,
            },
            "costs": {"approve_bad": COST_FN, "decline_good": COST_FP, "bayes_threshold": r(bayes_t, 4)},
            "validation": "Repeated stratified 5-fold cross-validation (5 repeats)",

In your browser · TypeScript

  1. 7

    Score it in the browser

    The exported coefficients are plain numbers, so the demo scales the inputs, adds up coefficient × value and applies the sigmoid. No server, no Python.

    src/lib/logreg.ts:11 · 20 lines

    export function predict(model: LogRegModel, input: Record<string, number | string>) {
      const contributions: Contribution[] = [];
      let logit = model.intercept;
    
      for (const [f, p] of Object.entries(model.numeric)) {
        const x = Number(input[f]);
        const effect = p.coef * ((x - p.mean) / p.scale);
        logit += effect;
        contributions.push({ feature: f, value: String(x), effect });
      }
      for (const [f, cats] of Object.entries(model.categorical)) {
        const v = String(input[f]);
        const effect = cats[v] ?? 0;
        logit += effect;
        contributions.push({ feature: f, value: v, effect });
      }
    
      const probability = 1 / (1 + Math.exp(-logit));
      return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Decoded every attribute (A11, A34…) back into plain words using the dataset's documentation.

  2. 2

    Left out sex and marital status, age and foreign-worker status so the model can't lean on them.

  3. 3

    Compared logistic regression, random forest and gradient boosting on 25 cross-validation folds; the spread was wider than the gaps, so the readable model stayed.

  4. 4

    Set the decision threshold at 1/6, where the expected cost of approving equals the cost of declining, and checked it on out-of-fold predictions.

Mean ROC-AUC across 25 folds

Logistic Regression0.779 ± 0.03chosen
Random Forest0.786 ± 0.03
Gradient Boosting0.783 ± 0.03

Honest note

The cost-aware rule declines 603 of 1,000 applications. That's what a 5:1 cost matrix asks for, but a real lender would also count the profit lost on good customers turned away. Removing sensitive attributes doesn't guarantee fairness on its own; proxies can remain.

Reproduce: ml/train.py → credit()

dataset source

Next project

Manufacturing Defects