←BaldanAll skillsLet's talk
Machine Learning lab

Anomaly detection

Seven months of taxi demand, and the days that broke the pattern.

5/5

Known events found

13

False alerts

5,160

Hours checked

|z| > 4.5

Threshold

Overview

Taxi demand in New York follows a strong daily and weekly rhythm. The useful question for an operations team is when that rhythm breaks: holidays, a marathon, a blizzard.

Instead of a black box, the detector compares each hour with what's normal for that weekday and hour, measures the gap with a robust z-score, and raises an alert past a threshold you can tune yourself below.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    10,320 points

    Passengers every 30 minutes

  2. 02

    Hourly

    5,160 hours

    Two half-hours summed into one

  3. 03

    Baseline

    7 × 24

    Median demand for each weekday and hour

  4. 04

    Score

    robust z

    Gap from expected, scaled by MAD, 3-hour smoothing

  5. 05

    Alert

    |z| > 4.5

    Picked from the threshold sweep

  6. 06

    Evaluate

    5/5 · 13

    Known events found · false alerts

Inside the model

Normal is different every hour.

No neural network here: a baseline per weekday and hour, and a robust score for how far each hour strays from it. Scroll through a real event.

Event
Oct 29Oct 30Oct 31Nov 1Nov 2Nov 3Nov 4Nov 5TAXI RIDES PER HOURHOW UNUSUAL (ROBUST Z-SCORE)+4.5−4.5

The data

Seven months of New York taxi demand.

A single number every 30 minutes, plus five events everyone agrees were unusual.

NAB nyc_taxi (Numenta Anomaly Benchmark)

Taxi passenger counts from the NYC Taxi & Limousine Commission, aggregated every 30 minutes and curated by Numenta for its anomaly benchmark, with labelled event windows.

10,320

raw points

5,160

hours

2014-07-01

from

2015-01-31

to

Dataset on GitHub (Numenta)

Column dictionary

● used by the model

ColumnTypeWhat it means
timestampdatetimeStart of each 30-minute window
valuenumberTaxi passengers in that window
combined_windows.jsonlabelsFive known events (marathon, holidays, blizzard), used only to score the detector
hourly valuederivedTwo half-hours summed into one hour
expectedderivedMedian demand for that weekday and hour
zderivedHow unusual the hour is (robust z-score, 3-hour smoothed)

Explore the data

Demand has a rhythm. Anomalies break it.

Average passengers by hour

Weekdays peak around 19:00, weekends around 0:00 (late nights).

WeekdayWeekend
0.0k25k50k0:006:0012:0018:0023:00
Typical demand: weekday × hour

The expected value the detector compares against. Darker means busier.

MonTueWedThuFriSatSun0h6h12h18h3.9k52k
How unusual each hour is (z-score)

Most hours sit near 0. Only 229 of 5,160 hours (4.4%) cross ±4.5.

010002000−4.5+4.5-8-5-2147

Try it

Jump to a moment, then tune the alarm.

Click a row to pin that hour on the chart. Then move the threshold.

Real rows from the dataset

click a row to try it in the demo ↓

HourPassengersExpectedz|z| > 4.5
Fri, Jul 4, 8 AM10,94138,959-11.50alert
Sat, Jul 5, 12 AM33,76548,974-7.19alert
Mon, Aug 4, 12 PM32,16032,3240.04normal
Tue, Aug 19, 2 AM6,7346,1390.11normal
Tue, Aug 26, 3 AM4,0854,085-0.04normal
Mon, Sep 1, 6 AM6,70217,946-5.51alert
Wed, Nov 12, 2 PM36,29237,015-0.45normal
Thu, Nov 27, 7 PM27,78849,724-9.25alert
Sat, Dec 27, 2 AM17,37034,660-7.69alert
Thu, Jan 1, 12 AM51,70024,42010.61alert
Thu, Jan 1, 2 AM51,5079,12719.16alert
Mon, Jan 26, 10 PM2,64935,948-14.45alert

Live demo

runs in your browser

Lower = more sensitive. Higher = fewer false alarms.

5/5

known events found

13

false alerts

Taxi passengers per hour, New York

passengers flagged hour known event
0k20k40k60k80k✓ NYC Marathon✓ Thanksgiving✓ Christmas✓ New Year✓ BlizzardJulAugSepOctNovDecJan

How well it works

Choosing the alarm level.

False alerts by threshold

All 5 known events are found up to |z| 4.5; from 5 one is missed.

05010023456Threshold |z|chosen
Known events

Each labelled window and whether the chosen threshold catches it.

  • NYC Marathonpeak z 4.9caught
  • Thanksgivingpeak z -10.5caught
  • Christmaspeak z -14.1caught
  • New Yearpeak z 19.2caught
  • Blizzardpeak z -15.8caught

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

nyc_taxi() from ml/train.py + 2 helpers (r, save)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load and resample to hours

    Reads the NAB taxi series, sums the half-hourly counts into hourly totals, and tags each hour with its weekday and hour of day.

    ml/train.py:393 · 6 lines

    print("nyc-taxi-anomaly")
    df = pd.read_csv(RAW / "nyc_taxi" / "nyc_taxi.csv", parse_dates=["timestamp"])
    s = df.set_index("timestamp")["value"].resample("1h").sum()
    hourly = s.to_frame("value")
    hourly["dow"] = hourly.index.day_of_week
    hourly["hour"] = hourly.index.hour
  2. 2

    Build a baseline and z-score

    Expected demand is the median for that weekday and hour; the gap from it is scaled by the MAD into a robust z-score and smoothed over 3 hours.

    ml/train.py:401 · 6 lines

    # Expected demand for that weekday-hour, robust to the anomalies themselves (median).
    baseline = hourly.groupby(["dow", "hour"])["value"].transform("median")
    resid = hourly["value"] - baseline
    mad = np.median(np.abs(resid - np.median(resid)))
    z = (resid / (1.4826 * mad)).rolling(3, center=True, min_periods=1).mean()
    hourly["baseline"], hourly["z"] = baseline, z
  3. 3

    Score alerts against known events

    Loads the five labelled event windows and defines how a threshold is judged: events caught versus false alerts, with flags within 6 hours grouped into one.

    ml/train.py:409 · 14 lines

    windows = json.loads((RAW / "nyc_taxi" / "combined_windows.json").read_text())["realKnownCause/nyc_taxi.csv"]
    windows = [[pd.Timestamp(a), pd.Timestamp(b)] for a, b in windows]
    names = ["NYC Marathon", "Thanksgiving", "Christmas", "New Year", "Blizzard"]
    
    def evaluate(threshold: float) -> dict:
        flags = hourly.index[np.abs(hourly["z"]) > threshold]
        hit = [any((t >= a) & (t <= b) for t in flags) for a, b in windows]
        outside = [t for t in flags if not any(a <= t <= b for a, b in windows)]
        events, last = 0, None  # group flags closer than 6h into one "alert"
        for t in outside:
            if last is None or (t - last) > pd.Timedelta(hours=6):
                events += 1
            last = t
        return {"threshold": threshold, "detected": int(sum(hit)), "of": len(windows), "false_alerts": events}
  4. 4

    Sweep the alert threshold

    Every z threshold from 2.0 to 6.0 in steps of 0.5 is scored this way, and 4.5 is set as the page's default.

    ml/train.py:425 · 4 lines

    sweep = [evaluate(t) for t in [2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0]]
    default = 4.5
    for e in sweep:
        print(f"  z>{e['threshold']}: detected {e['detected']}/{e['of']}, false alerts {e['false_alerts']}")
  5. 5

    Pick demo hours

    The 5 biggest moments inside known events (one per day), the 3 biggest outside them, and 4 ordinary hours with a z-score under 0.5.

    ml/train.py:431 · 19 lines

    # sample rows: biggest moments inside known events, a few outside, a few ordinary hours
    inside = hourly[[any(a <= t <= b for a, b in windows) for t in hourly.index]]
    outside = hourly.drop(inside.index)
    pick = pd.concat(
        [
            inside.reindex(inside["z"].abs().sort_values(ascending=False).index).groupby(inside.index.date).head(1).head(5),
            outside.reindex(outside["z"].abs().sort_values(ascending=False).index).groupby(outside.index.date).head(1).head(3),
            outside[outside["z"].abs() < 0.5].sample(4, random_state=SEED),
        ]
    ).sort_index()
    samples = [
        {
            "time": str(t),
            "value": int(row.value),
            "expected": int(row.baseline),
            "z": r(row.z, 2),
        }
        for t, row in pick.iterrows()
    ]
  6. 6

    Export everything the page needs

    Daily profiles, the weekday-hour heatmap, z histogram, threshold sweep and the full hourly series, written to src/data/models/nyc-taxi-anomaly.json.

    ml/train.py:452 · 36 lines

    weekday = hourly[hourly["dow"] < 5].groupby("hour")["value"].mean()
    weekend = hourly[hourly["dow"] >= 5].groupby("hour")["value"].mean()
    heat = hourly.groupby(["dow", "hour"])["value"].median().unstack()
    zb = np.arange(-8, 8.5, 0.5)
    zcounts = np.histogram(hourly["z"].clip(-8, 8), bins=zb)[0]
    
    save(
        "nyc-taxi-anomaly",
        {
            "dataset": {
                "name": "NAB nyc_taxi (Numenta Anomaly Benchmark)",
                "source": "https://github.com/numenta/NAB",
                "points_raw": int(len(df)),
                "hours": int(len(hourly)),
                "resolution": "30 min, aggregated to hourly",
                "period": [str(hourly.index.min().date()), str(hourly.index.max().date())],

In your browser · TypeScript

  1. 7

    Re-score the alerts in the browser

    The same evaluation as in Python: flag every hour beyond the threshold, count known events caught and group the false alerts.

    src/components/demos/anomaly-demo.tsx:27 · 15 lines

    // Same evaluation as ml/train.py: known events hit + false alerts grouped by 6h gaps.
    function evaluate(th: number) {
      const flags: number[] = [];
      for (let i = 0; i < N; i++) if (Math.abs(zs[i]) > th) flags.push(i);
      const inside = (i: number) => windows.some((w) => i >= w.a && i <= w.b);
      const hit = windows.map((w) => flags.some((i) => i >= w.a && i <= w.b));
      let alerts = 0;
      let last = -Infinity;
      for (const i of flags) {
        if (inside(i)) continue;
        if (i - last > 6) alerts++;
        last = i;
      }
      return { flags, hit, alerts };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Built the expected demand for each weekday-hour with the median, so outliers don't bend the baseline.

  2. 2

    Measured how far each hour sits from it with a robust z-score (MAD), smoothed over 3 hours.

  3. 3

    Swept the threshold and counted known events found versus false alerts.

Threshold sweep

|z| > 25/5 found · 86 false alerts
|z| > 2.55/5 found · 59 false alerts
|z| > 35/5 found · 41 false alerts
|z| > 3.55/5 found · 25 false alerts
|z| > 45/5 found · 18 false alerts
|z| > 4.55/5 found · 13 false alertschosen
|z| > 54/5 found · 11 false alerts
|z| > 5.54/5 found · 9 false alerts
|z| > 64/5 found · 7 false alerts

Honest note

The labelled windows span several days, which makes detection easier. Most of the "false alerts" land on July 4th weekend, Labor Day weekend and MLK Day: real holidays the benchmark simply doesn't label. The slider shows the trade-off.

Reproduce: ml/train.py → nyc_taxi()

dataset source

Next project

Credit Risk