Anomaly detection
Seven months of taxi demand, and the days that broke the pattern.
5/5
Known events found
13
False alerts
5,160
Hours checked
|z| > 4.5
Threshold
Overview
Taxi demand in New York follows a strong daily and weekly rhythm. The useful question for an operations team is when that rhythm breaks: holidays, a marathon, a blizzard.
Instead of a black box, the detector compares each hour with what's normal for that weekday and hour, measures the gap with a robust z-score, and raises an alert past a threshold you can tune yourself below.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
10,320 points
Passengers every 30 minutes
- 02
Hourly
5,160 hours
Two half-hours summed into one
- 03
Baseline
7 × 24
Median demand for each weekday and hour
- 04
Score
robust z
Gap from expected, scaled by MAD, 3-hour smoothing
- 05
Alert
|z| > 4.5
Picked from the threshold sweep
- 06
Evaluate
5/5 · 13
Known events found · false alerts
Inside the model
Normal is different every hour.
No neural network here: a baseline per weekday and hour, and a robust score for how far each hour strays from it. Scroll through a real event.
The data
Seven months of New York taxi demand.
A single number every 30 minutes, plus five events everyone agrees were unusual.
NAB nyc_taxi (Numenta Anomaly Benchmark)
Taxi passenger counts from the NYC Taxi & Limousine Commission, aggregated every 30 minutes and curated by Numenta for its anomaly benchmark, with labelled event windows.
10,320
raw points
5,160
hours
2014-07-01
from
2015-01-31
to
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| timestamp | datetime | Start of each 30-minute window |
| value | number | Taxi passengers in that window |
| combined_windows.json | labels | Five known events (marathon, holidays, blizzard), used only to score the detector |
| hourly value | derived | Two half-hours summed into one hour |
| expected | derived | Median demand for that weekday and hour |
| z | derived | How unusual the hour is (robust z-score, 3-hour smoothed) |
Explore the data
Demand has a rhythm. Anomalies break it.
Weekdays peak around 19:00, weekends around 0:00 (late nights).
The expected value the detector compares against. Darker means busier.
Most hours sit near 0. Only 229 of 5,160 hours (4.4%) cross ±4.5.
Try it
Jump to a moment, then tune the alarm.
Click a row to pin that hour on the chart. Then move the threshold.
Real rows from the dataset
click a row to try it in the demo ↓
| Hour | Passengers | Expected | z | |z| > 4.5 |
|---|---|---|---|---|
| Fri, Jul 4, 8 AM | 10,941 | 38,959 | -11.50 | alert |
| Sat, Jul 5, 12 AM | 33,765 | 48,974 | -7.19 | alert |
| Mon, Aug 4, 12 PM | 32,160 | 32,324 | 0.04 | normal |
| Tue, Aug 19, 2 AM | 6,734 | 6,139 | 0.11 | normal |
| Tue, Aug 26, 3 AM | 4,085 | 4,085 | -0.04 | normal |
| Mon, Sep 1, 6 AM | 6,702 | 17,946 | -5.51 | alert |
| Wed, Nov 12, 2 PM | 36,292 | 37,015 | -0.45 | normal |
| Thu, Nov 27, 7 PM | 27,788 | 49,724 | -9.25 | alert |
| Sat, Dec 27, 2 AM | 17,370 | 34,660 | -7.69 | alert |
| Thu, Jan 1, 12 AM | 51,700 | 24,420 | 10.61 | alert |
| Thu, Jan 1, 2 AM | 51,507 | 9,127 | 19.16 | alert |
| Mon, Jan 26, 10 PM | 2,649 | 35,948 | -14.45 | alert |
Live demo
runs in your browser
Lower = more sensitive. Higher = fewer false alarms.
5/5
known events found
13
false alerts
Taxi passengers per hour, New York
How well it works
Choosing the alarm level.
All 5 known events are found up to |z| 4.5; from 5 one is missed.
Each labelled window and whether the chosen threshold catches it.
- NYC Marathonpeak z 4.9caught
- Thanksgivingpeak z -10.5caught
- Christmaspeak z -14.1caught
- New Yearpeak z 19.2caught
- Blizzardpeak z -15.8caught
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
nyc_taxi() from ml/train.py + 2 helpers (r, save)
pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load and resample to hours
Reads the NAB taxi series, sums the half-hourly counts into hourly totals, and tags each hour with its weekday and hour of day.
ml/train.py:393 · 6 lines
print("nyc-taxi-anomaly") df = pd.read_csv(RAW / "nyc_taxi" / "nyc_taxi.csv", parse_dates=["timestamp"]) s = df.set_index("timestamp")["value"].resample("1h").sum() hourly = s.to_frame("value") hourly["dow"] = hourly.index.day_of_week hourly["hour"] = hourly.index.hour - 2
Build a baseline and z-score
Expected demand is the median for that weekday and hour; the gap from it is scaled by the MAD into a robust z-score and smoothed over 3 hours.
ml/train.py:401 · 6 lines
# Expected demand for that weekday-hour, robust to the anomalies themselves (median). baseline = hourly.groupby(["dow", "hour"])["value"].transform("median") resid = hourly["value"] - baseline mad = np.median(np.abs(resid - np.median(resid))) z = (resid / (1.4826 * mad)).rolling(3, center=True, min_periods=1).mean() hourly["baseline"], hourly["z"] = baseline, z - 3
Score alerts against known events
Loads the five labelled event windows and defines how a threshold is judged: events caught versus false alerts, with flags within 6 hours grouped into one.
ml/train.py:409 · 14 lines
windows = json.loads((RAW / "nyc_taxi" / "combined_windows.json").read_text())["realKnownCause/nyc_taxi.csv"] windows = [[pd.Timestamp(a), pd.Timestamp(b)] for a, b in windows] names = ["NYC Marathon", "Thanksgiving", "Christmas", "New Year", "Blizzard"] def evaluate(threshold: float) -> dict: flags = hourly.index[np.abs(hourly["z"]) > threshold] hit = [any((t >= a) & (t <= b) for t in flags) for a, b in windows] outside = [t for t in flags if not any(a <= t <= b for a, b in windows)] events, last = 0, None # group flags closer than 6h into one "alert" for t in outside: if last is None or (t - last) > pd.Timedelta(hours=6): events += 1 last = t return {"threshold": threshold, "detected": int(sum(hit)), "of": len(windows), "false_alerts": events} - 4
Sweep the alert threshold
Every z threshold from 2.0 to 6.0 in steps of 0.5 is scored this way, and 4.5 is set as the page's default.
ml/train.py:425 · 4 lines
sweep = [evaluate(t) for t in [2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0]] default = 4.5 for e in sweep: print(f" z>{e['threshold']}: detected {e['detected']}/{e['of']}, false alerts {e['false_alerts']}") - 5
Pick demo hours
The 5 biggest moments inside known events (one per day), the 3 biggest outside them, and 4 ordinary hours with a z-score under 0.5.
ml/train.py:431 · 19 lines
# sample rows: biggest moments inside known events, a few outside, a few ordinary hours inside = hourly[[any(a <= t <= b for a, b in windows) for t in hourly.index]] outside = hourly.drop(inside.index) pick = pd.concat( [ inside.reindex(inside["z"].abs().sort_values(ascending=False).index).groupby(inside.index.date).head(1).head(5), outside.reindex(outside["z"].abs().sort_values(ascending=False).index).groupby(outside.index.date).head(1).head(3), outside[outside["z"].abs() < 0.5].sample(4, random_state=SEED), ] ).sort_index() samples = [ { "time": str(t), "value": int(row.value), "expected": int(row.baseline), "z": r(row.z, 2), } for t, row in pick.iterrows() ] - 6
Export everything the page needs
Daily profiles, the weekday-hour heatmap, z histogram, threshold sweep and the full hourly series, written to src/data/models/nyc-taxi-anomaly.json.
ml/train.py:452 · 36 lines
weekday = hourly[hourly["dow"] < 5].groupby("hour")["value"].mean() weekend = hourly[hourly["dow"] >= 5].groupby("hour")["value"].mean() heat = hourly.groupby(["dow", "hour"])["value"].median().unstack() zb = np.arange(-8, 8.5, 0.5) zcounts = np.histogram(hourly["z"].clip(-8, 8), bins=zb)[0] save( "nyc-taxi-anomaly", { "dataset": { "name": "NAB nyc_taxi (Numenta Anomaly Benchmark)", "source": "https://github.com/numenta/NAB", "points_raw": int(len(df)), "hours": int(len(hourly)), "resolution": "30 min, aggregated to hourly", "period": [str(hourly.index.min().date()), str(hourly.index.max().date())],
In your browser · TypeScript
- 7
Re-score the alerts in the browser
The same evaluation as in Python: flag every hour beyond the threshold, count known events caught and group the false alerts.
src/components/demos/anomaly-demo.tsx:27 · 15 lines
// Same evaluation as ml/train.py: known events hit + false alerts grouped by 6h gaps. function evaluate(th: number) { const flags: number[] = []; for (let i = 0; i < N; i++) if (Math.abs(zs[i]) > th) flags.push(i); const inside = (i: number) => windows.some((w) => i >= w.a && i <= w.b); const hit = windows.map((w) => flags.some((i) => i >= w.a && i <= w.b)); let alerts = 0; let last = -Infinity; for (const i of flags) { if (inside(i)) continue; if (i - last > 6) alerts++; last = i; } return { flags, hit, alerts }; }
Notes
The decisions behind it.
Approach
- 1
Built the expected demand for each weekday-hour with the median, so outliers don't bend the baseline.
- 2
Measured how far each hour sits from it with a robust z-score (MAD), smoothed over 3 hours.
- 3
Swept the threshold and counted known events found versus false alerts.
Threshold sweep
Honest note
The labelled windows span several days, which makes detection easier. Most of the "false alerts" land on July 4th weekend, Labor Day weekend and MLK Day: real holidays the benchmark simply doesn't label. The slider shows the trade-off.
Reproduce: ml/train.py → nyc_taxi()
dataset sourceNext project
Credit Risk