←BaldanAll skillsLet's talk

Data Engineering · Incident simulator

Break the pipeline. Watch it recover.

Pipelines rarely fail loudly. More often they keep running and quietly publish something wrong. Here two copies of the same hourly pipeline replay real Jakarta air-quality hours through the same four incidents: one without guardrails, one with.

01 · The simulator

Same data, same incidents, two pipelines.

Press play: one run per simulated hour, four scheduled incidents. Or inject your own at any moment, and drag the timeline to rewind. Watch what readers see on each dashboard, and the postmortem underneath.

Simulated clock · real Jakarta PM2.5 hours

Ready

Both copies share 48 correct hours of history. Press play.

7 Nov 00:00drag to rewind or skip ahead9 Nov 23:00

Without guardrails

The naive pipeline

Fetch the newest hour · append · missing field = 0 · drop empty values · no checks, no alerts

Extract

ok

Validate

no checks

Load

ok

Serve

ok

What readers see

PM2.5 now

54.1

µg/m³

24-hour average

44.8

under Indonesia's 24-h standard

truth 44.8 (under standard) · matches

010020000:0012:0000:0012:00
stored hourtruthmissing for goodwrong (0)counted twice

0

hours missing

0

hours wrong

0

rows counted twice

0

alerts sent

run log

waiting for the first run

With guardrails

The guarded pipeline

Fetch since the watermark (re-read 5 h) · 3 retries · data contract · quarantine + bronze · MERGE on hour · alerts

Extract

ok

Validate

ok

Load

ok

Serve

ok

What readers see

PM2.5 now

54.1

as of 23:00

24-hour average

44.8

under Indonesia's 24-h standard

truth 44.8 (under standard) · matches

010020000:0012:0000:0012:00
stored hourtruthmissing, will be filledwrong (0)counted twice

0

hours waiting

0

hours wrong

0

rows counted twice

0

alerts sent

run log

waiting for the first run

Postmortem

Updates as the clock runs. Damage counts the hours each incident touched, as stored right now.

No incident yet. Press play, or inject one yourself.

The values are real hourly PM2.5 for Jakarta from Open-Meteo (CAMS model estimates, CC BY 4.0), 5 Nov to 9 Nov 2026. The API, the incidents and both pipelines are simulated in your browser.

02 · The guardrails

Four habits that turn incidents into non-events.

None of these is exotic. Together they make a pipeline that says when it is late, refuses to publish what it does not understand, can be re-run safely, and fixes its own gaps.

API down

Retry, alert, backfill from the watermark

  • Retry with growing waits: a blip should not page anyone
  • Alert on the first run that still fails, and say “delayed” on the dashboard
  • Load from the last good hour, not from “now”, so a gap fills itself

python · simplified pattern

def extract(since):
    for wait in (1, 2, 4):
        try:
            return get(API, start=since, end=now())
        except HTTPError:
            sleep(wait)
    alert("extract failing")
    raise

since = min(watermark + HOUR, now() - 5 * HOUR)
batch = extract(since)

Schema change

A data contract, quarantine and replay

  • Write down what a batch must look like, and check every one
  • A failing batch is held back, never half-published
  • The raw batch stays in bronze, so the fix replays it without asking the source again

python · simplified pattern

CONTRACT = {"time": str, "pm2_5": float}

def validate(batch):
    missing = CONTRACT.keys() - batch.keys()
    if missing:
        quarantine(batch)   # raw copy stays in bronze
        alert(f"contract: {missing} missing")
        return False
    return True

Duplicates

Idempotent loads: MERGE on the key

  • Every row has a key (here, the hour)
  • Loading the same batch twice leaves the table unchanged
  • Safe to retry any run, any time

sql · simplified pattern

MERGE silver.pm25 AS s
USING (SELECT DISTINCT hour, pm2_5
       FROM batch) AS b
ON s.hour = b.hour
WHEN MATCHED THEN
  UPDATE SET pm2_5 = b.pm2_5
WHEN NOT MATCHED THEN
  INSERT (hour, pm2_5)
  VALUES (b.hour, b.pm2_5)

Late data

A lookback window and partition rebuilds

  • Re-read the recent past on every run, not just the newest hour
  • An empty value is “not yet”, not “none”
  • When an old hour changes, rebuild the day it belongs to

python · simplified pattern

batch = extract(since=now() - 5 * HOUR)
merge(batch.dropna())       # empty hours stay open

for day in batch.days_changed():
    rebuild_gold(day)       # the daily average is redone

In this portfolio's own PM2.5 pipeline (ml/live_data.py) the retry with growing waits is real code, and every re-run replaces its output instead of appending to it. The contract, quarantine, watermark and MERGE are how the same job would run in production, for example on BigQuery with a scheduler.