01
Replayable
Raw data is kept untouched, so any table can be rebuilt after a bug without asking the source again.
Replay from bronze in the simulatorSkill 04 · Data Engineering
Getting data out of systems and APIs, landing it safely, checking it, shaping it and serving it, on a schedule, without anyone watching. Shown here with pipelines that really run.
The pipeline on this page, in numbers
5
API requests
42,600
hourly slots landed
7
checks on every load
37,409
clean hours kept
1,559
daily averages
2
pages fed
A pipeline that never sleeps
The real pipeline behind this site's air pages, as one picture. Every square is five days of real hourly data. Scroll and watch it get fetched, landed, checked, cleaned, summarised and served.
How I build them
What separates a script that worked once from a pipeline that can run every hour for years.
01
Raw data is kept untouched, so any table can be rebuilt after a bug without asking the source again.
Replay from bronze in the simulator02
Run a job twice and get the same table: loads MERGE on a key instead of appending.
Duplicates, absorbed03
Checks run before anything is published. A failing batch is held back, never half-published.
Live checks on three APIs04
Freshness and volume are watched on every run, and someone hears about it when they drift.
Break the data, watch it caughtExplore
Pipeline lab
Wikipedia edits processed live as a stream, live data-quality checks, and every pipeline that feeds this site, measured.
Incident simulator
Break a pipeline on purpose (API down, schema change, duplicates, late data) and compare a naive copy with a guarded one, hour by hour.
Architectures
Batch, streaming, lakehouse, CDC and reverse ETL, animated end to end with one record followed through each, plus the real setup of this site.
04Work case · pipelines and master data from real work, anonymised.
coming next