←BaldanAll skillsLet's talk

Data Engineering · Pipeline lab

Pipelines you can watch run.

Nothing here is a recording. A streaming pipeline you start yourself, data-quality checks that run against live APIs when you open the page, and the real batch pipelines that build this site, with their measured run times.

01 · Live stream

Wikipedia, processed as a stream.

Press start and this page connects to Wikimedia's public event stream and runs a small streaming pipeline in your browser: parse, filter, a privacy step, 10-second event-time windows with a watermark, and a side output for late events.

Wikimedia EventStreams · recentchange

Every change on every Wikimedia wiki, as it happens.

not connected

events processed

0

events per second

–

event data received

0 KB

runs for

2:00

Connects only when you press start, and pauses itself after two minutes or when you switch tabs: the stream is busy, so it uses data.

01

Source

Server-sent events over HTTPS, one JSON event per change

waiting

02

Parse & filter

Parse JSON; drop malformed events and Wikimedia's canary test events

waiting

03

Privacy map

Keep 5 of up to 25 fields; titles, usernames and comments are dropped

wiki · type · bot · size · time

04

Window

10-second tumbling windows by event time, 2 s allowed lateness

watermark –

05

Late → side output

Events for a window that already closed: counted apart, never rewrite a number

waiting

06

Aggregate → sink

Counts, bytes and busiest wikis per window, drawn below

waiting

Changes per 10-second window

By event time (when the change happened), people and bots stacked, last five minutes. Faded bars are windows still open.

Allowed lateness
01020−5 min−4 min−3 min−2 min−1 minnewestPress start: windows fill as edits arrive.
people bots late events for that window

The watermark trails the newest event time by the allowed lateness. When it passes the end of a window, that window closes and its numbers are final; anything that turns up later for it goes to the side output. More lateness means fewer late events, but every number is published later.

Freshness

Seconds from the change to your screen, last 500 events

Waiting for events.

Busiest wikis, last 60 seconds

By event time, from the windows above

The busiest wikis show up here.

Who changes what

Since start, people and bots by kind of change

People and bots, by kind of change.

Languages of Indonesia

Changes on these Wikipedias since start

Indonesian0
Javanese0
Sundanese0
Minangkabau0
Balinese0
Banjar0
Acehnese0
Buginese0
Madurese0
Gorontalo0
Banyumasan0
Batak Toba0
Batak Mandailing0
Nias0

Bytes added and removed

Page size changes per window, same windows as above

+5 KB−5 KB

What the pipeline keeps

The newest records after the privacy map: five fields, nothing that names a page or a person

Records appear here as they pass the privacy step.

Source: Wikimedia EventStreams (public, no key), the recentchange stream of every Wikimedia wiki, read straight from your browser. Only counts are computed and nothing is stored or sent anywhere.

02 · Live data quality

Checks that run before anyone trusts the numbers.

Three public APIs, fetched by your browser just now, each put through the same eight checks a production pipeline runs before it publishes. They should mostly pass, so break the data on purpose and watch which checks catch it.

running the checks…

Checking three live APIs

Break the data on purpose

The data exactly as the APIs sent it. Only this page's copy is changed.

Jakarta PM2.5

Open-Meteo Air Quality (CAMS), hourly

air-quality-api.open-meteo.com

USD → IDR

Frankfurter (ECB rates), business days

api.frankfurter.dev

id.wikipedia views

Wikimedia Pageviews API, daily, people only

wikimedia.org/api/rest_v1

Sources, fetched by your browser when this page opens: Open-Meteo (CC BY 4.0, CAMS data from Copernicus), Frankfurter (European Central Bank reference rates) and the Wikimedia Pageviews API.

03 · Pipelines behind this site

Every page here is fed by a pipeline.

The dashboards, maps and models on this site read small JSON files that batch pipelines build from raw data. Here is each one, re-run and timed for this page, with what goes in and what comes out.

7

batch pipelines re-run for this page

46.0 s

for all of them, one after another

40.9 MB

of raw files read

395 KB

of JSON served, about 104× smaller

The measured run

11 November 2026 · Python 3.12.9 · pandas 3.1.1 · a laptop

Retail profiling
4.3 s
Sales dashboard ETL
36.2 s
A/B test aggregates
2.7 s
Churn de-identification
0.7 s
Air quality history
0.8 s
Map shapes
0.5 s
Pipeline profile
0.8 s
0 s10 s20 s30 s40 s

Run by ml/de_catalog.py, which starts each script and times it. The sales ETL dominates: it rebuilds every country × month metric, the cohorts and RFM from half a million rows. On a scheduler the independent ones would run side by side.

batch4.3 s

Retail profiling

ml/retail_profile.py

  1. load the raw export
  2. →profile every column
  3. →find duplicates, cancellations, odd prices
  4. →count each cleaning rule
  5. →small aggregates for demos

in

ml/data/raw/retail/online_retail.pklthe UCI Excel file, cached as a pickle

27.4 MB · 541,909 rows

out

src/data/retail-profile.json

9 KB · 3191× smaller

Feeds Data models page (Raw to ready, method library)

batch36.2 s

Sales dashboard ETL

ml/retail_dashboard.py

  1. same cleaning rules
  2. →net revenue (minus cancellations)
  3. →metrics by country × month
  4. →products, timing, cohorts, RFM
  5. →serve as JSON

in

ml/data/raw/retail/online_retail.pklthe UCI Excel file, cached as a pickle

27.4 MB · 541,909 rows

out

src/data/retail-dashboard.json

src/data/sales-dashboard.json

53 KB · 515× smaller

Feeds Overview journey, Dashboards story, sales dashboard

batch2.7 s

A/B test aggregates

ml/ab_dashboard.py

  1. fix a typo in the source
  2. →segment bands
  3. →cell sums per arm (n, visits, buys, spend, spend²)
  4. →sample-ratio and balance checks

in

ml/data/raw/hillstrom/hillstrom.csv

4.0 MB · 64,000 rows

out

src/data/ab-dashboard.json

8 KB · 490× smaller

Feeds A/B test dashboard

batch0.7 s

Churn de-identification

ml/churn_dashboard.py

  1. drop IDs, gender, age, household
  2. →encode categories as digit strings
  3. →attach the trained model

in

ml/data/raw/telco/telco.csv

970 KB · 7,043 rows

out

src/data/churn-dashboard.json

149 KB · 7× smaller

Feeds Churn dashboard

batch0.8 s

Air quality history

ml/air_dashboard.py

  1. keep whole days before the fetch
  2. →daily, monthly, hour × month
  3. →days over WHO / Indonesian limits
  4. →14-day offline snapshot

in

ml/data/raw/live/pm25_jakarta.json

1.1 MB · 42,600 rows

out

src/data/air-dashboard.json

25 KB · 44× smaller

Feeds Jakarta air dashboard

batch0.5 s

Map shapes

ml/geo_prepare.py

  1. project lon/lat to SVG
  2. →simplify rings (Douglas–Peucker)
  3. →drop specks, keep islands
  4. →sample store dots inside provinces

in

ml/data/raw/geo/idn_adm1_simplified.geojson

ml/data/raw/geo/ne_110m_countries.geojson

7.4 MB · 211 rows

out

src/data/geo-shapes.json

145 KB · 51× smaller

Feeds Real maps in the chart library

batch0.8 s

Pipeline profile

ml/de_profile.py

  1. measure every stage
  2. →run the seven checks
  3. →extract the real code

in

ml/data/raw/live/pm25_jakarta.json

1.1 MB · 42,600 rows

out

src/data/de-pipeline.json

5 KB · 225× smaller

Feeds This hub's overview story

API ingestnot re-run here

Live API fetcher

ml/live_data.py

  1. yearly or monthly requests
  2. →retries with backoff, longer on HTTP 429
  3. →skip what is already cached
  4. →filter unsafe Wikipedia titles

in

Open-Meteo · Frankfurter · Wikimedia APIs

out

ml/data/raw/live/pm25_jakarta.json

ml/data/raw/live/usd_idr.json

ml/data/raw/live/wiki_id.json

6.2 MB

Feeds Live Forecast Lab, air dashboard, this hub

model trainingnot re-run here

Model training

ml/train.py

  1. one function per project
  2. →train, compare, pick
  3. →export weights and samples as JSON
  4. →parity checks with the browser

in

public datasets in ml/data/raw

out

src/data/models/ (24 JSON files)

2.9 MB

Feeds Every ML and DL project page

The live API fetcher is not re-run for this page because it calls the network, and model training is listed with its outputs only. Every number above comes from src/data/de-catalog.json, written by the measured run.