Data Engineering · Architectures
Six ways to move data, drawn end to end.
The same job, getting data from where it is made to where it is used, solved six ways. Each one is animated the way its data actually moves, with one record followed through every hop. The first is the real setup of this site.
01 · The gallery
Pick an architecture, then follow a record.
Watch how the data moves: in scheduled chunks, as a continuous stream, as change events from a database log, or out of the warehouse into other tools. Then step through one record and see its shape at every hop.
Reference design · example records
Batch ELT on GCP
The classic warehouse: load everything on a schedule, transform inside BigQuery, refresh the dashboards.
Follow one record
7 hops from source to use
Press play to follow one record through the whole architecture, or pick a step. The diagram dims everything except where the record is.
- 1An order is written at 14:05
- 202:00: the scheduler starts the run
- 3Only what changed since the last run
- 4Landed as files, never edited
- 5Loaded into BigQuery
- 6Transformed and tested in SQL
- 706:00: the manager sees yesterday
Freshness
Running cost
Build effort
Replay & backfill
Best for
- ✓Reporting, finance and KPI dashboards
- ✓Teams that know SQL
- ✓The first data platform most companies build
Watch out for
- !Stale by design: yesterday's numbers
- !Full reloads get slow and expensive: go incremental
- !Upstream schema changes break loads
02 · Side by side
Faster always costs something.
| Architecture | Typical latency | Freshness | Running cost | Build effort | Replay & backfill | Best for |
|---|---|---|---|---|---|---|
| This sitereal | history: when I re-run it · live tiles: seconds | Small data that fits in memory | ||||
| Batch ELT on GCP | hours (daily or hourly runs) | Reporting, finance and KPI dashboards | ||||
| Streaming | seconds | Fraud checks, monitoring, live operations | ||||
| Lakehouse | minutes to hours | BI and machine learning on the same data | ||||
| Change data capture | seconds to minutes | Copying app databases to analytics without nightly dumps | ||||
| Reverse ETL | minutes to hours | Putting scores and segments where teams work |
Dots are rules of thumb from practice, not benchmarks: a well-run streaming job can be cheap, and a careless batch job expensive. Click a row to open it in the gallery. Real platforms usually combine two or three: CDC feeding a lakehouse, batch building marts, reverse ETL pushing scores back out.
03 · Which one fits?
Four questions, one starting point.
Answer what you know about your data and who needs it. The recommendation explains itself and updates as you change answers.
01How fresh must the data be?
02Where does the data come from?
03Who uses the result?
04How big is it?
Your fit
Answer a question or two.
The scores update with every answer, and you can change your mind.