8 parts · 9 chapters
Data Engineering Basics
What a backend engineer needs to know about where the data goes after the transaction commits. Every company eventually asks questions the production database cannot answer cheaply; data engineering is the plumbing that answers them without hurting production.
Eight parts: OLTP versus OLAP; columnar storage measured with DuckDB (10 million rows: 862 MB of CSV became 88 MB of Parquet, and an aggregate dropped from 345 ms to 24 ms); ETL and ELT; warehouses and lakehouses; dimensional modelling; orchestration with Airflow and Dagster; data quality and data contracts; and change data capture from Postgres into analytics.
two worldsTransactions in rows, analytics in columns.
formatsParquet, ClickHouse, DuckDB and why columns compress.
pipelinesExtract, load, transform with dbt; orchestrated and tested.
storageWarehouses, lakes, lakehouses and open table formats.
modelsFacts, dimensions and slowly changing history.
freshnessCDC streams production changes into analytics in seconds.
00
OLTP versus OLAP
Two workloads, two shapes
1 ch · ~8 min01Columnar Storage: Parquet, ClickHouse, DuckDB
Measured with DuckDB · ClickHouse in one table
2 ch · ~12 min02ETL and ELT
Load raw, transform with dbt
1 ch · ~8 min03Warehouses and Lakehouses
Warehouses, lakes, lakehouses
1 ch · ~8 min04Dimensional Modelling
Facts, dimensions and grain
1 ch · ~8 min05Orchestration: Airflow and Dagster
Pipelines as code
1 ch · ~8 min06Data Quality and Data Contracts
Tests, reconciliation, contracts
1 ch · ~8 min07CDC into Analytics
From the WAL to the warehouse
1 ch · ~8 minBuilt on Postgres, Kafka and LedgersUses Postgres for the source, Kafka for CDC transport and Ledgers for the numbers that must reconcile.