Part 3 · 1 chapters · ~8 min

Warehouses and Lakehouses

Cloud data warehouses (BigQuery, Snowflake, Redshift), data lakes on object storage, lakehouses with open table formats (Iceberg, Delta Lake, Hudi), catalogues, separation of storage and compute, partitioning and clustering, cost models, and choosing for a small company versus a large one.

Warehouses, lakes, lakehouses

	warehouse	lake	lakehouse
storage	managed, proprietary format	files in object storage	open files + table format
transactions and schema	yes	no (files only)	yes (Iceberg, Delta)
engines	the vendor's	any, but no guarantees	many, on the same tables
good for	SQL analytics, BI, simple operations	raw, semi-structured, ML data	both, at scale, without lock-in

Choosing: a small company is usually best served by a managed warehouse (or ClickHouse, or even DuckDB on Parquet in object storage) plus dbt. Lakehouses pay off with large volumes, several engines, and ML workloads. Cost models differ: BigQuery on-demand bills by bytes scanned, so partitioning and selecting few columns cut the bill directly.

code

-- BigQuery: partition and cluster so queries scan less (and cost less)
CREATE TABLE analytics.transfers PARTITION BY DATE(created_at) CLUSTER BY account_id AS SELECT * FROM staging.transfers;

A LAKEHOUSE, LAYER BY LAYER

open files, a table format, and engines on top

swipe the figure sideways, or tap expand for full screen

1/4

object storage

Data lives in cheap object storage, separate from compute, so storage and compute scale and bill independently.

storage separated from computecheap and durable