CDC into Analytics
Change data capture from Postgres logical decoding, replication slots and their disk risk, Debezium event format, snapshots plus streaming, deletes and tombstones, schema changes in flight, sinks into warehouses and ClickHouse, the outbox as CDC for domain events, and reconciling CDC output against the source.
From the WAL to the warehouse
-- Postgres: enable logical decoding and a publication ALTER SYSTEM SET wal_level = 'logical'; -- restart required CREATE PUBLICATION analytics FOR TABLE transfers, accounts; ALTER SYSTEM SET max_slot_wal_keep_size = '50GB'; -- a stalled consumer cannot fill the disk -- watch slot lag SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag FROM pg_replication_slots; # Debezium connector (excerpt) connector.class=io.debezium.connector.postgresql.PostgresConnector plugin.name=pgoutput publication.name=analytics snapshot.mode=initial # copy existing rows first, then stream changes from the slot position topic.prefix=core
Deletes arrive as events with before images and op: d (plus a tombstone for compacted topics); the sink must apply them or analytics will count deleted rows forever. Schema changes flow through as new event shapes, which is why data contracts (previous part) matter. Domain events vs table CDC: table CDC exposes your internal schema; the outbox pattern (Workflows P8) publishes intentional events instead, often through the same Debezium pipeline.