Round four: “six teams want this data”

The real question underneath: how do you let the company build on the ledger without every new consumer becoming a change to the ledger?

Dispose of the obvious approach properly, because being able to say precisely why it fails is what justifies the infrastructure that replaces it.

// the design that seems reasonable and is not
await db.postTransfer(...);
await notifications.send(...);   // +40 ms
await fraud.score(...);          // +60 ms
await analytics.record(...);      // +30 ms
await loans.notify(...);          // +50 ms
return ok;                       // 180 ms of other people's latency

The three fatal properties

1. latency adds

              transfer p99 = ledger p99 + Σ(every consumer's p99)

              our 300 ms budget is gone by the fourth callee


2. availability multiplies

              6 dependencies at 99.9% each

              = 0.9996 = 99.4% → 52 hours of downtime a year

              the ledger becomes less available than its least available consumer


3. partial failure has no good answer

              money posted, notification failed. now what?

              roll back a committed posting? no.

              fail the request after taking the money? no.

              ignore it? then the event is lost.

And a fourth, which is organisational and in practice the most damaging: every new consumer means a code change, a review, and a deployment of the ledger. The most safety-critical service in the company becomes the one that changes most often, purely for reasons that have nothing to do with money.

The one case that must stay synchronous

Fraud, sometimes. If the business requires blocking a transaction before it posts, no amount of asynchrony helps: you need an in-line decision. The resolution is to split fraud into two paths, a fast synchronous check on the posting path with a hard timeout and a fail-open default, and a rich asynchronous analysis that consumes events. Part 9 builds exactly that.

The abstraction is older and simpler than the products built on it: an append-only, ordered, durable, replayable sequence of records, where each consumer tracks its own position independently.

Why this is not a queue

A distinction worth being crisp about, because interviewers probe it:

Our problem is one fact with many interested parties, and we want replay. That is a log. Part 6 will use an actual queue for outbound payout work, which is the other shape, and using both for their respective strengths is the correct answer rather than a compromise.

The event, designed

{
  "event_id":   "01JBQ2X...",        // ULID: unique AND sortable
  "event_type": "ledger.entry.posted",
  "version":    2,                    // schema version, always present
  "occurred_at": "2026-03-14T09:12:03.441Z",

  // the facts. a consumer must not need to call us back to understand this.
  "journal_id": "j-8f1a",
  "entry_id":   998812,
  "account_id": "acct-771",
  "amount":     "-50000",            // string: JSON numbers lose precision
  "currency":   "NGN",
  "kind":       "transfer",
  "balance_after": "75075",        // saves every consumer a query

  // provenance, so a consumer can trace and deduplicate
  "correlation_id": "req-4c2e",
  "shard": 7
}

Enough internals to justify the configuration choices rather than copying them.

What a partition physically is

A partition is a directory on a broker's disk, containing segment files. Each segment is an append-only sequence of records, with index files mapping offsets to byte positions.

          /var/lib/kafka/ledger.entry.posted-7/

            00000000000000000000.log   ← records, appended

            00000000000000000000.index ← offset → byte position

            00000000000000000000.timeindex

            00000000000524288000.log   ← next segment, rolled at 1 GB


          a write is an append to the active segment: a sequential write.

          a read is a seek via the index, then a sequential scan.


this is why Kafka is fast. it is not clever, it is sequential.

Two further mechanisms carry most of the performance: the page cache, since recently written records are served from memory without Kafka managing a cache itself, and zero-copy transfer via sendfile, which moves bytes from page cache to socket without passing through the application at all.

Replication and the in-sync replica set

Kafka guarantees total order within a partition and nothing across partitions. Every ordering decision follows from that one sentence.

1 partition   → global total order, one consumer maximum,
          ~10 MB/s ceiling

N partitions → order only within each, N consumers in
          parallel


          so: ordering and parallelism are the same dial

          the skill is choosing the smallest unit that needs ordering

What actually needs ordering in a bank

So per-account ordering is the requirement, which means account_id is the partition key. And one subtlety worth knowing: ordering also depends on producer configuration.

// max.in.flight.requests.per.connection > 1 with retries CAN reorder:
// request A fails and is retried while request B has already succeeded.
//
// enable.idempotence=true fixes this: the broker uses producer sequence
// numbers to reject out-of-order batches, so ordering survives retries
// even with up to 5 requests in flight. keep idempotence on.
enable.idempotence = true
max.in.flight.requests.per.connection = 5   // safe WITH idempotence
retries = Integer.MAX_VALUE
delivery.timeout.ms = 120000                // bound total attempt time

Same reasoning as the shard key in Part 3, different consequences, and worth doing explicitly because the right answer differs.

The hot partition, which is the same problem as the hot row

The fee revenue account appears on most transactions, so keying by account_id sends a large fraction of all events to one partition. The same structural fix applies, for the same reason:

// the fee account was already split into 64 sub-accounts in Part 3 to
// solve the hot row. that split solves the hot partition for free,
// because the sub-account id is what gets hashed.
function partitionKey(e: LedgerEvent): string {
  return e.account_id;   // 'fee_revenue:41' spreads; 'fee_revenue' would not
}

Sizing the partition count

          peak events/s = 464 transfers × ~3.2 entries ≈ 1,500 events/s

          allow 4x growth                       = 6,000 events/s


          one consumer instance handles ~500 events/s (with database work)

          → need 12 parallel consumers at peak


          partitions must be ≥ desired consumers, and cannot be reduced later

          → choose 48 partitions: 4x current need, room to grow consumers


why not 1,000? each partition costs file handles, memory,

          and rebalance time. more partitions also means more end-to-end latency

          for replication fan-out.
        

Here is the bug that makes naive event publishing unacceptable for money. It is the most important thing in this part.

// this code is wrong, and the wrongness is not visible by reading it
await db.postTransfer(...);            // 1. commits to Postgres
await kafka.publish(entryPosted);      // 2. publishes to Kafka

Two independent systems, two separate commits, no transaction spanning them. There is a window between them, and a crash in that window leaves the two permanently disagreeing.

The four outcomes, two of which are bugs

Reordering the two statements does not help, it only changes which bug you get. Wrapping them in a try/catch does not help, because the process can die between them. Retrying the publish does not help, because the retry state itself lives in the memory that just died.

the general statement


          you cannot atomically commit to two systems that do not

          participate in a shared transaction.


therefore: write to ONE system, and derive the other from it.


          this is the entire idea behind both solutions that follow.
        

The fix follows directly from the general statement: write to one system. The event is committed into the database, in the same transaction as the money.

CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,   -- monotonic: preserves order
  aggregate_id TEXT NOT NULL,           -- becomes the partition key
  event_type   TEXT NOT NULL,
  payload      JSONB NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ                -- null until relayed
);

-- partial index: only unpublished rows, so it stays tiny regardless of volume
CREATE INDEX outbox_pending ON outbox (id) WHERE published_at IS NULL;

-- one transaction. the money and the intent to announce it are inseparable.
BEGIN;
  INSERT INTO journal ...;
  INSERT INTO entries ...;   -- debit
  INSERT INTO entries ...;   -- credit
  INSERT INTO outbox (aggregate_id, event_type, payload)
       VALUES ($account, 'ledger.entry.posted', $json);
COMMIT;
-- either all four rows exist, or none do. the window is gone.

The relay

// a separate process, running continuously. its only job is to move rows
// from the outbox to Kafka. it may crash freely; nothing is lost.
async function relay() {
  while (running) {
    const batch = await db.query(`
      SELECT id, aggregate_id, event_type, payload FROM outbox
       WHERE published_at IS NULL
       ORDER BY id                 -- order preserved: BIGSERIAL is monotonic
       LIMIT 500
       FOR UPDATE SKIP LOCKED      -- many relay instances, disjoint work
    `);
    if (!batch.length) { await sleep(50); continue; }

    // publish first, mark second. a crash between them causes a
    // DUPLICATE, never a loss. duplicates are handled by consumers.
    await kafka.sendBatch(batch.map(toRecord));
    await db.query(`UPDATE outbox SET published_at = now() WHERE id = ANY($1)`,
                   [batch.map(r => r.id)]);
  }
}

Costs, honestly

Change data capture reads the database's own replication log and turns committed changes into events. Same guarantee, obtained differently, and you should be able to argue both sides.

          Postgres logical replication → Debezium → Kafka


          the WAL already is an ordered, durable log of every commit.

          CDC simply reads it. no application change at all.
        

A consumer group is a set of instances that share a topic's partitions. Each partition is assigned to exactly one member, which is what gives both parallelism and per-partition ordering.

          48 partitions, 12 consumer instances → 4 partitions each

          one instance dies → its 4 are reassigned → 11 instances, ~4.4 each


          more consumers than partitions → the extras sit idle

partition count is the hard ceiling on parallelism

Rebalancing, and why it can become a storm

When membership changes, the group must reassign partitions. Under the default eager protocol, every consumer stops consuming during the reassignment. A poorly tuned deployment turns that into a cascade:

// the settings that prevent it
max.poll.records = 100            // small batches: bounded time per poll cycle
max.poll.interval.ms = 300000     // generous ceiling before eviction
session.timeout.ms = 45000        // heartbeat failure detection
heartbeat.interval.ms = 3000      // heartbeats run on a separate thread

// and the important structural one: cooperative rebalancing, which
// revokes only the partitions that must move instead of stopping everyone.
partition.assignment.strategy = CooperativeStickyAssignor

Lag as the primary health metric

          lag = log end offset − committed consumer offset


          lag in records is what tools report.

          lag in seconds is what the business cares about:

              time_lag ≈ record_lag ÷ consumption_rate


alert on time lag, not record lag. 50,000 records behind

          at 10,000/s is 5 seconds and fine. at 100/s it is 8 minutes and an incident.
        

Our relay deliberately produces duplicates rather than losses, and Kafka delivers at least once. So every consumer must be safe to run twice on the same event. That is a requirement on consumers, which the platform team must make easy to satisfy.

Three ways to be idempotent, in order of preference

// the standard shape: marker and effect in one transaction
async function handle(e: LedgerEvent) {
  await db.transaction(async tx => {
    try {
      await tx.query(
        `INSERT INTO processed_events (consumer, event_id) VALUES ($1, $2)`,
        ['notifications', e.event_id]
      );
    } catch (err) {
      if (isUniqueViolation(err)) return;   // already done. skip.
      throw err;
    }
    await doTheWork(tx, e);   // same transaction: cannot diverge
  });
}

Keeping the dedup store bounded

-- monthly partitions, so cleanup is a DROP rather than a mass DELETE.
-- retention only needs to exceed the topic's retention.
CREATE TABLE processed_events (
  consumer TEXT NOT NULL,
  event_id TEXT NOT NULL,
  seen_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, event_id)
) PARTITION BY RANGE (seen_at);
-- DROP TABLE processed_events_2026_01;  ← instant, no vacuum pressure

The event schema is now a public API consumed by six teams and, because of replay, by historical versions of those consumers. It must evolve without coordinated deployments.

Format choice

The topic layout

The compacted topic deserves a note: log compaction retains only the latest record per key, so a new consumer can rebuild current account state by reading it once rather than replaying all history. It gives you a snapshot as a stream.

What changed, and the cost accepted

Which is the opening the interviewer takes next, because a bank that cannot lend is not much of a bank.