Part 21 · 6 chapters · ~40 min

The whole board

Sixteen rounds, and every box on the whiteboard arrived because something broke. This part walks the finished design end to end, traces one payment through every component that touches it, collects every tradeoff into one table, and then closes with the twenty questions most likely to be asked about it, answered.

252

The complete architecture, walked end to end

The eight layers, and the one thing each is responsible for

Layer	Responsible for	Built in
Channels	Accepting instructions: app, POS, card network, partner API	Parts 8, 14
Authorisation tier	Deciding yes or no inside a budget, with stand-in	Parts 8, 9
Ledger	The invariant. Sole writer, sharded by account	Parts 1, 2, 3
Event backbone	Telling everyone what happened, exactly once in effect	Part 4
Product services	Product rules, expressed as balanced entries	Parts 5, 14
Connectors	Money crossing our boundary, and the unknown state	Parts 6, 7
Assurance	Proving the money is right, and catching adversaries	Parts 9, 10
Data platform	Serving history, answering questions, keeping ten years	Parts 11, 15

the sentence that frames the whole design

"Everything here exists to protect one invariant: entries sum to zero per journal per currency, and money is never created or destroyed. Sharding, suspense accounts, idempotency keys, the outbox, reconciliation and the correctness signal are all mechanisms for keeping that true while the system gets faster, bigger and more distributed. The products are the reason it exists; the invariant is the reason it can be trusted."

the whole board

built one round at a time

swipe the figure sideways, or tap expand for full screen

1/10

channels

Channels accept instructions: the app, POS terminals, the card network and partner APIs. Four different protocols, chosen by conversation shape in Part 14.

253

A payment, traced through every component

One transfer: ₦50,000 from a Nigerian customer to an account at another bank, with a ₦10 fee. Here is everything it touches, with the part that introduced each step.

t	What happens	Part
`0 ms`	App sends `POST /transfers` with a client-generated idempotency key, over REST	2, 14
`3`	Gateway terminates TLS, authorises the customer, attaches a request id	1
`6`	Payments service resolves the beneficiary: NIBSS name enquiry, cached 60 s	6
`52`	Customer confirms the returned name. Last chance on a rail with no reversal	6
`55`	Velocity limits checked in Redis, atomically, in one Lua round trip	5, 9
`58`	Fraud score computed: rules, then the boosted model. Fails open	9
`70`	Sanctions screening. Fails closed, because it is statutory	9
`74`	Ledger shard resolved from `hash(account_id)` via the shard map	3
`76`	`BEGIN`. Journal row inserted, unique idempotency key	1, 2
`78`	Debit inserted with the balance check fused into the statement, floor from the overdraft facility	1, 5, 14
`79`	Credit to `settle_suspense:nibss`, so the money is in a named place	6
`80`	Fee entries: debit customer, credit `fee_income:transfer:41`, a sharded sub-account	3, 5
`81`	Invariant asserted in-transaction, per currency	1, 7
`82`	Outbox row written in the same transaction. No dual write	4
`84`	`COMMIT`. WAL fsync, synchronous replica acknowledges	1
`91`	`202 Accepted` returned, with `balance_effective: false` stated honestly	3
`140`	Outbox relay publishes to Kafka, keyed by account id	4
`180`	Cache invalidated; the client's next read carries a watermark anyway	3
`210`	Notification consumer: dedup on event id, preferences, quiet hours, in-app written first	12
`400`	Dispatcher calls NIBSS, outside any transaction, with our journal id as the reference	6
`2.1 s`	NIBSS returns accepted. State becomes `awaiting_confirmation`	6
`4 s`	Bigtable updated, so the app's history screen shows it	11
`30 s`	Continuous invariant check passes over this journal	10
`8 min`	CDC lands the rows in bronze; silver conforms them	11
`2 h`	NIBSS confirms settlement. Suspense cleared to the nostro account	6
`T+1`	Reconciled against the NIBSS settlement report, matched on our reference in pass 1	10
`T+1`	Included in the close, trial balance signed off, figures frozen	10
`90 d`	Partition ages to warm storage	15
`3 y`	Archived to Parquet, manifest records the per-currency sums	15
`7 y`	Retention expires. Still readable if a dispute requires it	15

the thing to notice in that table

91 milliseconds of synchronous work, and seven years of consequences. The customer's experience is one response; the system's obligations run for the statutory retention period. Most of the design is not about the 91 milliseconds, and being able to walk both timescales in one answer is what a staff-level response looks like.

254

Every major tradeoff, in one table

Decision	Chosen	Cost accepted	Would change if…
Balance representation	Derived from entries	O(n) reads, needing snapshots	Never. This is the foundation
Money type	Integer minor units	Exponent handling everywhere	Never
Ledger database	Postgres	Sharding needed for scale	DynamoDB at a DynamoDB-first shop, accepting a Streams-based invariant check
Consistency on the money path	Strong, CP	Availability during partitions	Never for posting. Authorisation got its own answer
Isolation level	READ COMMITTED plus conditional writes	Write skew needs explicit handling	SERIALIZABLE for transactions with multi-row invariants
Shard key	`hash(account_id)`	Cross-shard transfers need a saga	`customer_id` if intra-customer transfers dominated
Cross-shard atomicity	Saga with suspense	A brief non-atomic window	Co-locate related accounts if true atomicity were required
Event publishing	Transactional outbox	An extra write and a relay to run	CDC where the consumer wants table shape, as analytics does
Delivery semantics	At-least-once + idempotent handlers	Consumers must all deduplicate	Never. Exactly-once delivery does not exist
Fraud on the sync path	Fails open	Some fraud passes during an outage	Never. A fraud outage must not be a payments outage
Sanctions screening	Fails closed	Cross-border payments queue	Never. It is a statutory obligation
Card authorisation availability	Stand-in with caps	Bounded, quantified overdraw	Tighter caps if losses exceeded the model
Failover after a timeout	Forbidden for payments	Slower resolution of unknowns	Permitted for notifications, where a duplicate costs ₦4
Analytics location	Fully isolated	Minutes of lag	Never. It is an availability decision
Region boundary	Legal, not technical	No cross-region failover	Never. It is not ours to change
Correction mechanism	New linked entries	Net effect is a recursive query	Never. Editability destroys the audit property

how to use this table in a room

The fourth column is the valuable one. "I would change this if Z" turns a preference into a judgement, and it pre-empts the interviewer's follow-up. Note also how many say never: those are the decisions that are not tradeoffs at all but properties of the domain, and knowing which is which is most of the skill.

255

What we deliberately did not build

Naming your own omissions before they are found is one of the strongest moves available, because it demonstrates that the scope was chosen rather than reached.

Not built	Why it was excluded	What it would need
KYC and onboarding	A large domain of its own, mostly orthogonal to the ledger	Document capture, liveness, bureau checks, tiering
Authentication and sessions	Solved generically, with nothing bank-specific in the architecture	OAuth, device binding, step-up flows
Customer support tooling	Consumes the same events and data; adds no new structure	Case management, a read model, strict access control
The mobile app	A client of the APIs we designed	Offline handling, push registration, secure storage
Treasury and liquidity management	Genuinely important and largely a business function	Position limits, hedging execution, cash forecasting
Credit scoring models	A data science problem consuming our events	Bureau integration, feature engineering, model governance
Merchant acquiring	The other side of Part 8, and a full product line	Terminal estate, merchant onboarding, settlement
Multi-tenancy for BaaS	Would change isolation everywhere	Tenant in every key, per-tenant limits and reporting

and the things that would genuinely be hard to add later

Multi-tenancy. Retrofitting a tenant dimension into a sharded ledger is a migration of every row and every query. If BaaS were plausible, it belongs in round one.
A different shard key. Changing from account to customer means rewriting the data. The logical-shard layer helps with rebalancing, not with re-keying.
Making entries mutable. Impossible in practice, and the fact that it is impossible is the point.
Changing the minor-unit convention. Every stored amount would need reinterpretation, and a partial migration would be silently wrong.

the distinction worth making out loud

Separate "not built, easy to add" from "not built, and expensive later". A support tool is a service away. Multi-tenancy is a rewrite. Naming which of your omissions are cheap and which are one-way doors shows you understand the difference between scope and architecture.

256

How this generalises beyond banking

The patterns here are not banking patterns. They are patterns for any system where a quantity must be conserved and the record must be trustworthy.

Pattern	Also applies to
Double-entry with a conserved quantity	Inventory and warehouses, ad impressions and budgets, API credits, carbon accounting, in-game currencies
Append-only with derived state	Event sourcing generally, git, CRDTs, audit systems, anything needing time travel
Idempotency on the intent	Every API that mutates anything over an unreliable network
Transactional outbox	Any service that must update a database and tell somebody, which is most of them
Saga with a named intermediate state	Order fulfilment, seat reservation, multi-step provisioning, anything distributed and multi-step
Available versus actual	Inventory reservations, seat holds, rate-limit budgets, capacity planning
Split it, reassemble on read	Every hot key, hot partition, hot leader and hot lock you will ever meet
Reconcile against an external truth	Any integration where both sides keep records, which is every integration
Correctness as a monitored signal	Any system that can be fast, available and wrong at the same time

worked numbers

          the four constraints that produced all of it:


            1. a quantity must be conserved

            2. networks are unreliable and duplicates happen

            3. multiple parties keep their own records

            4. somebody has to prove it was right, later


wherever all four hold, this architecture is the answer,

whatever the quantity happens to be.

257

Defending it: the twenty hardest questions

The questions most likely to follow this design, with the answer in the form it should be given: direct, then the reason, then the cost.

01Why not just store the balance? It is so much simpler.

Because a stored balance and its history are two facts the database does not force to agree, so a bug can leave them divergent with no way to tell which is right. Derived balances make them one fact. The cost is O(n) reads, which snapshots bound.

02Is double-entry not over-engineering for a wallet app?

It is two extra columns and one extra table. In exchange, every product in round five needed no schema change, and the invariant is a query anyone can run. The cost is low and it is paid once.

03What happens if two transfers hit the same account simultaneously?

They serialise, because the balance check is fused into the INSERT rather than done as a prior SELECT. The second transaction blocks on the row lock, re-evaluates against the committed state, and matches zero rows if funds are insufficient.

04Your client retried and you charged them twice. Defend yourself.

You cannot, because the idempotency key is UNIQUE on the journal row, so the retry fails at the cheapest possible point before any entry exists. We also fingerprint the body, so reusing a key with a different payload is a 422 rather than a silent replay.

05Why not 2PC for cross-shard transfers?

Because 2PC's failure mode is held locks on the ledger when the coordinator dies after PREPARE, which at this volume is the worst outcome available. The saga's cost is a brief non-atomic window, which I make safe by parking the money in a named suspense account so every shard still sums to zero.

06Where is the money during that window?

In in_transit:{shard}, a real account with a real balance. It is never nowhere. Anything lingering there past a threshold is an alert, and that same account is a reconciliation surface in round ten.

07Why an outbox instead of just publishing after commit?

Because publishing after commit is a dual write: a crash between them loses the event permanently and silently. The outbox commits the intent with the money, so the failure mode becomes a delayed or duplicated event rather than a lost one.

08Your relay publishes before marking published, so it creates duplicates.

Deliberately. A duplicate is recoverable and a loss is not, and every consumer deduplicates on event id anyway. That is at-least-once delivery with idempotent handlers, which produces an exactly-once effect.

09Kafka claims exactly-once. Why not use it?

Kafka's exactly-once spans consume, process and produce within Kafka. It does not extend to our database or to NIBSS. The moment an external system is involved, idempotent processing is the only mechanism that works.

10A provider timed out. Did the payment happen?

Unknown, and unknown is a state I model rather than an error I collapse. The money stays in suspense, a resolver queries by our own reference with backoff, and the provider's daily file is the backstop. I will not guess, because guessing wrong in either direction costs real money.

11So retry it on a different provider.

Never on an unknown. That is how the same payment gets sent twice. Failover is permitted after an authoritative rejection or an open circuit breaker, because in both cases we know no money moved.

12Your fraud model is down. Do you stop taking payments?

No. Fraud fails open, because declining everything costs more than the fraud that passes during an outage, and the asynchronous path still sees every transaction. Sanctions screening fails closed, because that one is statutory.

13Your fraud rule locked out 200,000 customers. What now?

It did not, because there is a rate limit on the blocking action itself. Above a few hundred blocks a minute the system suspends enforcement and pages, on the assumption that my rule is wrong rather than that fraud increased a thousandfold.

14How do you know the ledger is correct right now?

Seven checks at four cadences. But the important distinction is that the invariant proves internal consistency, not correctness. A perfectly balanced ledger can be entirely wrong. Correctness comes from reconciling against every counterparty daily.

15A reconciliation break of ₦340 has appeared every day for a month.

That is a systemic bug, not noise. A small break is a large break that has not happened yet, and a recurring one is almost always a rounding error multiplied by volume. Thresholds set triage priority, never tolerance.

16Why can the EU region not fail over to Nigeria?

Because the region boundary is legal rather than technical. Resilience has to be built within each region across availability zones, which costs more than a global active-active design. That is a constraint I inherit, not one I chose.

17A customer demands erasure under GDPR. Your ledger is append-only.

The ledger never held personal data: it holds account ids and amounts. So we erase the profile and retain the financial record under the legal-obligation exemption, with the account id no longer resolvable to a person. Where personal data does sit in immutable archives, we destroy the per-subject encryption key.

18Everything is green and the CEO says transactions are down 80%.

Then my technical signals are all measuring a system that is happily doing nothing, which is why transaction volume compared against the same hour last week is a paging alert. A handler returning 200 with a rejection body improves latency, error rate and CPU simultaneously.

19You are ten years in. Are queries slower?

No, because the operational unit is one 34 GB monthly partition on one of 4,096 shards, not 35 TB. Balance reads are bounded by snapshots, history reads by partition pruning, and old partitions age out to cheaper tiers with the invariant preserved through the archive manifest.

20What would you do differently if you started again?

Three things. Decide multi-tenancy in round one, because retrofitting it is a rewrite. Put the event schema under a registry from the first event, since the ones written before that were the hardest to evolve. And build reconciliation in round one rather than round ten, because every suspense account created before it existed was unmonitored until then.

the answer shape that works for all twenty

Direct answer first, reason second, cost third. Never start with the reasoning and arrive at the answer, because the interviewer cannot tell whether you know until the end. And where the honest answer is "I cannot", say so and describe what you do instead, as in question ten. Confidence about uncertainty is more convincing than certainty about everything.

The module, in one place

P0How To Design A System In A RoomThe frame, the estimates, the vocabulary8 ch P1Round One: “Design A Wallet”Double-entry, derived balances, why RDBMS12 ch P2Round Two: “Two Requests, Same Wallet”MVCC, locking, deadlocks, idempotency10 ch P3Round Three: “10 Million A Day”Hot rows, sharding, the saga, snapshots, cache11 ch P4Round Four: “Six Teams Want This Data”Kafka internals, the dual write, the outbox13 ch P5Round Five: “Loans, Overdraft, Liens”Three balances, holds, accrual, GL mapping11 ch P6Round Six: “Connect To The Outside World”Six rails, the unknown state, breakers, routing13 ch P7Round Seven: “Multi-Currency”Per-currency invariant, FX position, residency9 ch P8Round Eight: “Issue Cards”The 2-second budget, stand-in, clearing, PCI11 ch P9Round Nine: “Stop The Fraud”Rules, features, graphs, blast radius, AML13 ch P10Round Ten: “Prove The Money Is Right”Trial balance, matching passes, breaks, close10 ch P11Round Eleven: “The Business Wants Numbers”Bigtable, BigQuery, medallion, reporting11 ch P12Round Twelve: “Tell The Customer”Fan-out, dedup versus aggregation, receipts8 ch P13Round Thirteen: “Run It At 3AM”SLOs, the fifth signal, runbooks, kill switches11 ch P14Round Fourteen: “How Does Anything Else Talk To It?”gRPC, SSE, ISO 8583, the posting API, mTLS21 ch P15Round Fifteen: “Ten Years Of Data”B-trees, partitioning, tiering, archive, GDPR18 ch P16Bonus Round: “Now Design A Different One”Retail bank, neobank, PSP, POS, lending8 ch