M9: A Trading System at Scale
A desk trading front end in four rounds: the latency budget hop by hop, the order book as typed arrays with strict sequencing drawn on canvas and the tape as a ring, order entry with risk on the client and idempotent submission by client order id, and degraded modes as first-class states. Then what the nine systems taught together.
The brief and the questions
"A trading front end for the desk: a live order book and tape for a few hundred symbols, a chart, order entry with our risk rules, positions and P&L. Traders have three monitors and no patience. It must never show a stale price as if it were live, and it must never send an order twice."
- Tick rates? 50 to 200 a second per active symbol; 2,000 a second at the open or on news; a screen watches 5 to 20 symbols.
- What is the latency budget, and who sets it? The desk: tick-to-pixel p99 under 50 ms on the office network; click-to-ack p99 under 100 ms. Measured, not estimated.
- What is the feed contract? Snapshot then sequenced deltas per symbol; a gateway with timestamps; binary available.
- What are the risk rules, and where do they live? Buying power, position limits, price bands, lot sizes, halts; authoritative on the server; the client should pre-check so rejects are rare.
- Devices and network? Desk workstations, wired, three screens, Chrome; occasionally a laptop on VPN.
- What may be sampled, and what must be exact? Charts may sample; the book must be exact to the last delta; the tape must show every print (within a scroll-back); orders and positions must be exact.
- Failure modes the desk has seen? Stale feeds that looked live; duplicate orders after a retry; a frozen screen during a burst; a reconnect that re-enabled entry before positions were current.
- Compliance? Confirmations over thresholds; an audit of what the user saw when they clicked (the tick age at submit).
- FR: book and tape per symbol; charts; order entry with pre-checks and confirmations; order states through fills; positions and P&L; degraded modes; a latency readout.
- NFR: tick-to-pixel p99 under 50 ms; click-to-ack p99 under 100 ms; 60 fps through a 2,000 tick/s burst; the book exact to the last delta; the tape complete within 500 prints; no duplicate orders ever; entry disabled until state is current after a reconnect; every displayed price carries its age; memory flat over a trading day.
v1: JSON ticks through React state, and the budget
// v1: a tick is a message, a message is setState, setState is a render. correct at 5 ticks a second; seconds behind at 2,000
rt.subscribe(`book:${sym}`, m => setBook(b => applyDelta(b, JSON.parse(m)))) // ~8 ms render per tick; at 200/s the renders queue
function Book({ book }) { return book.asks.slice(0, 20).map(l => <Row key={l.px} px={l.px} sz={l.sz} />) } // 40 rows re-rendered per tick- Inbound v1: exchange 2 ms, gateway 3 ms, WebSocket 15 ms, JSON.parse and setState and render 8 ms, paint up to 16 ms: ~45 ms, of which the frontend adds ~25, and it grows with the rate because each tick is a render. At 200 ticks a second the book is behind; at 2,000 it is seconds behind while looking live: the failure the desk named.
- Outbound v1: a fetch per order (setup, JSON), the network twice, the gateway and the exchange: ~44 ms with ~5 from the frontend. Acceptable, until a server risk check is added as a gate before sending.
- The budget the desk set: tick-to-pixel p99 under 50 ms, click-to-ack p99 under 100 ms, 60 fps through a burst, no displayed price older than the last tick received. Written down; measured per tick.
- Binary ticks (40 bytes, decoded with a DataView in microseconds, no allocation) instead of 400 bytes of JSON.
- No React state in the hot path: ticks apply into a data structure; the book is drawn on canvas once per frame from it.
- A warm connection for orders, binary messages, and risk checked before the click rather than after.
- Timestamps at every hop so the budget is observable, and a readout on screen so the trader knows when the number is late.
Round two: the book as data, the tape as a ring
// v2: the book as typed arrays with integer ticks; deltas applied in place; a gap resnapshots; a canvas reads it per frame
class Book {
// bids sorted descending, asks ascending; parallel sizes; a Map for O(1) lookup; depth capped at what the feed sends
bidPx = new Int32Array(512); bidSz = new Int32Array(512); nBid = 0; bidIdx = new Map<number, number>()
askPx = new Int32Array(512); askSz = new Int32Array(512); nAsk = 0; askIdx = new Map<number, number>()
seq = 0; changed = new Set<number>() // price levels touched since the last frame (for highlights)
snapshot(s: Snapshot) { this.seq = s.seq; this.nBid = this.nAsk = 0; this.bidIdx.clear(); this.askIdx.clear(); for (const [px, sz] of s.bids) this.set(0, px, sz); for (const [px, sz] of s.asks) this.set(1, px, sz) }
apply(d: Delta): 'ok' | 'gap' {
if (d.seq !== this.seq + 1) return 'gap' // the only correct response: stop, resubscribe, resnapshot
this.seq = d.seq; this.set(d.side, d.px, d.sz); this.changed.add(d.px); return 'ok'
}
private set(side: 0 | 1, px: number, sz: number) {
const [P, S, idx] = side === 0 ? [this.bidPx, this.bidSz, this.bidIdx] : [this.askPx, this.askSz, this.askIdx]
const i = idx.get(px)
if (i !== undefined) { if (sz === 0) this.remove(side, i); else S[i] = sz; return }
if (sz === 0) return
const n = side === 0 ? this.nBid : this.nAsk
let lo = 0, hi = n; while (lo < hi) { const m = (lo + hi) >> 1; if (side === 0 ? P[m] > px : P[m] < px) lo = m + 1; else hi = m } // binary search the insertion point
P.copyWithin(lo + 1, lo, n); S.copyWithin(lo + 1, lo, n); P[lo] = px; S[lo] = sz // splice O(depth) in a typed array: microseconds
for (let k = lo; k <= n; k++) idx.set(P[k], k)
side === 0 ? this.nBid++ : this.nAsk++
}
// remove(): the mirror; decode(): a DataView over the 40-byte binary frame: seq u32, side u8, px i32 (ticks), sz i32
}
// render, once per frame: const top = Math.min(20, book.nAsk); for (let i = top - 1; i >= 0; i--) drawRow(ctx, book.askPx[i] / 1e4, book.askSz[i], cum, book.changed.has(book.askPx[i])); …; book.changed.clear()
// feed → book: rt.subscribe(`book:${sym}`, m => { if (m.kind === 'snapshot') book.snapshot(m); else if (book.apply(m) === 'gap') rt.resubscribe(`book:${sym}`) })- The feed contract, strictly: snapshot then deltas with sequence numbers; a delta that is not lastSeq + 1 is a gap, and the only correct response is to stop, resubscribe and take a new snapshot (M8's subscription model at its strictest). Never apply out of order; never guess.
- The structure: prices as integer ticks (floats are for formatting only; a book in floats is wrong money), sorted typed arrays per side with parallel sizes, a Map for O(1) level lookup, binary-search insertion and
copyWithinfor the splice. 2,000 deltas a second is ~1 ms a second of CPU. Nothing is coalesced away because applying is cheaper than deciding not to; the book is exact to the last delta. - The render reads the top 20 levels per side once per rAF and draws price, size and cumulative depth bars on a canvas, highlighting the changed-set recorded by the apply step with a 300 ms fade. ~1 ms per frame at any tick rate (M4).
- The tape is a ring of the last 500 prints drawn per frame, with a count for prints beyond the visible rows in a burst; scroll-back switches to a virtualised DOM list (M6) over history fetched on demand (M1). This is the one place "frames, not messages" is wrong, and the ring satisfies it.
- Where React is: the chrome, the order ticket, the positions grid, and derived numbers (mid, spread) at a throttled rate. The book and tape are canvases React mounts once through refs and never re-renders (the React course part 12's escape hatches).
- v2 buys a book exact to the last delta at 60 fps through any burst and ~9 ms of frontend latency flat with rate; pays: a binary feed contract and decoder (complexity), imperative data structures and canvases outside React (complexity; a different testing style), and integer-tick arithmetic everywhere prices are touched (complexity, correct).
Round three: order entry, risk on the client, idempotent submission
// v3: order entry with continuous client-side risk, a warm binary submit, and a per-order state machine keyed by client id
const risk = useRiskTopic(account) // buying power, positions, bands, halts: updated by fills over M8
const check = useMemo(() => validate(ticket, risk, lastTrade), [ticket, risk, lastTrade]) // { ok, reasons[] } on every keystroke
<button disabled={!check.ok} onClick={() => submit(ticket)}>Submit</button>
{check.reasons.map(r => <p className="reason" key={r.code}>{r.text}</p>)}
function submit(t: Ticket) {
const clientOrderId = crypto.randomUUID()
orders.upsert({ clientOrderId, ...t, state: 'sending', sentAt: performance.now() }) // the pending row, immediately
rt.sendBinary(encodeOrder({ clientOrderId, ...t })) // warm socket; no fetch setup, no JSON
const timer = setTimeout(() => { if (orders.get(clientOrderId)?.state === 'sending') rt.sendBinary(encodeOrder({ clientOrderId, ...t })) }, 2000) // resend, same id: server dedupes
pending.set(clientOrderId, timer)
}
rt.subscribe(`orders:${account}`, m => { // ack / reject / fill / cancel-ack, all correlated by id
if (m.kind === 'ack') orders.update(m.clientOrderId, { orderId: m.orderId, state: 'accepted' })
if (m.kind === 'reject') orders.update(m.clientOrderId, { state: 'rejected', reason: m.reason })
if (m.kind === 'fill') orders.update(m.clientOrderId, o => ({ filled: o.filled + m.qty, state: o.filled + m.qty >= o.qty ? 'filled' : 'partial' }))
clearTimeout(pending.get(m.clientOrderId)); pending.delete(m.clientOrderId)
})
rt.on('reconnected', async () => { const snap = await api.orders.snapshot(account); orders.reconcile(snap); for (const id of pending.keys()) if (!snap.has(id)) orders.update(id, { state: 'unknown' }) }) // honest: "unknown: check"
// confirmations: notional over threshold → a modal showing symbol, side, qty, price, notional, tick age; "flatten" has its own- A server-side risk check added as a round trip before every send pushes click-to-ack past 100 ms; a slow ack and a retried click produce two orders; a fill lands while the ticket is open and the buying power shown is wrong. The numbers are the gate on the critical path and the absence of an id.
- Risk state on the client as a topic (M8): buying power, positions, bands, lot sizes, halts, permissions, updated as fills land. Validation runs on every keystroke and names its reasons inline; the submit button is enabled only for an order the server will accept. The gate moved off the click by running continuously.
- Preview, never authority: the server re-checks atomically at submit; the occasional client-passed, server-rejected order is shown with the server's reason; the client is slightly more conservative than the server, never less. The rules engine is shared and versioned together.
- Submission as a binary message on the warm socket with a
clientOrderId; a pending row at once; ack, reject, fills and cancel-acks correlated by id through a state machine (sending, accepted, partial, filled, rejected, cancelled, unknown). - Idempotent, always: no ack in 2 s resends with the same id; the server deduplicates; a connection that dies mid-submit leaves the row "unknown: check" until the reconnect reconciles against the orders snapshot. The client never creates a second order because a response was slow.
- Confirmations over a notional or size threshold showing symbol, side, quantity, price, notional and tick age (the audit's "what they saw"); modifier keys on shortcuts; a dedicated flatten button. The desk sets thresholds; the server keeps hard limits regardless.
- v3 buys order entry that is fast, honest and safe to retry; pays: a shared, versioned rules engine (complexity, a contract), per-order state machines and reconciliation (complexity), and confirmation friction chosen on purpose (latency).
Round four: degraded modes
Every failure the desk named is a mode the screen must detect, show, restrict and recover from. v4 makes them first-class states rather than things that happen to the UI.
- Feed lag: per-tick tick-to-apply over budget for 5 s → the readout turns amber then red and the book header says "delayed 240 ms" → market orders require a confirmation showing the tick age → automatic recovery.
- Feed stall: no tick for N seconds on an active symbol while the heartbeat still answers (the subtle one) → the book greys with "no data since 12:04:31" → entry disabled for that symbol except cancels → resubscribe; stay degraded while the gateway reports the feed down.
- Disconnected: the socket is closed and reconnecting → a banner with attempt count and last known time; book and tape frozen and greyed; positions "as of" → no new orders; cancels queued or refused by desk policy → on reconnect, resnapshot book, orders, positions and risk and reconcile pending orders by id before entry is enabled.
- Halt and auction: the venue's status topic → a label and the venue's display rules (indicative price in auction) → only the order types the venue allows → the venue's topic says when.
- Client overload: rAF cadence under 50 fps for 2 s, apply lag growing with the network fine, memory high → "reduced mode" → depth from 50 to 10 levels, charts and news paused, the tape as a count → automatic when the cadence returns.
- v4 buys a screen that is honest under every failure the desk could name; pays: detectors and watchdogs per mode (complexity), a mode state machine the whole UI respects (complexity), a strict resnapshot-before-enable on reconnect (latency, chosen), and the design time of agreeing each mode's restrictions with the desk and compliance.
- Stop: every number in the brief is met and measured on screen. Algorithmic order types, multi-leg tickets and a replay tool for the audit are the next products on the same base; the server side of all of it is the Distributed and SRE courses.
The whole board, the course, and the exercise
| Round | The number | The break | The design | Paid in |
|---|---|---|---|---|
| v1 | 5 ticks/s | 200 to 2,000 ticks/s: a render per tick; seconds behind while looking live | JSON into React state; a budget written down with the desk | Nothing yet; the budget is the deliverable |
| v2 | 2,000 ticks/s; p99 50 ms | Frontend latency grows with rate | Binary ticks; the book as typed arrays with integer ticks and strict sequencing; canvas per frame; the tape as a ring; React for the chrome only | A binary contract; imperative structures outside React; integer arithmetic |
| v3 | Click-to-ack 100 ms; a retried click | A risk gate on the critical path; duplicate orders; stale buying power | Risk as a client topic with continuous validation; warm binary submit with client order ids; per-order state machines; reconciliation; confirmations with tick age | A shared rules engine; state machines; chosen friction |
| v4 | Every failure the desk has seen | Stale that looks live; entry enabled before state is current | Five degraded modes with detectors, displays, restrictions and recoveries; resnapshot before enable | Watchdogs; a mode machine; agreements with the desk |
- The number decides the design (rows, pixels, ticks, editors, tabs), and the design decides which number breaks it next.
- Data crosses boundaries by format: columnar buffers to workers, sized media to slots, binary ticks to typed arrays, ids to every view. The format is chosen before the mechanism.
- Frames are the unit of rendering; messages are inputs to frames; the tape is the exception, and it is a ring.
- Identity beats position: keys, cursors, CRDT ids, client order ids, relative positions. Everything that survives change is anchored to an id.
- Consistency is a chosen setting: approximate counts, 30 s staleness, convergence without intent, a preview that is not the authority. Each is a sentence that names what was paid.
- Failure is a state, not an event: degraded modes, "unknown: check", "no data since", "12 new". The honest screen shows its own age.