Part 9 · 6 chapters · ~50 min

M9: A Trading System at Scale

A desk trading front end in four rounds: the latency budget hop by hop, the order book as typed arrays with strict sequencing drawn on canvas and the tape as a ring, order entry with risk on the client and idempotent submission by client order id, and degraded modes as first-class states. Then what the nine systems taught together.

56

The brief and the questions

the brief

"A trading front end for the desk: a live order book and tape for a few hundred symbols, a chart, order entry with our risk rules, positions and P&L. Traders have three monitors and no patience. It must never show a stale price as if it were live, and it must never send an order twice."

the questions, and the answers
  1. Tick rates? 50 to 200 a second per active symbol; 2,000 a second at the open or on news; a screen watches 5 to 20 symbols.
  2. What is the latency budget, and who sets it? The desk: tick-to-pixel p99 under 50 ms on the office network; click-to-ack p99 under 100 ms. Measured, not estimated.
  3. What is the feed contract? Snapshot then sequenced deltas per symbol; a gateway with timestamps; binary available.
  4. What are the risk rules, and where do they live? Buying power, position limits, price bands, lot sizes, halts; authoritative on the server; the client should pre-check so rejects are rare.
  5. Devices and network? Desk workstations, wired, three screens, Chrome; occasionally a laptop on VPN.
  6. What may be sampled, and what must be exact? Charts may sample; the book must be exact to the last delta; the tape must show every print (within a scroll-back); orders and positions must be exact.
  7. Failure modes the desk has seen? Stale feeds that looked live; duplicate orders after a retry; a frozen screen during a burst; a reconnect that re-enabled entry before positions were current.
  8. Compliance? Confirmations over thresholds; an audit of what the user saw when they clicked (the tick age at submit).
the requirements, with numbers
  1. FR: book and tape per symbol; charts; order entry with pre-checks and confirmations; order states through fills; positions and P&L; degraded modes; a latency readout.
  2. NFR: tick-to-pixel p99 under 50 ms; click-to-ack p99 under 100 ms; 60 fps through a 2,000 tick/s burst; the book exact to the last delta; the tape complete within 500 prints; no duplicate orders ever; entry disabled until state is current after a reconnect; every displayed price carries its age; memory flat over a trading day.
the thesis
A trading screen has a latency budget it must spend deliberately, and failure modes it must show rather than hide. Every design from M1 to M8 appears here with its strictest setting: binary over the subscription model, the book as a data structure, the tape as the one place frames-not-messages is wrong, orders as idempotent state machines, and degraded modes as first-class states.
57

v1: JSON ticks through React state, and the budget

code
// v1: a tick is a message, a message is setState, setState is a render. correct at 5 ticks a second; seconds behind at 2,000
rt.subscribe(`book:${sym}`, m => setBook(b => applyDelta(b, JSON.parse(m))))      // ~8 ms render per tick; at 200/s the renders queue
function Book({ book }) { return book.asks.slice(0, 20).map(l => <Row key={l.px} px={l.px} sz={l.sz} />) }   // 40 rows re-rendered per tick
the latency budget, hop by hop
  1. Inbound v1: exchange 2 ms, gateway 3 ms, WebSocket 15 ms, JSON.parse and setState and render 8 ms, paint up to 16 ms: ~45 ms, of which the frontend adds ~25, and it grows with the rate because each tick is a render. At 200 ticks a second the book is behind; at 2,000 it is seconds behind while looking live: the failure the desk named.
  2. Outbound v1: a fetch per order (setup, JSON), the network twice, the gateway and the exchange: ~44 ms with ~5 from the frontend. Acceptable, until a server risk check is added as a gate before sending.
  3. The budget the desk set: tick-to-pixel p99 under 50 ms, click-to-ack p99 under 100 ms, 60 fps through a burst, no displayed price older than the last tick received. Written down; measured per tick.
where v2 will spend
  1. Binary ticks (40 bytes, decoded with a DataView in microseconds, no allocation) instead of 400 bytes of JSON.
  2. No React state in the hot path: ticks apply into a data structure; the book is drawn on canvas once per frame from it.
  3. A warm connection for orders, binary messages, and risk checked before the click rather than after.
  4. Timestamps at every hop so the budget is observable, and a readout on screen so the trader knows when the number is late.
THE LATENCY BUDGET
from exchange tick to pixel, and from click to acknowledged order, in milliseconds
swipe the figure sideways, or tap expand for full screen
1/6
inbound v1
Inbound, as found in v1: exchange → market data feed (2 ms) → the firm's gateway (3 ms) → WebSocket to the browser (15 ms on a good link) → JSON.parse of a 400-byte tick (0.05 ms, but ×200 a second) → a setState per tick → React render of the order book (8 ms) → paint (next frame: up to 16 ms). Frontend share: ~25 to 40 ms of a ~60 ms total, and it grows with rate because each tick is a render.
58

Round two: the book as data, the tape as a ring

code
// v2: the book as typed arrays with integer ticks; deltas applied in place; a gap resnapshots; a canvas reads it per frame
class Book {
  // bids sorted descending, asks ascending; parallel sizes; a Map for O(1) lookup; depth capped at what the feed sends
  bidPx = new Int32Array(512); bidSz = new Int32Array(512); nBid = 0; bidIdx = new Map<number, number>()
  askPx = new Int32Array(512); askSz = new Int32Array(512); nAsk = 0; askIdx = new Map<number, number>()
  seq = 0; changed = new Set<number>()                       // price levels touched since the last frame (for highlights)
  snapshot(s: Snapshot) { this.seq = s.seq; this.nBid = this.nAsk = 0; this.bidIdx.clear(); this.askIdx.clear(); for (const [px, sz] of s.bids) this.set(0, px, sz); for (const [px, sz] of s.asks) this.set(1, px, sz) }
  apply(d: Delta): 'ok' | 'gap' {
    if (d.seq !== this.seq + 1) return 'gap'                  // the only correct response: stop, resubscribe, resnapshot
    this.seq = d.seq; this.set(d.side, d.px, d.sz); this.changed.add(d.px); return 'ok'
  }
  private set(side: 0 | 1, px: number, sz: number) {
    const [P, S, idx] = side === 0 ? [this.bidPx, this.bidSz, this.bidIdx] : [this.askPx, this.askSz, this.askIdx]
    const i = idx.get(px)
    if (i !== undefined) { if (sz === 0) this.remove(side, i); else S[i] = sz; return }
    if (sz === 0) return
    const n = side === 0 ? this.nBid : this.nAsk
    let lo = 0, hi = n; while (lo < hi) { const m = (lo + hi) >> 1; if (side === 0 ? P[m] > px : P[m] < px) lo = m + 1; else hi = m }   // binary search the insertion point
    P.copyWithin(lo + 1, lo, n); S.copyWithin(lo + 1, lo, n); P[lo] = px; S[lo] = sz   // splice O(depth) in a typed array: microseconds
    for (let k = lo; k <= n; k++) idx.set(P[k], k)
    side === 0 ? this.nBid++ : this.nAsk++
  }
  // remove(): the mirror; decode(): a DataView over the 40-byte binary frame: seq u32, side u8, px i32 (ticks), sz i32
}
// render, once per frame: const top = Math.min(20, book.nAsk); for (let i = top - 1; i >= 0; i--) drawRow(ctx, book.askPx[i] / 1e4, book.askSz[i], cum, book.changed.has(book.askPx[i])); …; book.changed.clear()
// feed → book: rt.subscribe(`book:${sym}`, m => { if (m.kind === 'snapshot') book.snapshot(m); else if (book.apply(m) === 'gap') rt.resubscribe(`book:${sym}`) })
v2
  1. The feed contract, strictly: snapshot then deltas with sequence numbers; a delta that is not lastSeq + 1 is a gap, and the only correct response is to stop, resubscribe and take a new snapshot (M8's subscription model at its strictest). Never apply out of order; never guess.
  2. The structure: prices as integer ticks (floats are for formatting only; a book in floats is wrong money), sorted typed arrays per side with parallel sizes, a Map for O(1) level lookup, binary-search insertion and copyWithin for the splice. 2,000 deltas a second is ~1 ms a second of CPU. Nothing is coalesced away because applying is cheaper than deciding not to; the book is exact to the last delta.
  3. The render reads the top 20 levels per side once per rAF and draws price, size and cumulative depth bars on a canvas, highlighting the changed-set recorded by the apply step with a 300 ms fade. ~1 ms per frame at any tick rate (M4).
  4. The tape is a ring of the last 500 prints drawn per frame, with a count for prints beyond the visible rows in a burst; scroll-back switches to a virtualised DOM list (M6) over history fetched on demand (M1). This is the one place "frames, not messages" is wrong, and the ring satisfies it.
  5. Where React is: the chrome, the order ticket, the positions grid, and derived numbers (mid, spread) at a throttled rate. The book and tape are canvases React mounts once through refs and never re-renders (the React course part 12's escape hatches).
the sentence
  1. v2 buys a book exact to the last delta at 60 fps through any burst and ~9 ms of frontend latency flat with rate; pays: a binary feed contract and decoder (complexity), imperative data structures and canvases outside React (complexity; a different testing style), and integer-tick arithmetic everywhere prices are touched (complexity, correct).
V2: THE ORDER BOOK AS DATA, NOT COMPONENTS
snapshots, deltas, sequence gaps, and a canvas that draws the book once per frame
swipe the figure sideways, or tap expand for full screen
1/6
the feed
The feed contract: subscribe(book, symbol, depth) → a snapshot { seq, bids[], asks[] } then deltas { seq, side, price, size } where size 0 removes the level. The client holds lastSeq; a delta with seq ≠ lastSeq + 1 means a gap: stop applying, resubscribe, take the new snapshot. Never apply out of order; never guess.
59

Round three: order entry, risk on the client, idempotent submission

code
// v3: order entry with continuous client-side risk, a warm binary submit, and a per-order state machine keyed by client id
const risk = useRiskTopic(account)            // buying power, positions, bands, halts: updated by fills over M8
const check = useMemo(() => validate(ticket, risk, lastTrade), [ticket, risk, lastTrade])   // { ok, reasons[] } on every keystroke
<button disabled={!check.ok} onClick={() => submit(ticket)}>Submit</button>
{check.reasons.map(r => <p className="reason" key={r.code}>{r.text}</p>)}

function submit(t: Ticket) {
  const clientOrderId = crypto.randomUUID()
  orders.upsert({ clientOrderId, ...t, state: 'sending', sentAt: performance.now() })       // the pending row, immediately
  rt.sendBinary(encodeOrder({ clientOrderId, ...t }))                                       // warm socket; no fetch setup, no JSON
  const timer = setTimeout(() => { if (orders.get(clientOrderId)?.state === 'sending') rt.sendBinary(encodeOrder({ clientOrderId, ...t })) }, 2000)   // resend, same id: server dedupes
  pending.set(clientOrderId, timer)
}
rt.subscribe(`orders:${account}`, m => {                      // ack / reject / fill / cancel-ack, all correlated by id
  if (m.kind === 'ack') orders.update(m.clientOrderId, { orderId: m.orderId, state: 'accepted' })
  if (m.kind === 'reject') orders.update(m.clientOrderId, { state: 'rejected', reason: m.reason })
  if (m.kind === 'fill') orders.update(m.clientOrderId, o => ({ filled: o.filled + m.qty, state: o.filled + m.qty >= o.qty ? 'filled' : 'partial' }))
  clearTimeout(pending.get(m.clientOrderId)); pending.delete(m.clientOrderId)
})
rt.on('reconnected', async () => { const snap = await api.orders.snapshot(account); orders.reconcile(snap); for (const id of pending.keys()) if (!snap.has(id)) orders.update(id, { state: 'unknown' }) })   // honest: "unknown: check"
// confirmations: notional over threshold → a modal showing symbol, side, qty, price, notional, tick age; "flatten" has its own
the break
  1. A server-side risk check added as a round trip before every send pushes click-to-ack past 100 ms; a slow ack and a retried click produce two orders; a fill lands while the ticket is open and the buying power shown is wrong. The numbers are the gate on the critical path and the absence of an id.
v3
  1. Risk state on the client as a topic (M8): buying power, positions, bands, lot sizes, halts, permissions, updated as fills land. Validation runs on every keystroke and names its reasons inline; the submit button is enabled only for an order the server will accept. The gate moved off the click by running continuously.
  2. Preview, never authority: the server re-checks atomically at submit; the occasional client-passed, server-rejected order is shown with the server's reason; the client is slightly more conservative than the server, never less. The rules engine is shared and versioned together.
  3. Submission as a binary message on the warm socket with a clientOrderId; a pending row at once; ack, reject, fills and cancel-acks correlated by id through a state machine (sending, accepted, partial, filled, rejected, cancelled, unknown).
  4. Idempotent, always: no ack in 2 s resends with the same id; the server deduplicates; a connection that dies mid-submit leaves the row "unknown: check" until the reconnect reconciles against the orders snapshot. The client never creates a second order because a response was slow.
  5. Confirmations over a notional or size threshold showing symbol, side, quantity, price, notional and tick age (the audit's "what they saw"); modifier keys on shortcuts; a dedicated flatten button. The desk sets thresholds; the server keeps hard limits regardless.
the sentence
  1. v3 buys order entry that is fast, honest and safe to retry; pays: a shared, versioned rules engine (complexity, a contract), per-order state machines and reconciliation (complexity), and confirmation friction chosen on purpose (latency).
V3: ORDER ENTRY, RISK ON THE CLIENT, AND IDEMPOTENT SUBMISSION
the click that moves money, and everything that happens before and after it
swipe the figure sideways, or tap expand for full screen
1/6
risk state
Risk state on the client: buying power, open position per symbol, max order size, price bands (a percentage from the last trade), trading halts, and the account's permissions; subscribed as a topic (M8) so it updates as fills land. Validation runs on every keystroke against it and shows the reason inline ("exceeds buying power by 1,200"). The button is disabled for an order that would fail.
60

Round four: degraded modes

Every failure the desk named is a mode the screen must detect, show, restrict and recover from. v4 makes them first-class states rather than things that happen to the UI.

v4: five modes, each with a detector, a display, a restriction and a recovery
  1. Feed lag: per-tick tick-to-apply over budget for 5 s → the readout turns amber then red and the book header says "delayed 240 ms" → market orders require a confirmation showing the tick age → automatic recovery.
  2. Feed stall: no tick for N seconds on an active symbol while the heartbeat still answers (the subtle one) → the book greys with "no data since 12:04:31" → entry disabled for that symbol except cancels → resubscribe; stay degraded while the gateway reports the feed down.
  3. Disconnected: the socket is closed and reconnecting → a banner with attempt count and last known time; book and tape frozen and greyed; positions "as of" → no new orders; cancels queued or refused by desk policy → on reconnect, resnapshot book, orders, positions and risk and reconcile pending orders by id before entry is enabled.
  4. Halt and auction: the venue's status topic → a label and the venue's display rules (indicative price in auction) → only the order types the venue allows → the venue's topic says when.
  5. Client overload: rAF cadence under 50 fps for 2 s, apply lag growing with the network fine, memory high → "reduced mode" → depth from 50 to 10 levels, charts and news paused, the tape as a count → automatic when the cadence returns.
the sentence, and the stop
  1. v4 buys a screen that is honest under every failure the desk could name; pays: detectors and watchdogs per mode (complexity), a mode state machine the whole UI respects (complexity), a strict resnapshot-before-enable on reconnect (latency, chosen), and the design time of agreeing each mode's restrictions with the desk and compliance.
  2. Stop: every number in the brief is met and measured on screen. Algorithmic order types, multi-leg tickets and a replay tool for the audit are the next products on the same base; the server side of all of it is the Distributed and SRE courses.
V4: DEGRADED MODES
what the screen does when the feed is late, the connection is gone, or the exchange is halted
swipe the figure sideways, or tap expand for full screen
1/6
feed lag
Feed lag: tick→apply over the budget (p99 over 50 ms for 5 s). Detector: the per-tick measurement from the latency figure. State: the lag readout turns amber, then red; the book header says "delayed 240 ms". Restriction: none at amber; at red, market orders require a confirmation showing the last tick age. Recovery: automatic when lag drops.
61

The whole board, the course, and the exercise

RoundThe numberThe breakThe designPaid in
v15 ticks/s200 to 2,000 ticks/s: a render per tick; seconds behind while looking liveJSON into React state; a budget written down with the deskNothing yet; the budget is the deliverable
v22,000 ticks/s; p99 50 msFrontend latency grows with rateBinary ticks; the book as typed arrays with integer ticks and strict sequencing; canvas per frame; the tape as a ring; React for the chrome onlyA binary contract; imperative structures outside React; integer arithmetic
v3Click-to-ack 100 ms; a retried clickA risk gate on the critical path; duplicate orders; stale buying powerRisk as a client topic with continuous validation; warm binary submit with client order ids; per-order state machines; reconciliation; confirmations with tick ageA shared rules engine; state machines; chosen friction
v4Every failure the desk has seenStale that looks live; entry enabled before state is currentFive degraded modes with detectors, displays, restrictions and recoveries; resnapshot before enableWatchdogs; a mode machine; agreements with the desk
what the nine systems taught, together
  1. The number decides the design (rows, pixels, ticks, editors, tabs), and the design decides which number breaks it next.
  2. Data crosses boundaries by format: columnar buffers to workers, sized media to slots, binary ticks to typed arrays, ids to every view. The format is chosen before the mechanism.
  3. Frames are the unit of rendering; messages are inputs to frames; the tape is the exception, and it is a ring.
  4. Identity beats position: keys, cursors, CRDT ids, client order ids, relative positions. Everything that survives change is anchored to an id.
  5. Consistency is a chosen setting: approximate counts, 30 s staleness, convergence without intent, a preview that is not the authority. Each is a sentence that names what was paid.
  6. Failure is a state, not an event: degraded modes, "unknown: check", "no data since", "12 new". The honest screen shows its own age.
the exercise, for the course
Pick the system you own that is closest to one of the nine. Write its brief in one paragraph, ask the questions from its part, and put numbers on the requirements. Then write the sentence for the round it is in and the number that moves it to the next. That sentence, in a design review, is the whole skill.