Part 1 · 2 chapters · ~20 min
The CPU
How a modern core gets several instructions done per cycle: pipelining with data and control hazards and forwarding; branch prediction and the cost of a misprediction, the sorted-array effect; out-of-order execution; SIMD; Spectre; and a table of what each event costs in cycles.
3
Pipelines and hazards
an assembly line for instructions
- Without a pipeline: five cycles per instruction.
- With a pipeline: one instruction completes per cycle once the pipeline is full. Latency per instruction does not change.
- Data hazards: a later instruction needs a result that is not ready yet.
- Forwarding hides most of them. A load followed immediately by a use still stalls.
- Control hazards: after a branch, the CPU must wait or guess.
- Real pipelines are deep and wide, so every miss and wrong guess costs more.
PIPELINING AND HAZARDS
five instructions overlapping in five stages, and the stalls when one needs another's result
swipe the figure sideways, or tap expand for full screen
1/6
no pipeline
Without pipelining: each instruction goes through fetch (IF), decode (ID), execute (EX), memory (MEM) and write-back (WB) before the next starts: five cycles per instruction.
4
Prediction, reordering, SIMD and the cost of a miss
keeping the arithmetic busy
- Branch prediction handles regular patterns almost perfectly.
- A misprediction flushes 15 to 20 cycles of work.
- Sorted data or branchless code avoids mispredictions.
- Out-of-order execution hides latency behind independent work.
- SIMD runs one instruction across many lanes of data.
- Spectre is speculation's side channel, and the reason browsers isolate sites.
code
// try it in Node: the branch-prediction effect, visible from JavaScript
const N = 1 << 22;
const data = new Uint8Array(N).map(() => Math.random() * 256);
const sorted = data.slice().sort();
const sum = (xs: Uint8Array) => { let s = 0; for (let k = 0; k < xs.length; k++) if (xs[k] >= 128) s += xs[k]; return s; };
for (const [name, xs] of [['random', data], ['sorted', sorted]] as const) {
sum(xs); // warm up the JIT
const t = performance.now(); for (let r = 0; r < 10; r++) sum(xs);
console.log(name, (performance.now() - t).toFixed(0), 'ms');
}
// measured on Apple silicon, Node 25: random 148 ms, sorted 35 ms (about 4×) on the same values| event | approximate cost (cycles at ~3 GHz) |
|---|---|
| simple ALU op (add, and) | 1 (and several per cycle) |
| correctly predicted branch | ~0 to 1 |
| multiply / divide (integer) | 3 / 20 to 40 |
| L1 cache hit | ~4 |
| branch misprediction | 15 to 20 |
| L2 / L3 hit | ~12 / ~40 |
| main memory (cache miss) | 200 to 300 |
| system call | hundreds to thousands |
| context switch | thousands, plus cold caches afterwards |
BRANCH PREDICTION, OUT-OF-ORDER, SIMD
how a modern core guesses, reorders and widens to keep its pipeline full
swipe the figure sideways, or tap expand for full screen
1/6
prediction
Branch prediction: the predictor records the history of each branch (and of recent branches together) and guesses the next outcome before it is known. Loop branches and branches on regular data are predicted over 99% of the time. The fetch continues down the guessed path speculatively.