Part 8 · 1 chapters · ~8 min

Backoff, Jitter and Load Balancing

Exponential backoff variants (full, equal and decorrelated jitter) and why jitter matters, retry budgets, load-balancing algorithms (round robin, least connections, power of two choices, EWMA, hashing) and their behaviour under slow or failing backends, and herding effects.

Spreading load and retries

code

// jitter variants (AWS Architecture Blog, "Exponential Backoff and Jitter")
full         = random(0, min(cap, base * 2 ** attempt))
equal        = half + random(0, half)            where half = min(cap, base * 2 ** attempt) / 2
decorrelated = min(cap, random(base, previousSleep * 3))
// power of two choices
const pick = (bs) => { const a = bs[rand(bs.length)], b = bs[rand(bs.length)]; return a.inFlight <= b.inFlight ? a : b; };

LOAD BALANCING ALGORITHMS

which backend gets the next request

swipe the figure sideways, or tap expand for full screen

1/6

round robin

Fine when backends are identical and requests similar; poor when one backend is slow (it gets the same share and queues grow).

simple, load-blindslow backends still get traffic