Part 6 · 1 chapters · ~8 min

Bayesian Thinking

Conditional probability, Bayes' theorem, base rates and the base-rate fallacy, a fraud flag worked through, alert precision and on-call fatigue, priors and posteriors, Bayesian A/B testing with Beta distributions, credible intervals, and updating beliefs during incidents.

Base rates decide everything

code

// P(fraud | flag) = P(flag | fraud) P(fraud) / P(flag)
const sens = 0.99, fpr = 0.02, base = 0.005;
const posterior = sens * base / (sens * base + fpr * (1 - base));     // 0.199

// the same arithmetic for alerts: an alert that fires on 2% of healthy minutes, when real incidents
// occupy 0.1% of minutes, is mostly noise however good it looks on incidents (SRE course: page on SLO burn)

// Bayesian A/B: Beta(1 + conversions, 1 + misses) per arm; P(B > A) by sampling
const beta = (a: number, b: number) => { /* sample via two gamma draws */ };
const pBbetter = Array.from({ length: 20000 }, () => beta(1 + cb, 1 + nb - cb) > beta(1 + ca, 1 + na - ca)).filter(Boolean).length / 20000;

During incidents, think in likelihoods: "if the database were the cause, how likely is what we see?" versus the same for the network. Evidence that is equally likely under every hypothesis (CPU is high everywhere) should not move your belief.

WHAT A FRAUD FLAG MEANS

model: 99% sensitive, 2% false-positive rate; 0.5% of transactions are fraud

swipe the figure sideways, or tap expand for full screen

1/4

the model looks great

The model catches 99% of fraud and wrongly flags only 2% of legitimate transactions.

99% sensitive, 2% FPRlooks excellent