Part 4 · 2 chapters · ~12 min

Statistical Thinking and Being Fooled by Numbers

Base rates and the base rate fallacy, regression to the mean, Simpson's paradox, correlation and causation, small samples and noise, percentages of percentages, misleading charts, and questions to ask of any number.

9

Simpson's paradox and friends

Numbers in dashboards feel objective. Most are aggregates, and aggregates can say the opposite of what is true in every part.

SIMPSON'S PARADOX IN A LOAN APPROVAL FUNNEL
approval rate per segment vs overall (illustrative)
app A, salaried80% of 900app B, salaried85% of 100app A, self-employed30% of 100app B, self-employed35% of 900app A overall75%app B overall40%
swipe the figure sideways, or tap expand for full screen
1/4
per segment
Within each segment, version B approves more applicants than version A: 85% vs 80% for salaried, 35% vs 30% for self-employed.
B wins in every segmentsalaried and self-employed alike
10

Questions to ask of any number

trapexamplequestion
base rate fallacya fraud model 99% accurate flags a user; fraud is 1 in 1,000, so most flags are false positiveshow common is the thing at all?
regression to the meanthe worst-performing week improves after a "fix", as it would have anywaywas the starting point extreme?
correlation, not causationusers who enable notifications retain better (engaged users do both)could a third factor cause both?
small samples"conversion up 50%!" from 4 to 6 signupshow many observations?
relative vs absolute"errors doubled" from 0.01% to 0.02%what are the absolute numbers?
code
the fraud flag, worked (base rate 1 in 1,000; model catches 99% of fraud; 1% false positive rate)
of 100,000 users: 100 fraudsters → 99 flagged;  99,900 honest → 999 flagged
P(fraud | flagged) = 99 / (99 + 999) ≈ 9%     → 91% of flagged users are honest