Part 3 · 1 chapters · ~8 min

Confidence Intervals

Point estimates versus intervals, the standard error, intervals for means and proportions, what a 95% confidence interval does and does not mean, the bootstrap for percentiles and other awkward statistics, interval width and sample size, and showing uncertainty on dashboards.

Intervals for anything with the bootstrap

code

// 95% bootstrap confidence interval for p99
const smp = latencies.slice(0, 2000);
const boots = Array.from({ length: 2000 }, () => {
  const r = Array.from({ length: smp.length }, () => smp[Math.floor(rnd() * smp.length)]);
  return pct(r, 99);
});
[pct(boots, 2.5), pct(boots, 97.5)]     // [96, 697] ms around a point estimate of 511 ms

// interval for a proportion (normal approximation): 52 failures in 10,000 transfers
const p = 52 / 10000, se = Math.sqrt(p * (1 - p) / 10000);
[p - 1.96 * se, p + 1.96 * se]          // ≈ [0.38%, 0.66%]

What 95% means: if you repeated the whole procedure many times, about 95% of the intervals would contain the true value. It is not "95% probability the true value is in this particular interval" (that is the Bayesian credible interval of part 7, which often comes out similar).

HOW UNCERTAIN IS A p99 FROM 2,000 SAMPLES?

bootstrap 95% interval from the simulated latency data

swipe the figure sideways, or tap expand for full screen

1/4

the estimate

From 2,000 requests the p99 estimate is 511 ms. It rests on only about 20 requests in the tail.

511 ms from ~20 tail pointsfew points decide it