Part 8 · 1 chapters · ~8 min

Config Incidents and How to Prevent Them

The recurring shapes of configuration outages (global pushes, missing keys, leaked secrets, expired certificates, stale flags, drift), the guard for each, and a config incident review template.

13

Six incident shapes

CONFIG INCIDENTS, AND THE GUARD THAT STOPS EACH
patterns behind real outages
bad value, global pushA typo pushed to every region atonce. Guard: staged rollout ofconfig, like code.missing keyA new required key absent in oneenvironment. Guard: schemavalidation at boot and in CI.secret in a logConfig dumped on startup includingpasswords. Guard: redaction andsecret types that refuse to print.expired certificateA cert nobody owned expired at 2a.m. Guard: short-lived automatedcerts and expiry alerts.stale flagA two-year-old flag flipped bymistake re-enabled dead code.Guard: flag expiry and removal.driftProd edited by hand differs fromGit. Guard: GitOps reconciliationand drift alerts.
swipe the figure sideways, or tap expand for full screen
1/6
global push
Large cloud outages have come from a single configuration change pushed everywhere at once. The defence is the same as for code: canary by region, watch, then continue.
stage config rolloutsregion by region, with automatic halt