Part 5 · 1 chapters · ~8 min
Secret Rotation Without Downtime
Why one-step rotation breaks clients, the overlap pattern with two valid credentials, rotation for database passwords, API keys, signing keys (key ids and JWKS) and TLS certificates, verification before revocation, and automating rotation so it is routine.
9
The overlap pattern, by secret type
| secret | overlap mechanism |
|---|---|
| database password | two database users alternated, or dynamic per-pod credentials from Vault |
| third-party API key | providers that allow two active keys; switch, verify, delete |
| JWT signing key | publish both public keys in JWKS with key ids (kid); sign with the new one; drop the old after the longest token lifetime |
| TLS certificate | issue the new cert before expiry (cert-manager, ACME), reload without restart; short lifetimes force automation |
| encryption key | versioned keys (Vault transit, KMS): decrypt with any version, encrypt with the newest, rewrap in the background |
ROTATING A SECRET WITH ZERO DOWNTIME
two valid credentials overlap; nothing ever holds only an invalid one
swipe the figure sideways, or tap expand for full screen
1/6
the problem
Changing a password in one step breaks every client that still has the old one: some pods restart before others, some caches hold the old value.
single-step rotation breaks running clientsthere is always a laggard