Part 9 · 1 chapters · ~8 min

Capstone: Rotate a Database Password Under Load

The capstone: run a service at steady load, rotate its database credentials with the overlap pattern (or switch it to Vault dynamic credentials), and prove zero failed requests with metrics.

The exercise

code

setup
  ledger service (any language) → PgBouncer → Postgres, load generator at 500 req/s, dashboards for errors and p99
  two database roles: ledger_blue (active), ledger_green (created, same grants)

run
  1 set ledger_green's new password; store it in the secrets manager as version N+1
  2 consumers reload credentials from the mounted file (or rolling restart); the pool drains old connections
  3 verify: SELECT usename, count(*) FROM pg_stat_activity GROUP BY 1;   → only ledger_green
  4 ALTER ROLE ledger_blue NOLOGIN; then rotate blue's password for next time
  5 evidence: error rate flat, p99 flat, zero authentication failures in the Postgres log

stretch
  replace both roles with Vault database secrets engine credentials (1 h leases) and repeat the measurement