Part 3 · 1 chapters · ~12 min
Incident Response
Running an incident with structure: declaring early, the commander who coordinates, operations making announced changes, mitigation before diagnosis, communication on a cadence, explicit resolution and handover, with severity definitions and customer wording for a fintech.
6
Roles, severity, communication and runbooks
run it, do not fight it
- Declare early, with a severity, a channel and a timeline document.
- The commander coordinates and does not debug.
- Operations: one change at a time, announced before it is made.
- Mitigate first: roll back, fail over, switch, disable, shed load.
- Communications on a fixed cadence. Silence is the thing users do not forgive.
- Resolve explicitly, then assign the postmortem and file follow-ups.
| severity | definition (a fintech) | response | comms |
|---|---|---|---|
| SEV1 | money movement down or wrong for many users; data exposure; regulatory impact | all hands, exec informed, 24/7 until resolved | status page within 15 min; updates every 30 min; regulator notified per rules |
| SEV2 | a key journey degraded (transfers slow, one bank failing); budget burning fast | on-call + owning team; commander assigned | status page if customer-visible; updates hourly |
| SEV3 | minor degradation, workaround exists, no money at risk | owning team in working hours | internal channel |
| SEV4 | cosmetic or internal-only | ticket | none |
code
STATUS PAGE (customer-facing, plain words) Investigating: Transfers to some banks are delayed Some transfers to GTBank and Access Bank are taking longer than usual to arrive. Your money is safe. Transfers that are delayed will either complete or be reversed automatically, so please don't retry them. Next update by 14:30 WAT.
the frontend's role in an incident
The app is the first place users see an incident. The Trust course part 7 designs the degraded banner and the "confirming" states, and the incident runbook includes the flag that turns the banner on. Turning on the banner is often the very first mitigation.
RUNNING AN INCIDENT
from the page to resolution: declaring, roles, mitigation first, communication, and handing back
swipe the figure sideways, or tap expand for full screen
1/6
declare
Declare early: the on-call engineer is paged, confirms user impact, and declares an incident with a severity, opening a channel (#inc-2026-10-06-transfers) and a timeline doc. Declaring is cheap and can be downgraded; under-declaring delays help. If in doubt, declare.