Part 3 · 2 chapters · ~16 min
Failure Analysis
FMEA on the clearance checker: failure modes, effects, severity, occurrence and detection scores, risk priority numbers, actions that drive the top RPN from 450 to 30, and the severity rule; then a fault tree from the never-allowed outcome down to basic events, AND gates from independent checks, and a safety case.
4
FMEA
bottom-up: every way each part can fail
- Failure modes, listed component by component.
- Effects, traced out to the customer.
- Scores for severity, occurrence and detection.
- RPN is their product. Sort by it.
- Actions lower occurrence, or improve detection.
- The severity rule: a 9 or 10 always gets action, whatever the RPN.
| score | severity | occurrence | detection |
|---|---|---|---|
| 1-2 | no noticeable effect | almost never (< 1 in 100,000) | caught automatically before any customer impact |
| 3-4 | minor inconvenience, workaround | rare | caught by monitoring within minutes |
| 5-6 | degraded service, support contacts | occasional | caught by a daily check or reconciliation |
| 7-8 | money or data wrong for a customer | frequent | caught only when a customer complains |
| 9-10 | regulatory breach, irreversible loss, safety | almost certain | probably never caught |
FMEA: FAILURE MODES AND EFFECTS
every component, every way it can fail, scored for severity, occurrence and detection, before anyone writes code
swipe the figure sideways, or tap expand for full screen
1/6
failure modes
Pick a component: the clearance checker, which decides whether a loan is fully cleared. List its failure modes: it reads a stale balance from a lagging replica; it ignores fees; it treats a received but reversible payment as settled; it times out; it approves a joint loan with only one borrower cleared.
5
Fault trees, blast radius and safety cases
top-down: what could cause the outcome you must never allow
- The top event is one undesired outcome.
- OR gates: any one cause is enough.
- Single points of failure show up as basic events that reach the top on their own.
- AND gates from independent checks multiply probabilities together.
- Tamper evidence means an attacker needs two independent compromises.
- The safety case: an argument with evidence that others can attack.
blast radius
The Cloud Engineering course part 1 measures blast radius in failure domains. This part measures it in consequences. A bug that delays letters is a ticket. A bug that issues one wrong letter is a regulatory event. FMEA severity and fault-tree top events are how that difference gets written down before design, rather than after the incident.
FAULT TREES AND SAFETY CASES
starting from the outcome you must never allow and working down to the basic events that cause it
swipe the figure sideways, or tap expand for full screen
1/6
top event
The top event: "a clearance letter is issued for a customer who still owes money". Everything in the tree is a way this can happen.