Part 9 · 1 chapters · ~12 min
Dependencies and Graceful Degradation
How one slow dependency becomes an outage and how to stop it: timeouts and deadlines, retries with backoff, jitter and a budget, circuit breakers, bulkheads and load shedding, and degradation behaviour decided per dependency before the incident, with the UI states it needs.
14
Keeping a slow dependency contained
code
// a minimal circuit breaker with a timeout and a retry budget
class Breaker {
private failures = 0; private openedAt = 0; private state: 'closed' | 'open' | 'half' = 'closed';
constructor(private threshold = 5, private coolMs = 10_000) {}
async call<T>(fn: (signal: AbortSignal) => Promise<T>, timeoutMs: number): Promise<T> {
if (this.state === 'open') {
if (Date.now() - this.openedAt < this.coolMs) throw new Error('circuit open'); // fail fast
this.state = 'half'; // let a trial through
}
try {
const r = await fn(AbortSignal.timeout(timeoutMs));
this.failures = 0; this.state = 'closed'; return r;
} catch (e) {
if (++this.failures >= this.threshold || this.state === 'half') { this.state = 'open'; this.openedAt = Date.now(); }
throw e;
}
}
}
const retryTokens = { n: 0, max: 0 }; // refilled at 10% of request volume each second
async function withRetry<T>(fn: () => Promise<T>, idempotent: boolean): Promise<T> {
for (let attempt = 0; ; attempt++) {
try { return await fn(); }
catch (e) {
if (!idempotent || attempt >= 2 || retryTokens.n <= 0) throw e;
retryTokens.n--;
await sleep(Math.random() * 100 * 2 ** attempt); // full jitter
}
}
}| dependency | timeout | retry? | without it, the product |
|---|---|---|---|
| bank rails | 1.5 s per call | with idempotency key | accepts as pending; status page and push on completion |
| KYC vendor | 5 s | yes, read calls | saves progress; "we will notify you" |
| exchange rates | 500 ms | yes | shows last rate with its age; blocks trades past 30 s |
| analytics | 200 ms, fire and forget | no | nothing visible |
the frontend's half
A degradation plan only works if the UI has the states. "Pending", "rate is 40 s old" and "statements temporarily unavailable" are screens someone must design and build, which makes this a frontend staff conversation as much as a backend one.
DEPENDENCIES AND GRACEFUL DEGRADATION
timeouts, retries with budgets, circuit breakers, bulkheads and load shedding: how one slow dependency stays one slow dependency
swipe the figure sideways, or tap expand for full screen
1/6
the cascade
The cascade: the bank adapter goes from 200 ms to 20 s. With no timeout, every transfer request holds a connection for 20 s; the pool fills; now balance checks, which do not even touch the bank, fail too. One slow dependency became a full outage.