Part 9 · 1 chapters · ~12 min

Dependencies and Graceful Degradation

How one slow dependency becomes an outage and how to stop it: timeouts and deadlines, retries with backoff, jitter and a budget, circuit breakers, bulkheads and load shedding, and degradation behaviour decided per dependency before the incident, with the UI states it needs.

14

Keeping a slow dependency contained

code
// a minimal circuit breaker with a timeout and a retry budget
class Breaker {
  private failures = 0; private openedAt = 0; private state: 'closed' | 'open' | 'half' = 'closed';
  constructor(private threshold = 5, private coolMs = 10_000) {}
  async call<T>(fn: (signal: AbortSignal) => Promise<T>, timeoutMs: number): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.openedAt < this.coolMs) throw new Error('circuit open');   // fail fast
      this.state = 'half';                                                             // let a trial through
    }
    try {
      const r = await fn(AbortSignal.timeout(timeoutMs));
      this.failures = 0; this.state = 'closed'; return r;
    } catch (e) {
      if (++this.failures >= this.threshold || this.state === 'half') { this.state = 'open'; this.openedAt = Date.now(); }
      throw e;
    }
  }
}

const retryTokens = { n: 0, max: 0 };                     // refilled at 10% of request volume each second
async function withRetry<T>(fn: () => Promise<T>, idempotent: boolean): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try { return await fn(); }
    catch (e) {
      if (!idempotent || attempt >= 2 || retryTokens.n <= 0) throw e;
      retryTokens.n--;
      await sleep(Math.random() * 100 * 2 ** attempt);    // full jitter
    }
  }
}
dependencytimeoutretry?without it, the product
bank rails1.5 s per callwith idempotency keyaccepts as pending; status page and push on completion
KYC vendor5 syes, read callssaves progress; "we will notify you"
exchange rates500 msyesshows last rate with its age; blocks trades past 30 s
analytics200 ms, fire and forgetnonothing visible
the frontend's half
A degradation plan only works if the UI has the states. "Pending", "rate is 40 s old" and "statements temporarily unavailable" are screens someone must design and build, which makes this a frontend staff conversation as much as a backend one.
DEPENDENCIES AND GRACEFUL DEGRADATION
timeouts, retries with budgets, circuit breakers, bulkheads and load shedding: how one slow dependency stays one slow dependency
swipe the figure sideways, or tap expand for full screen
1/6
the cascade
The cascade: the bank adapter goes from 200 ms to 20 s. With no timeout, every transfer request holds a connection for 20 s; the pool fills; now balance checks, which do not even touch the bank, fail too. One slow dependency became a full outage.