Part 6 · 2 chapters · ~18 min

Failure Detection and Idempotency

Turning silence into suspicion: heartbeats, the timeout trade-off, phi accrual, gray failure, leases and fencing tokens; then why exactly-once delivery is impossible and exactly-once effect is engineering: idempotency keys, natural idempotency, consumer dedup and Kafka's transactional boundary.

Heartbeats, timeouts, leases and fencing

silence is a suspicion, not a fact

Heartbeats with a missed-count threshold.
The timeout trade-off: a short one causes false failovers, a long one detects real failures slowly.
Adaptive detection: a phi accrual detector measures suspicion relative to normal behaviour.
Gray failure: check what actually matters, and eject outliers based on real traffic.
Leases: authority that expires unless renewed.
Fencing tokens: storage rejects any write from a stale holder.

code

-- fencing at the storage layer: the lock service hands out increasing tokens
UPDATE jobs SET owner = $1, fence = $2, state = 'running'
WHERE id = $3 AND fence < $2;          -- a write carrying an older token updates 0 rows

-- a paused worker with token 33 wakes after worker B (token 34) took over:
-- its write matches 0 rows, and it must abandon the job

FAILURE DETECTION

heartbeats, timeouts, the phi accrual detector, leases, and why "dead" is always a guess

swipe the figure sideways, or tap expand for full screen

1/6

heartbeats

Heartbeats: each node sends "I am alive" every second; a monitor marks a node suspect after N missed heartbeats. Simple and universal (Kubernetes node status, load balancer health checks, Raft). The question is what N should be.

Idempotency and exactly-once in effect

duplicates are certain; make them harmless

At-most-once loses messages.
At-least-once duplicates them.
Idempotency keys store the result together with the effect.
Prefer natural idempotency: set, upsert, conditional writes.
Consumer dedup: an inbox table written in the same transaction as the effect.
Kafka's exactly-once stops at Kafka's boundary.

code

// an idempotency-key middleware: the key and the effect commit together
app.post('/transfers', async (req, res) => {
  const key = req.header('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'Idempotency-Key required' });

  const result = await db.tx(async t => {
    const prior = await t.oneOrNone(
      'INSERT INTO idem(key, request_hash) VALUES ($1, $2) ON CONFLICT (key) DO NOTHING RETURNING key',
      [key, hash(req.body)]);
    if (!prior) {                                        // seen before: return what we returned then
      const row = await t.one('SELECT request_hash, response FROM idem WHERE key = $1 FOR UPDATE', [key]);
      if (row.request_hash !== hash(req.body)) throw new Conflict('key reused with a different body');
      return row.response;                               // may be null if the first attempt is still running
    }
    const transfer = await createTransfer(t, req.body);   // the effect
    await t.none('UPDATE idem SET response = $2 WHERE key = $1', [key, transfer]);
    return transfer;
  });
  res.status(201).json(result);
});

the frontend's part

The key is generated when the user starts the action (the form mounts, or Pay is first pressed) and reused for every retry of that action, including after a page refresh if it is persisted. A key generated on each click defeats the whole mechanism. The Trust course part 3 builds this into the deposit and withdrawal flows.

EXACTLY-ONCE IN EFFECT

at-most-once, at-least-once, and the idempotency that turns duplicates into no-ops

swipe the figure sideways, or tap expand for full screen

1/6

at-most-once

At-most-once: send once, never retry. Nothing is ever duplicated and some messages are lost. Acceptable for metrics samples and fire-and-forget telemetry; never for money or orders.