Part 6 · 1 chapters · ~10 min

Toil and Automation

Toil defined and recognised, measured for two weeks and ranked by frequency, time, risk and interruption; automated up a ladder from runbook to fully automatic, with causes fixed and product gaps sent to product; tracked with a target.

10

Measuring toil and what to automate first

manual work that scales with the service is a bug
  1. Recognise toil: work that would disappear if the system were better engineered.
  2. Measure it for two weeks, then sum it by category.
  3. Rank it by frequency × time, weighted by risk and interruption.
  4. Climb the automation ladder: runbook, script, tool, approved automation, fully automatic.
  5. Fix the cause, and send product gaps to product.
  6. Track it with a target. Toil that grows with traffic is a scaling bug.
code
// from runbook to self-service: the stuck-transfer reversal as a safe internal tool
// before: an engineer runs 4 SQL statements by hand at 2 a.m.
export async function reverseStuckTransfer(transferId: string, actor: Staff) {
  const t = await transfers.get(transferId);
  if (t.status !== 'PENDING' || ageMinutes(t) < 30) throw new Refuse('not stuck');
  const psp = await psps[t.provider].status(t.providerRef);       // ask, never guess (Distsys part 0)
  if (psp === 'SUCCESS') return transfers.markSucceeded(t, { actor });
  if (psp === 'PROCESSING') throw new Refuse('provider still processing: retry later');
  return ledger.reverse(t, { idempotencyKey: `${t.id}:reverse`, actor, reason: 'stuck > 30m, PSP failed' });
}
// audited, idempotent, refuses unsafe cases, usable by support with approval
TOIL AND WHAT TO AUTOMATE FIRST
recognising toil, measuring it, and choosing automation by frequency, time and risk
swipe the figure sideways, or tap expand for full screen
1/6
what toil is
What toil is: manually reversing a stuck transfer, restarting a consumer that wedges every few days, rotating a certificate by hand, provisioning a merchant account by running a script, answering the same "why was I charged" ticket. Not toil: designing the reversal tool, writing a postmortem, improving an alert. The test: would it disappear if the system were better engineered?