Part 9 · 1 chapters · ~8 min

Capstone: A Workflow Engine in Miniature

Building a small durable workflow engine on Postgres: workflow and step tables, claiming with SKIP LOCKED and leases, retries with backoff, timers, compensations, idempotent steps, a status API, and tests that kill workers mid-workflow.

The build

code

mini-workflows/
  schema.sql        workflows(id, type, state, input, version) · steps(workflow_id, name, state, attempts, result, next_run_at, lease_until)
  engine.ts         define('crossBorder', [step('debit', debit, refund), step('fx', bookFx, reverseFx), step('credit', creditMpesa)])
  runner.ts         loop: claim due steps (FOR UPDATE SKIP LOCKED, lease 60 s) → run → record result or schedule retry with jitter
  compensate.ts     on permanent failure: run compensations of completed steps in reverse, each idempotent
  timers.ts         step.sleep('24h') = next_run_at in the future; no thread held
  api.ts            GET /workflows/:id → state, steps, attempts, last error
tests
  - kill -9 a runner mid-step: another runner reclaims after the lease and the step runs at most once in effect
  - a permanently failing credit step triggers reverse compensations exactly once each
  - 1,000 concurrent workflows complete with 4 runners; measure throughput and the oldest-step age