10 parts · 12 chapters
Jobs, Queues and Workflows
Most important backend work does not happen inside a request: receipts, reconciliation, payouts, statement generation, retries against flaky partners. Background work is where reliability is won or lost, because failures happen when nobody is watching.
Ten parts: background jobs and why they exist; delivery semantics and idempotent consumers; retries, backoff and dead letters; RabbitMQ and SQS; cron and scheduling at scale; sagas; durable execution with Temporal; the outbox and inbox patterns; observability for asynchronous work; and a capstone building a small workflow engine.
jobsEnqueue ids, idempotent handlers, priorities and concurrency limits.
semanticsAt-most-once, at-least-once, exactly-once in effect.
failureRetries with backoff and jitter, poison messages, dead letters.
brokersRabbitMQ exchanges and acks, SQS visibility timeouts and FIFO.
workflowsSagas, compensation, Temporal's durable execution.
reliabilityOutbox and inbox, observability, and the capstone engine.
00
Background Jobs and Why They Exist
Requests stay fast; work happens later · Queues, priorities and limits
2 ch · ~12 min01Delivery Semantics and Idempotent Consumers
Three semantics, one practical answer
1 ch · ~8 min02Retries, Backoff and Dead Letters
Retry well, then give up well
1 ch · ~8 min03RabbitMQ and SQS
Two brokers
1 ch · ~8 min04Cron and Scheduling at Scale
Firing once, on time
1 ch · ~8 min05Sagas
Steps and compensations · When compensation fails
2 ch · ~12 min06Durable Execution with Temporal
Code that survives crashes
1 ch · ~8 min07Outbox and Inbox
Both ends of a reliable message
1 ch · ~8 min08Observability for Asynchronous Work
Seeing the invisible
1 ch · ~8 min09Capstone: A Workflow Engine in Miniature
The build
1 ch · ~8 minBuilt on the Production Stack and KafkaProduction Stack part 5 introduced worker libraries per language; Kafka and Redis cover two common backends; Ledgers covers payout state machines.