Part 5 · 1 chapters · ~8 min

Background Workers

The worker pattern across runtimes (BullMQ, Celery, Sidekiq, Solid Queue, asynq, Spring Batch and @Async), enqueuing ids, retries with backoff, dead letters, scheduling with single-fire locks, and sizing workers separately from web.

One pattern, every language

runtime	library	backend
Node	BullMQ	Redis
Python	Celery (+ beat), RQ, Dramatiq	Redis or RabbitMQ
Ruby	Sidekiq; Solid Queue (Rails 8 default)	Redis; the database
Go	asynq, River	Redis; Postgres
Java	Spring Batch, @Async with executors, JobRunr, Quartz	database, or a broker
Elixir	Oban	Postgres
any, at scale	Temporal (durable workflows)	its own cluster (course 22)

code

# Celery: an idempotent task with retries
@app.task(bind=True, autoretry_for=(TimeoutError,), retry_backoff=True, retry_jitter=True, max_retries=8)
def send_receipt(self, transfer_id):
    t = Transfer.objects.get(pk=transfer_id)
    if t.receipt_sent_at: return                          # idempotent: already done
    email.send(t.owner.email, render_receipt(t))
    Transfer.objects.filter(pk=t.pk, receipt_sent_at=None).update(receipt_sent_at=now())

BACKGROUND WORKERS

the same codebase, a different entry point, pulling jobs from a queue

swipe the figure sideways, or tap expand for full screen

1/5

enqueue

The web process records the job (type, arguments, idempotency key) in the queue and returns. Arguments should be ids, not whole objects, so workers load fresh data.

enqueue ids, not objectsworkers load fresh state