Part 9 · 2 chapters · ~12 min

Distributed Job Scheduler

Scheduling millions of delayed and recurring jobs: durable job records, claiming with SKIP LOCKED and leases, at-least-once execution with idempotent jobs, retries, cron expansion, partitioning by due time, and fairness between tenants.

19

Brief, questions and numbers

the brief
  1. Services schedule one-off jobs (send a reminder in 3 days) and recurring jobs (reconcile daily at 01:00) that must run on time, even through crashes.
questionanswer we assume
volume?50M scheduled jobs per day
precision?within seconds
duplicates?tolerable if jobs are idempotent; never silently skipped
tenancy?one tenant must not starve others
visibility?operators can see, retry and cancel jobs
code
jobs       = 50M/day ≈ 580/s average, bursts of 50k due at the top of each hour
rows       = 50M/day; keep 30 days ≈ 1.5B rows → partition by due date, drop old partitions
claim rate = 50k in a burst / 20 dispatchers × 100 per query ≈ 25 queries per dispatcher
DISTRIBUTED JOB SCHEDULER
durable job records, time-bucketed polling, leases, and at-least-once execution
schedule APIrun at / cronjobs tabledue_at, state, leasedispatchers × Nclaim due jobswork queueper job typeworkersexecute, heartbeatfailed / DLQafter retries
swipe the figure sideways, or tap expand for full screen
1/5
durable record
Scheduling writes a job row: type, payload, due_at, state = scheduled. The database is the source of truth, so a crash loses nothing.
every job is a durable rowcrashes lose nothing
20

v1, the break, and v2

v1. A single cron process on one server reads a crontab and runs jobs in-process.

The break. One machine is a single point of failure, jobs are lost on restart, nothing scales past one box, and there is no visibility or retry.

v2. A jobs table partitioned by due date, dispatchers claiming with FOR UPDATE SKIP LOCKED and leases, typed work queues, idempotent workers with heartbeats, backoff retries and a failed state, cron expansion, per-tenant concurrency limits, and an operator UI.

the sentence
v2 buys durable, horizontally scaled, observable scheduling, and pays with at-least-once semantics every job must handle and a busy jobs table to keep healthy.