Part 7 · 1 chapters · ~8 min
Streams, Pub/Sub and Redis as a Queue
Lists as simple queues with BRPOP and BLMOVE (reliable queue pattern), pub/sub and its fire-and-forget semantics, streams with consumer groups, pending lists and claiming, how BullMQ, Sidekiq and Celery use Redis, and when to move to a real broker.
9
Queues, pub/sub and streams
code
BLMOVE queue:receipts processing:w1 RIGHT LEFT 5 # reliable queue: move atomically to a processing list
LREM processing:w1 1 <job> # remove when done (a reaper re-queues stuck jobs)
PUBLISH rates '{"NGN-USD":"1500.00"}' # pub/sub: delivered to connected subscribers only, never stored
XADD events MAXLEN ~ 1000000 * type transfer.completed id t_81
XGROUP CREATE events notifier $ MKSTREAM
XREADGROUP GROUP notifier w1 COUNT 10 BLOCK 5000 STREAMS events >
XACK events notifier 1728390000000-0
XAUTOCLAIM events notifier w2 60000 0-0 COUNT 10 # take over entries pending > 60 s| mechanism | delivery | persistence | use |
|---|---|---|---|
| pub/sub | at most once, to connected subscribers | none | cache invalidation broadcasts, live updates |
| lists | at least once with the BLMOVE pattern | with AOF | simple job queues (Sidekiq, RQ) |
| streams | at least once, consumer groups, acks | with AOF, capped | event logs, BullMQ-style queues |
STREAMS AND CONSUMER GROUPS
an append-only log with acknowledgements inside Redis
swipe the figure sideways, or tap expand for full screen
1/5
append
Producers XADD entries with auto-generated time-based IDs; the stream keeps them in order (capped with MAXLEN to bound memory).
XADD appends with ordered IDsMAXLEN ~ caps length