Chat and Presence
One-to-one and group chat with persistent connections, ordered delivery per conversation, offline sync, storage partitioned by conversation, and approximate presence.
Brief, questions and numbers
- Users send messages to individuals and groups in real time, see delivery and read receipts, and see who is online.
| question | answer we assume |
|---|---|
| users? | 20M daily active, 5M concurrent connections at peak |
| messages? | 1B/day |
| ordering? | per conversation, not global |
| history? | kept for years, synced across devices |
| groups? | up to 1,000 members |
messages = 1B/day ≈ 11,600/s average, ~40,000/s peak connections= 5M concurrent / 100k per gateway node ≈ 50 gateway nodes storage = 1B × 300 B ≈ 300 GB/day ≈ 110 TB/year (replicated ×3) presence = 5M keys refreshed every 30 s ≈ 170k Redis writes/s: shard it
v1, the break, and v2
v1. Clients poll a REST endpoint every few seconds; messages live in one Postgres table indexed by conversation and time.
The break. Polling 20M users every 3 seconds is millions of requests per second of mostly empty responses, and a single Postgres table cannot absorb 40k writes per second plus history reads.
v2. WebSocket gateways with a user-to-gateway registry, a message service assigning per-conversation sequence numbers, a wide-column store partitioned by conversation, pub/sub between gateway nodes, push notifications for offline users, and TTL-based presence in sharded Redis.