Queues absorb spikes and decouple services — until a poison message or a thundering retry storm takes the whole tier down.
Design for retries
Every job is idempotent and carries an idempotency key, so running it twice is harmless. Failures back off exponentially with jitter.
- Idempotency keys so duplicate delivery is safe
- Dead-letter queues for messages that keep failing
- Backpressure so a slow consumer cannot melt the producer
Observe the backlog
We alert on queue age, not just depth — a growing oldest-message age is the earliest sign something downstream is stuck.