What Message Queues Solve: Async, Buffering, Decoupling

Three values

Typical shape

Producer to queue (durable) to consumer. The queue buffers in the middle; the producer can fail without losing work, and a slow consumer will not crush upstream.

Two caveats to settle

1. Duplicate consumption: retries can deliver a message twice, so the consumer must be idempotent (dedup table, state machine);
2. Ordering: one queue is usually ordered, but after sharding the same business key must route to the same shard to keep order.

Versus RPC

RPC is a synchronous "I need the result now" call; a queue is an asynchronous "handle it later" notice. Deduct stock on order via RPC, send the email via queue.

Real-world cases: three queue-induced incidents

1. Duplicate messages causing double charges: most queues guarantee at-least-once delivery, so consumers must be idempotent (a business unique key or dedup table).
2. Out-of-order handling: consuming partitions in parallel for throughput scrambled "create" and "pay" for the same order. When ordering matters, route the same ordering key to the same partition.
3. Backlog nobody noticed: with no queue-length monitoring, millions of messages piled up until a disk alert. Track backlog and consumer lag, and autoscale consumers.

FAQ

Can messages be lost? It depends on configuration: durability, replication and acknowledgements must all be right, or a restart can lose them. What is a dead-letter queue for? Holding messages that failed repeated retries, so the main queue is not blocked and humans can investigate. When should I not use a queue? When you need a synchronous result, strong consistency or minimal latency, a direct call fits better. How do I get exactly-once? End to end it is impractical; teams approximate it with at-least-once delivery plus idempotent consumers.

Message design: idempotency keys and retry policy

1. Give messages an idempotency key: use a business identity (order ID plus action) for deduplication rather than the message ID, which can differ across redeliveries;
2. A dedup table on the consumer: a unique index that makes "insert equals dedupe" is more reliable under concurrency than check-then-write;
3. Tier your retries: retryable failures (network blips, downstream throttling) with exponential backoff; non-retryable ones (invalid input, business rejection) go straight to the dead-letter queue rather than being retried pointlessly;
4. Bound timeouts and attempts: the timeout must be shorter than the caller's patience and retries must be capped, or one wobble becomes an avalanche;
5. Ordering versus parallelism: partition by business key when ordering matters; otherwise consume in parallel for throughput instead of serialising "just in case".

The queue itself needs observability too: backlog, consumer lag, retry rate and dead-letter growth belong on dashboards and alerts, or you discover problems only when a disk alarm fires.

Combining with business consistency

1. Keep local writes and publishing consistent: if the write and the publish are not in one transaction you get data without a message or the reverse. Writing to a local outbox and publishing from there guarantees eventual delivery.
2. Allow human intervention: beyond a dead-letter queue, provide authorised views, replays and discards so incidents do not require editing the database directly.
3. Have a fallback for critical paths: if the queue is down for long, decide explicitly whether to reject the request or degrade to synchronous handling instead of leaving callers hanging.
4. Keep payloads lean: send identifiers and necessary fields, letting consumers fetch current state, so stale values never ride along in the message.
5. Version for compatibility: producers and consumers release independently, so add fields as optional and remove them in stages.

Queues turn synchronous calls into asynchronous collaboration and move consistency responsibility to the business side. Agreeing these conventions up front gives you the decoupling without intractable data problems.