Message Queues vs Pub/Sub

A message broker is a server that receives messages from producers and delivers them to consumers, decoupling the two in time, in rate, and in knowledge of each other. A producer sends without waiting for the consumer to be running, and the broker stores the message until it is delivered. Two delivery models exist: the message queue (point-to-point) and publish/subscribe.

Message queue (point-to-point)

A queue stores messages in order. Each message is delivered to exactly one consumer, even when several consumers read the same queue. Several consumers on one queue are competing consumers: they share the work, and adding consumers increases throughput.

graph LR
    P1["Producer A"] --> Q["Queue"]
    P2["Producer B"] --> Q
    Q --> C1["Consumer 1"]
    Q --> C2["Consumer 2"]
    Q --> C3["Consumer 3"]

Typical uses: distributing tasks (image processing, email sending, order fulfilment), smoothing traffic bursts so that downstream systems run at a steady rate, and isolating a slow component from a fast one.

Publish/subscribe

In publish/subscribe (pub/sub), a producer publishes to a topic. Every subscription on the topic receives its own copy of each message. Within one subscription, consumers may compete, so each subscription behaves as a queue.

graph LR
    P["Publisher"] --> T["Topic: order-created"]
    T --> S1["Subscription: billing"]
    T --> S2["Subscription: shipping"]
    T --> S3["Subscription: analytics"]
    S1 --> B1["Billing consumers"]
    S2 --> B2["Shipping consumers"]
    S3 --> B3["Analytics consumers"]

The publisher does not know how many subscribers exist. Adding a new subscriber requires no change to the publisher. Typical uses: broadcasting events to several services, cache invalidation, and fan-out notifications.

Many systems combine both: RabbitMQ binds several queues to one exchange (pub/sub across queues, competing consumers within each queue), and Kafka consumer groups give each group its own copy of the stream, with competition among the members of a group. See Kafka internals and RabbitMQ, SQS, and broker comparison.

Push and pull delivery

Brokers deliver in one of two ways. In push delivery the broker sends messages to connected consumers, up to a limit on unacknowledged messages per consumer (RabbitMQ prefetch). In pull delivery the consumer requests messages when it has capacity (SQS receive calls, Kafka fetch requests). Pull lets each consumer control its own rate, so a slow consumer accumulates backlog in the broker instead of memory. Push gives lower delivery latency but needs a per-consumer limit to avoid overwhelming a slow consumer.

Acknowledgements

A broker must know when a message has been processed, so it can delete it or hand it to someone else after a failure.

1. The broker delivers the message to a consumer.
2. The consumer processes it.
3. The consumer sends an acknowledgement (ack). The broker then removes the message (or advances the position).
4. If the consumer fails before acking, the broker redelivers the message to a consumer.

Acknowledging before processing (auto-ack) means a crash loses the message: at-most-once. Acknowledging after processing means a crash between processing and ack causes redelivery and a duplicate: at-least-once. A negative acknowledgement (nack) signals failure and requests redelivery or dead-lettering. See delivery semantics and idempotency.

Visibility timeout

Brokers such as Amazon SQS do not hold a per-consumer connection state. Instead, when a consumer receives a message, the message becomes invisible to other consumers for a visibility timeout.

sequenceDiagram
    participant P as Producer
    participant Q as Queue
    participant C1 as Consumer 1
    participant C2 as Consumer 2
    P->>Q: send M
    C1->>Q: receive
    Q-->>C1: M (invisible for timeout)
    Note over C1: crashes before delete
    Note over Q: timeout expires, M visible again
    C2->>Q: receive
    Q-->>C2: M
    C2->>Q: delete M

• If the consumer deletes the message before the timeout, processing is complete.
• If the timeout expires first, the message reappears and another consumer receives it.
• Set the timeout longer than the normal processing time. If it is too short, healthy but slow processing yields duplicate deliveries. If it is too long, a crashed consumer's messages are delayed. Some brokers let a consumer extend the timeout while working.

Connection-oriented brokers such as RabbitMQ instead requeue unacknowledged messages when the consumer's channel or connection closes, and can also enforce a delivery acknowledgement timeout.

Ordering

• A single queue with a single consumer preserves order.
• Competing consumers do not preserve processing order: consumer 2 may finish message 2 before consumer 1 finishes message 1, and a redelivered message arrives after later ones.
• To keep order where it matters, partition the stream by a key (for example the account id). All messages with one key go to the same partition or message group, which is processed by one consumer at a time. Order is then guaranteed per key, not globally. Kafka partitions and SQS FIFO message group ids implement this.
• Global ordering limits throughput to what one consumer can process.

Delivery guarantees in practice

At-least-once with idempotent consumers is the standard design. A consumer must tolerate the same message twice.

Queue or pub/sub

• Work that one worker should perform once: queue.
• The same fact that several services react to independently: pub/sub (or a log with several consumer groups).
• Need to replay history, or add a consumer that reads old messages: a retained log rather than a delete-on-ack queue.
• Need per-message routing, priorities, delayed delivery: a queue broker with those features.

Interview questions

• What is the difference between competing consumers and multiple subscriptions? Competing consumers share one stream, so each message is handled once by one of them. Separate subscriptions each receive every message.
• What happens if a consumer crashes after processing but before acknowledging? The broker redelivers the message, so it is processed twice. Consumers must be idempotent under at-least-once delivery.
• How is ordering preserved with multiple consumers? Partition messages by a key so all messages for a key go to one consumer at a time. Order holds per key, not across keys.
• Why is the visibility timeout a trade-off? A short one causes duplicate processing for slow messages; a long one delays retry after a consumer crash.