RabbitMQ, SQS, and Broker Comparison

Message brokers differ in their core model. Some are smart brokers that route messages with rich rules and track per-message state (RabbitMQ, SQS). Others are log-based brokers that append records to a retained log and leave position tracking to consumers (Kafka). The model determines ordering, replay, routing, and scaling behavior. The general models are described in message queues vs pub/sub; log internals in Kafka internals.

RabbitMQ

RabbitMQ is a broker that implements AMQP 0-9-1 (and supports other protocols such as MQTT and STOMP through plugins). Producers never send directly to a queue. They publish to an exchange, which routes each message to zero or more queues according to bindings.

graph LR
    P["Producer"] -->|"routing key: order.eu.created"| X["Topic exchange"]
    X -->|"binding: order.*.created"| Q1["Queue: billing"]
    X -->|"binding: order.eu.#"| Q2["Queue: eu-shipping"]
    X -->|"binding: audit.#"| Q3["Queue: audit"]
    Q1 --> C1["Billing consumers"]
    Q2 --> C2["Shipping consumers"]

A binding links an exchange to a queue with a binding key or pattern. The default (nameless) exchange is a direct exchange that routes a message to the queue whose name equals the routing key, which lets a producer appear to publish directly to a queue.

Key mechanics:

• Queues hold messages until consumers acknowledge them. Consumers receive messages pushed by the broker, limited by a prefetch count (the number of unacknowledged messages allowed per consumer), which provides flow control. With prefetch 1, a consumer receives a new message only after acknowledging the previous one, so slow consumers do not hoard messages while others are idle. With prefetch 100, a slow consumer can hold up to 100 messages unavailable to other consumers, but network round trips per message are lower.
• Acknowledgements: a consumer acks after processing. An unacknowledged message is requeued if the channel closes. A consumer can nack or reject, with or without requeue.
• Publisher confirms: the broker confirms to the producer that it took responsibility for a message (after persisting it, for durable queues and persistent messages).
• Durability: durable queues and persistent messages survive a broker restart. Quorum queues replicate a queue across nodes using the Raft consensus protocol and are the recommended replicated queue type. Streams add an append-only, replayable log to RabbitMQ.
• Dead letter exchanges: rejected, expired, or overflowing messages can be rerouted to another exchange. See backpressure, retries, and dead letter queues.
• Other features: per-message and per-queue TTL, priority queues, and delayed delivery through plugins.
• Messages are removed once acknowledged, so a message cannot be re-read later (except with streams).

Ordering: messages in one queue are delivered in order to a single consumer. With several consumers, or after requeueing, processing order is not guaranteed.

Amazon SQS

Amazon Simple Queue Service (SQS) is a managed queue service. There are no servers to run, capacity scales automatically, and messages are stored redundantly across multiple availability zones. Consumers poll (pull) for messages over HTTP.

Core behavior:

• A received message becomes invisible for the visibility timeout (default 30 seconds). The consumer deletes it after processing. If it is not deleted, it reappears.
• Messages are retained for a configurable period (default 4 days, up to 14 days).
• Long polling waits for messages up to a configured time (at most 20 seconds), reducing empty responses and the number of requests.
• A dead-letter queue can be attached: after a message has been received a configured number of times without deletion (maxReceiveCount), it moves to the DLQ.
• Delay queues and per-message delay postpone delivery.
• SQS has no built-in fan-out. To deliver one message to several queues, publish to Amazon SNS and subscribe several SQS queues to the SNS topic.

Standard vs FIFO

In a FIFO queue, the message group id defines the ordering scope. Messages in one group are delivered in order, and while one message of a group is in flight, later messages of that group are not delivered to other consumers. The design mirrors keyed partitioning in Kafka.

Kafka vs RabbitMQ vs SQS

Choosing

• Event log, replay, several consumers, stream processing, very high volume: Kafka.
• Per-message routing, priorities, delayed messages, request/reply, moderate volume: RabbitMQ.
• Simple work queue on AWS with minimal operations: SQS (standard), or FIFO when order or deduplication is required.
• Consider the operational cost. A self-run Kafka or RabbitMQ cluster needs capacity planning, upgrades, and monitoring, whereas SQS is a service.
• Whatever the choice, consumers must handle duplicates and design for retries. See delivery semantics and idempotency.

Interview questions

• Why does a RabbitMQ producer publish to an exchange instead of a queue? The exchange separates routing from storage. The same message can be routed to several queues by key patterns, and queues can be added or rebound without changing the producer.
• How does an SQS FIFO queue keep order while allowing parallelism? Ordering applies within a message group id. Messages of different groups can be processed concurrently, while messages of the same group are delivered one at a time in order.
• Why can Kafka replay messages while RabbitMQ queues and SQS cannot? Kafka retains records in the log after they are read, and each consumer group tracks its own offset. RabbitMQ classic and quorum queues and SQS delete a message once it is acknowledged or deleted.
• How do you fan out one message to several SQS queues? Publish to an SNS topic and subscribe each queue to the topic.