Recommended Free Tools
A message broker is not a pipe with storage attached. It accepts a message under one rule about what counts as accepted, keeps it under another rule about where and how many copies survive, and releases it to readers under a third rule about when a message counts as handled. The storage primitive and those acknowledgment boundaries decide what the broker can promise about ordering, replay, durability, and recovery after a failure.
Apache Kafka, RabbitMQ, and NATS JetStream make different choices at each of those points. They are used below as examples of design trade-offs, not as a ranking of which product is better.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Distributed Systems | $32.68 | Buy on Amazon |
| 2 |
|
Understanding Distributed Systems, Second Edition: What every developer should know about large... | $32.41 | Buy on Amazon |
| 3 |
|
Distributed Systems | $35.00 | Buy on Amazon |
| 4 |
|
Foundations of Scalable Systems: Designing Distributed Architectures | $42.49 | Buy on Amazon |
| 5 |
|
Distributed Systems: Concepts and Design | $255.63 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
How a distributed message broker works
Whatever the product, a broker follows the same four-stage path. Each stage is a place where a message can be lost, delayed, or delivered twice.
- Accept. The publisher sends a message and waits for a confirmation. What that confirmation means is the first design decision. In some systems it means one node wrote the record; in others it means a majority of replicas agreed.
- Store. The broker writes the message into a log, queue, or stream and may copy it to other nodes. The storage unit determines ordering, how long messages are kept, and how a consumer later finds its place.
- Deliver and acknowledge. A consumer receives the message and reports back. The broker uses that report to decide whether the message is finished or must be sent again.
- Recover. When a node, a leader, or a network link fails, a replacement takes over. The questions after any failure are which accepted messages survived in a form the new leader can serve, and which acknowledgments were never seen.
Two consequences follow. A broker can behave correctly at every stage and still hand a consumer the same message twice, because the broker and the consumer can disagree about what happened last. And the strength of a guarantee depends on the replication and acknowledgment settings in use, not on the product category.
#1 Best Overall
How message brokers store messages
The storage primitive is the most important design choice because it fixes what a reader can see, in what order, and for how long.
Apache Kafka: replicated partition logs
Kafka routes records into topic partitions. Each partition has a leader and zero or more followers. Followers pull records from the leader and append the same ordered records at matching offsets. Ordering is guaranteed within a partition, not across a whole topic. Partitions buy parallelism, so ordering and throughput are traded against each other through how records are assigned to partitions.
Kafka defines a committed record in terms of the in-sync replica set (ISR). Consumers only see committed messages, and producers choose how much acknowledgment they wait for. The Design section of the Apache Kafka 3.4 documentation states that a committed message is protected while at least one in-sync replica remains alive. It does not guarantee availability during network partitions. Records stay in the log according to the configured retention rather than being deleted when read, so a consumer can move back to an earlier offset and reprocess within the retention window.
RabbitMQ: exchanges, queues, and queue types
RabbitMQ separates routing from storage. Exchanges and bindings decide which queues receive a message; the queues hold it. The queue type determines persistence and reading behavior.
- Quorum queues are durable, replicated structures based on the Raft consensus algorithm. One member acts as leader, handles state-changing operations, and replicates each change to followers. A majority of members must agree on queue state. RabbitMQ’s 4.3 documentation defines the quorum as (N/2)+1 members, so a three-member group needs two members to agree and can lose one. A publisher confirm for a quorum queue means the message reached a quorum. Manual consumer acknowledgments let unprocessed messages return for another attempt. This safety costs latency and workload capacity.
- Classic queues have different persistence semantics from quorum queues. This article covers quorum queues in detail and does not analyse classic-queue durability.
- Streams have different persistence and reading semantics. Readers read from a position in the stream rather than taking messages off a queue.
RabbitMQ’s guidance points toward another queue type or a stream for temporary queues, low-latency workloads, very large backlogs, or large fanouts. Its 4.3 quorum-queue documentation also suggests reviewing whether some queues can become classic queues or streams once a deployment needs more than approximately 5,000 quorum queues. That is operational guidance, not a hard product limit.
NATS JetStream: streams and consumers
Core NATS delivers messages to subscribers connected at the time and does not replay them, so its delivery is at-most-once. JetStream adds persistence. A stream stores messages whose subjects match configured patterns and assigns each one a sequence number. Consumers are server-side views of a stream that track their own progress independently of other consumers. Streams can keep messages in memory or on disk, and retention and replication are configurable.
Storage models side by side
| Design question | Apache Kafka | RabbitMQ quorum queue | RabbitMQ stream | NATS JetStream stream |
|---|---|---|---|---|
| Storage unit | Partition log within a topic | Replicated queue (Raft) | Replayable stream | Messages matching subject patterns |
| Replication | Leader and followers; a record is committed when the in-sync replica set has it | A majority of members must agree on queue state | Not stated in the cited RabbitMQ guidance | Configurable per stream |
| What a write confirmation means | Governed by the producer’s acknowledgment setting and the in-sync replicas | The message reached a quorum | Not stated in the cited RabbitMQ guidance | Not stated in the cited NATS documentation |
| Who tracks read position | The consumer, through committed offsets | The broker, through manual acknowledgments | The reader, by position | A server-side consumer |
| What triggers redelivery | A consumer resumes from its last committed offset after a failure | An unacknowledged message returns for another attempt | Readers resume at their own position | No acknowledgment arrives before the acknowledgment timeout |
| Ordering boundary | Partition | Not stated in the cited RabbitMQ guidance | Position within the stream | Stream sequence numbers |
| Replay of acknowledged messages | Yes, within the retention window | No; acknowledged messages leave the queue | Yes, within the retention window | Yes, within the stream’s retention settings |
Kafka vs RabbitMQ for reliable messaging
The useful question is not which broker is more reliable. Each can be configured for durable, replicated writes. The question is what the broker must remember and who decides what happens next. The documentation cited here is product documentation, not a neutral performance study, and it does not establish that one broker is faster. Throughput and latency depend on the workload and have to be measured on your own.
Choose by workload shape:
- Work distribution. If each job should be handled once by one of many workers and then removed, a queue model fits. RabbitMQ’s queues, with manual acknowledgments and dead-lettering, are built around that flow.
- Replayable history. If several independent consumers need the same events, or a new consumer must read history, a partitioned log or stream fits. Kafka’s partition logs and RabbitMQ streams both serve this need with different trade-offs.
- Ordering. Kafka orders records per partition, so the key used to assign records to partitions is a correctness decision. Check RabbitMQ’s ordering behavior for the queue type you plan to use rather than assuming it.
- Routing. If messages must be routed by binding rules to several queues, RabbitMQ’s exchange model is designed for that. Kafka topics and JetStream subject patterns select records by name.
- Consumer progress. Kafka consumers manage their own offsets, JetStream consumers are server-side views, and RabbitMQ queues track deliveries against acknowledgments.
NATS JetStream sits between these models. It gives a stream replay and independent consumer positions, but it is built on a messaging core that does not persist messages by itself.
Before choosing, answer these questions in writing:
- Must each message be processed by exactly one worker, or by every interested reader?
- Does a new consumer need history, and for how long must it be kept?
- Which key must stay in order, and how much parallelism do you need?
- How long can producers and consumers pause during leader election?
- Will the cluster span data centers, and what is the round-trip time between them?
- Who owns the retry policy and the dead-letter destination?
What does at-least-once delivery mean?
At-least-once delivery means the broker keeps a message until it receives an acknowledgment and sends it again when none arrives. Nothing is silently dropped along that path, but the same message can reach a consumer more than once. Duplicates are the price of not losing work.
The acknowledgment point decides which guarantee you get. The Design section of the Apache Kafka 3.4 documentation puts it this way:
“Otherwise, Kafka guarantees at-least-once delivery by default, and allows the user to implement at-most-once delivery by disabling retries on the producer and committing offsets in the consumer prior to processing a batch of messages.”
Rank #3
The rule behind that statement: the later a consumer acknowledges relative to processing, the more duplicates it can see and the less work it can lose. Acknowledging before processing shifts the trade-off to at-most-once, where a crash after the acknowledgment loses the work.
The same principle appears in each layer:
- Publisher side. A publisher that does not receive a confirmation has to assume the message may not have arrived and send it again.
- Consumer side. If the consumer crashes after processing but before acknowledging, the broker redelivers the message.
- NATS JetStream. If an acknowledgment does not arrive in time, the consumer receives the message again. This is the at-least-once path. Core NATS, without a stream, does not replay.
A consumer should not treat the arrival of a message, or the broker’s own receipt, as proof that the work succeeded. Acknowledge only after the handler’s side effects are committed.
Can a message broker guarantee exactly-once delivery?
Not end to end on its own. An exactly-once claim holds only inside the boundary of the components that take part in it, and every boundary outside that needs its own mechanism.
Inside the broker’s own transactions
Kafka documents exactly-once semantics for Kafka Streams and for its transactions. Reads, state changes, and writes that stay inside Kafka are committed together or not at all. That guarantee is real, but it covers work whose inputs and outputs are Kafka records. The Apache Kafka 3.4 documentation describes this model; confirm the behavior for the version you run.
Between the broker and your database
If a handler writes to a database, the broker’s transaction cannot cover that write. A crash between the database commit and the broker acknowledgment means the message comes back. The fix is either to commit the database change and a record of the processed message together, or to make the handler idempotent so that a second run has no further effect.
External side effects
Emails, payments, and calls to third-party APIs sit outside every broker boundary. Only the receiving system can prevent a repeat, usually through an idempotency key it honors. If it offers none, deduplicate on your side before making the call. A crash during the call can still leave the outcome unknown, and that case needs a manual check against the receiving system.
Building an idempotent consumer
- Choose an identifier for each message. A broker-assigned identifier (Kafka’s topic, partition, and offset, or JetStream’s stream sequence number) catches redelivery of the same record. It does not catch a publisher’s retransmission, which the broker stores as a new record. For that case, use an ID the publisher assigns and includes in the message.
- In one database transaction, make the local state change and insert the identifier into a processed-messages table with a unique constraint.
- If the insert violates the constraint, the message was already handled. Skip the state change and acknowledge.
- Acknowledge the broker only after that transaction commits.
What happens when a message broker goes down?
The answer depends on what failed and where the message was at the time. Replicated brokers are designed to keep accepted data safe and to restore service. These are separate properties, and service can be interrupted even when the data is safe.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallA node or leader fails
A replicated system elects a replacement leader from surviving replicas. In RabbitMQ quorum queues, in-flight deliveries pause during the election. Consumers attached to the failed node recover, and consumers connected to other nodes are re-registered after the election. RabbitMQ’s clustering guide says a cleanly detected node crash normally leads to an election within about a second. Silent network failures depend on the failure detector and its settings, so there is no fixed timing for them. That figure is RabbitMQ-specific guidance, not a general failover guarantee.
Kafka also elects a new leader for a partition, and clients may need to reconnect. The pause your system experiences depends on producer retry settings and how much consumer lag you can tolerate. Measure it in your own environment rather than taking it from a product page.
A network partition or lost quorum
Majority-based replication favors one consistent history over availability on both sides of a split. The side with a majority keeps accepting quorum-dependent writes, and the minority side cannot. The Kafka design documentation likewise does not guarantee availability during network partitions.
Placement matters as much as software. RabbitMQ’s clustering guide says a two-data-center layout cannot protect against losing the majority site. It describes three data centers as the practical minimum for tolerating the loss of any one site, with the placements the guide lists. Every replicated operation, including publisher confirms, pays the cross-site round trip. The same guide describes a 10–100 ms p99 round-trip time as viable across data centers or regions, with latency costs to plan for. It does not recommend clustering above 100 ms or where packet loss is visible. Where inter-site links are unstable, it recommends connecting independent clusters asynchronously with Shovel or Federation rather than stretching one cluster across the link.
Free tools Windows power users keep installed
One-click scans. No signup required.
Uncertain acknowledgments and duplicates
A publisher that loses its connection before receiving a confirm cannot know whether the broker accepted the message. RabbitMQ’s reliability guide advises retransmitting unconfirmed messages, which can create duplicates when the broker’s confirmation was lost in transit. A consumer can also see a message again after it has already processed it, if the acknowledgment never reached the broker. Idempotent handling, described above, is the defense.
Best Value
“Data safety is a joint responsibility of RabbitMQ nodes, publishers and consumers.”
RabbitMQ reliability documentation
Consumer failure and retry loops
Redelivery protects work from being silently discarded, but it can also loop indefinitely. A message that always fails will keep returning unless the handler has bounded retries, a poison-message policy, and a dead-letter or quarantine destination. RabbitMQ’s 4.3 quorum-queue documentation describes poison-message handling, delayed retry, and at-least-once dead lettering as features. Confirm how they behave in the RabbitMQ version you run before relying on them.
Storage and correlated failure
Replication protects against independent failures, not correlated ones. Losing every replica, a shared storage or power failure, an operator error, or a retention setting that deletes data before consumers read it can lose messages regardless of the protocol. Kafka’s guarantee holds only while an in-sync replica survives, and RabbitMQ quorum availability depends on a majority. Neither system supports a claim that messages can never be lost.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Recovering after a broker incident
Recovery is a sequence of checks rather than a restart:
- Establish what survived: which nodes, replicas, or sites are up, and whether a quorum (RabbitMQ) or an in-sync replica set (Kafka) still exists for each affected queue or partition.
- Confirm that leadership has settled and that consumers have reconnected to the new leader.
- Measure redelivery and unacknowledged counts. A spike means consumers are reprocessing messages, which is safe only if their handlers are idempotent.
- Inspect dead-letter or quarantine destinations for messages that failed repeatedly, and decide how to replay them.
- Reconcile external side effects for any handler that crashed between an external call and its acknowledgment, using a check against the receiving system.
Verify these settings before relying on any guarantee:
Quick Recap
- Producer acknowledgment and retry settings in the client library version you run.
- Replication and in-sync replica requirements (Kafka) or quorum size (RabbitMQ) for each topic or queue.
- Whether each consumer acknowledges only after its side effects commit.
- Retention windows measured against the longest consumer lag you will tolerate.
- Failure-detection settings for silent network failures in RabbitMQ clusters.
- Version-specific defaults. The Kafka behavior described here comes from the 3.4 documentation, and the RabbitMQ quorum-queue figures come from the 4.3 documentation.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
