No—not automatically. A broker can prevent certain duplicate sends or atomically coordinate message offsets with output records inside its own transaction boundary. That does not guarantee that a database update, payment request, email, or other external effect happens only once. If a handler performs the effect and crashes before its acknowledgment is committed, the message may be delivered again. The practical fix is to make the business operation safe to repeat.
What does “exactly once” actually promise?
The phrase can describe several different guarantees: a producer retry not creating a second broker message, a consumer’s progress being committed atomically with output records, or a business action occurring only once. Those are not interchangeable. To evaluate an exactly-once claim, identify what is counted, which systems participate, how duplicates are identified, and how long that identity is remembered.
As an Amazon Associate I earn from qualifying purchases.
- Delivery: whether a broker sends or exposes a message, possibly more than once.
- Processing: whether a consumer handles a message, possibly more than once after a retry or crash.
- Business effect: whether a durable change—such as charging an account—occurs more than once.
A broker can provide a real exactly-once guarantee within a defined scope. The mistake is extending that guarantee to a handler’s unrelated side effects without checking whether the destination participates in the same atomic transaction.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhy can a consumer process the same message twice?
A consumer has to make two durable facts agree: that the business work happened and that the broker should consider the input complete. A crash between those operations creates a choice.
#1 Best Overall
Commit progress before doing the work
If the consumer records its offset or acknowledgment first and then crashes before completing the business action, the broker may resume after the message. The action can be lost.
Do the work before committing progress
If the business action succeeds but the consumer crashes before its offset or acknowledgment is durable, the broker can deliver the message again. The action may repeat. This is the familiar at-least-once failure window described in Apache Kafka’s design documentation.
When the broker and destination do not share an atomic transaction, there is no single commit that proves both “the external effect happened” and “the broker recorded completion.” Kafka documents atomic coordination for Kafka offsets and Kafka output records; for other destination systems, it says exactly-once behavior generally requires cooperation from those systems. Two-phase commit can coordinate systems that support it, but an ordinary HTTP API or independently committed database transaction is not silently included in a broker transaction.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What Kafka’s exactly-once support covers
Kafka’s clearest exactly-once case is a consume-transform-produce flow that reads Kafka records and writes results back to Kafka. The consumer’s input offsets can be committed in the same Kafka transaction as the output records. Downstream consumers should use read_committed isolation so they do not see records from aborted transactions. Within this Kafka-to-Kafka boundary, the transaction can prevent a retry from making the input progress and output diverge.
Kafka’s design documentation describes a direct transactional pattern using a producer per consumer instance, with consumer auto-commit disabled. The documented settings include enable.auto.commit=false and isolation.level=read_committed; a producer’s transactional.id supports recovery and fencing of a prior instance. Check the documentation for the Kafka version you deploy before applying configuration details, because the design page follows the moving trunk documentation.
Kafka’s idempotent producer uses sequence numbers to suppress duplicate writes within a producer session. Transactions and a stable transactional ID extend recovery behavior across application sessions. These mechanisms still do not mean every consumer sees every committed transaction as one indivisible bundle in every circumstance: Kafka’s KIP-98 notes that compaction, log-segment deletion, seeking, or consuming only some participating partitions can affect what a consumer sees.
Rank #3
Where Kafka’s guarantee stops
If a Kafka handler calls a payment API or writes to a separate relational database, Kafka cannot atomically include that action merely by committing its offset transactionally. A crash after the external action but before the offset commit can lead to redelivery. The handler or destination needs its own duplicate-safe mechanism.
How SQS duplicate handling differs
Amazon SQS distinguishes standard queues from FIFO queues, and neither distinction removes the need to understand the scope of the guarantee.
| Queue or flow | Documented behavior | Important boundary |
|---|---|---|
| SQS standard queue | At-least-once delivery; AWS advises consumers to be idempotent. | A message may be received again, including when a server retains a copy after a receive/delete sequence. |
| SQS FIFO producer sends | Repeated sends with the same deduplication identity are suppressed within a five-minute deduplication interval. | This is a send-deduplication window, not a guarantee that an external handler effect cannot repeat later. |
| SNS FIFO delivered to SQS FIFO | AWS describes end-to-end conditions for this combination, including queue availability and permission, timely processing and deletion before visibility timeout expiry, no message filtering, and no network disruption that prevents acknowledgment. | The guarantee is conditional on the documented setup and failure assumptions. |
For SQS FIFO, a producer can use an explicit message deduplication ID or content-based deduplication. The latter hashes the message body with SHA-256 but excludes message attributes. If attributes distinguish two different business events while their bodies are the same, content-based deduplication can treat those sends as duplicates. AWS’s SQS FAQ also notes that a producer may resend after losing a response; FIFO removes duplicates within its five-minute interval, while standard queues can still deliver duplicates.
That five-minute interval is specific to SQS FIFO’s producer deduplication behavior. It is not a safe default for how long an application should retain a record that a business event has already been applied.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How do you prevent a retry from charging a customer twice?
Give each business event a stable identity and make the durable business update conditional on that identity. The identity should represent the event or operation, not a delivery attempt: retries must reuse it.
- Put a stable business event ID in the message. Preserve it across producer retries, redelivery, and any client retry that represents the same intended action.
- In one database transaction, claim the ID and apply the change. Insert the ID into an inbox or idempotency table with a uniqueness constraint, then perform the business update in that same transaction. If the ID already exists, treat the event as already applied rather than applying the change again.
- Commit the database transaction before acknowledging the broker. If the process crashes after the database commit but before acknowledgment, redelivery hits the unique key and becomes a no-op.
- Keep the response safe for callers too. If a client can retry the request, return a result consistent with the original operation rather than initiating a new one because the response was lost.
- Persist outgoing events with the business update. If the database change must produce another message, write an outbox row in the same transaction. A separate relay publishes it with a stable event ID, and downstream consumers apply their own idempotency check.
- Retain idempotency records for the real replay horizon. Keep them at least as long as messages can be replayed or redriven and clients can retry. A broker’s short duplicate-suppression window may not cover those application-level paths.
This inbox-and-unique-key approach makes the database transaction the boundary for the business change and its deduplication record. The outbox addresses the related problem of committing a database update while ensuring its outgoing event is eventually published. These are application design patterns, not automatic Kafka or SQS guarantees.
How to state an exactly-once guarantee accurately
A useful guarantee names its boundary rather than promising that a handler runs once under all conditions. Specify:
- System and flow: which broker, producer, consumer, and destination are involved.
- Atomic boundary: whether the transaction covers broker records only, broker offsets plus broker output, or the external business destination too.
- Duplicate identity and retention: whether identity comes from a producer sequence, explicit deduplication ID, body hash, or business key, and how long it is retained.
- Failure assumptions: what happens on process crashes, lost acknowledgments, consumer rebalances, visibility-timeout expiry, replay, redrive, or network disruption.
- Ordering scope: whether ordering applies globally, per partition or group, or within a product-specific message group.
For example, “Kafka atomically commits consumed offsets with output records written to Kafka, with downstream readers using read_committed” describes a bounded transaction. “The payment handler runs exactly once” is a much broader claim and is not established by that Kafka transaction alone.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
