For most Kafka consumers, the practical goal is not to make a handler run exactly once. It is to make a repeated execution produce no extra business effect. Process a record, persist the effect, then commit its offset; if a crash causes redelivery, an idempotent destination operation can safely absorb it.
Why Kafka consumers can process a record again
A consumer controls its position in Kafka’s log. If it completes work but fails before saving its progress, a replacement or restarted consumer can read the record again. The ordering of the work and offset save therefore determines the failure mode. Apache Kafka’s design documentation cautions that exactly-once claims need to be examined in light of failures and the system boundaries involved.
As an Amazon Associate I earn from qualifying purchases.
| Ordering | Failure consequence | Delivery behavior |
|---|---|---|
| Save the offset, then perform the work | A crash after saving progress but before completing the work can skip that record from the application’s perspective. | At-most-once: work may be lost, but a saved position prevents that record from being processed again. |
| Perform the work, then save the offset | A crash after the effect but before saving progress can cause the record to be processed again. | At-least-once: the described ordering avoids skipping work, but allows redelivery. |
For many applications, the second ordering is safer when the destination effect is idempotent. Offset management does not itself make an external operation repeat-safe.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →What idempotent means at the destination
An operation is idempotent when applying it more than once produces the same resulting state as applying it once. For example, setting order 482 to “shipped” by its stable business key can produce the same stored state on every retry. Kafka’s design documentation uses a keyed update that overwrites the same record to illustrate this pattern.
#1 Best Overall
When an upsert is enough
An upsert or state-setting update is a good fit when each event describes the desired state, repeated writes are acceptable, and the key identifies the same business entity. A stable key alone is not sufficient if the operation remains cumulative: “increment the balance by $10” increments again on each replay unless the destination also deduplicates the event or the operation is redesigned as a repeat-safe state update.
When the destination should record event identity
For a non-idempotent business mutation, persist a stable event identity alongside the mutation in one destination transaction. Enforce uniqueness on that identity so a replay cannot execute the business action a second time. This is an application and destination design pattern; Kafka does not create the external database constraint for you.
When the destination is an API
Send a stable idempotency key only if the API documents support for that mechanism. Without it, a timeout can leave the caller unsure whether the remote action succeeded. Kafka offsets cannot resolve that ambiguity: the application may need reconciliation or an outbox/inbox design.
Recommended Free Tools
Choose the guarantee at the boundary that matters
| Approach | Best fit | Failure behavior | Main constraint |
|---|---|---|---|
| At-least-once plus an idempotent destination operation | Most consumers whose destination can upsert, deduplicate, or transact a business mutation with an event key. | A call may be repeated after redelivery, while the business effect remains stable if designed correctly. | Idempotency must exist in application logic or at the destination. |
| Kafka transactions | Kafka-to-Kafka processing where output records and consumed offsets must advance atomically. | Aborted work and offsets can be retried together; transactional output should be read with read_committed. |
Requires the transaction protocol, correct offset handling, and Kafka output. |
| Destination transaction/checkpoint cooperation | A system that needs a stronger atomic relationship between output and consumed position at an external destination. | The destination controls durable output and progress together. | The destination must cooperate; Kafka alone cannot provide this atomicity. |
Kafka producer idempotence is not consumer idempotence
Producer idempotence addresses duplicate records that can arise when a producer retries. The Kafka 3.9.2 Java KafkaProducer API documentation limits that guarantee to one producer session and says application-level resends are not deduplicated. It does not prevent a consumer from repeating a database write, sending an email twice, or repeating another external side effect.
Rank #3
That API documentation says enable.idempotence defaults to true starting with Kafka 3.0. These are Java client and version-specific details; check the documentation for the client version actually deployed rather than assuming identical defaults elsewhere.
Use Kafka transactions for Kafka-to-Kafka atomicity
When a consumer transforms Kafka records into other Kafka records, a transaction can couple output records with the consumed offsets. The application must follow the transaction protocol, disable automatic offset commit, include the consumed offsets in the transaction, and have downstream consumers use read_committed so they do not expose aborted transactional output. On an abort, restore or re-fetch from the committed position as described in Kafka’s design documentation and producer API documentation.
For the Kafka 3.9 producer, setting transactional.id enables transaction semantics across producer sessions and implies idempotence; without it, the documented guarantee is limited to idempotent delivery. Kafka’s 3.9 producer configuration also notes that the default transaction-state-topic setup expects at least three brokers for production. Check the actual broker topology and durability requirements before changing replication settings.
Why a Kafka transaction cannot include an unrelated database or API
A Kafka transaction can atomically cover Kafka records and offsets when the protocol is followed. It does not automatically enlist an unrelated database or remote API in the same transaction. If the consumer commits the external effect and then crashes before the Kafka offset is committed, the event may be replayed; if it commits the offset first and then fails, the external effect can be skipped.
To get a stronger guarantee at an external destination, that system must cooperate—for example, by committing its business mutation and checkpoint together, or by supporting a durable idempotency key. Otherwise, use at-least-once processing with a repeat-safe effect and a recovery plan for ambiguous outcomes.
Where Kafka Streams fits
Kafka Streams provides integrated processing guarantees across input offsets, output topics, and state stores, as described in its 4.1 core concepts documentation. That is a bounded Kafka Streams use case, not evidence that arbitrary side effects to external systems happen exactly once.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
