A Kafka consumer can apply the same business event more than once after a restart or rebalance. The usual cause is a crash after the application has performed its work but before Kafka has recorded the consumer’s progress. Design for that replay with an idempotent effect or a transaction that includes both the result and the offset; Kafka offsets alone cannot make an arbitrary database write or API call exactly once.
Why does a Kafka consumer process an event again?
Kafka records consumer progress as offsets. A committed offset identifies the next record the consumer should read, not the last record it finished. Processing a record and committing progress are separate actions unless the application coordinates them with a transaction.
As an Amazon Associate I earn from qualifying purchases.
- The consumer polls a record at offset N.
- The application writes to a database or performs another business action.
- The consumer commits offset N+1, indicating that the next record to read is N+1.
If the process dies between the business action and the commit, Kafka still has the older committed position. A restarted or reassigned consumer resumes there and may apply the effect for N again. Kafka describes at-least-once delivery as the default: “Otherwise, Kafka guarantees at-least-once delivery by default, and allows the user to implement at-most-once delivery by disabling retries on the producer and committing offsets in the consumer prior to processing a batch of messages.” Apache Kafka 4.1 Design documentation.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThis is a replay possibility, not a promise that every event appears exactly twice. The number of repeats depends on when failures occur, how offsets are committed, and whether the destination itself retries or deduplicates the operation.
#1 Best Overall
What does the commit order change?
Commit after successful work: replay is possible, unfinished work is not skipped
Committing only after successful processing gives at-least-once behavior. If a database insert succeeds and the consumer crashes before its Kafka offset commit, the record can be read again and the insert attempted again. Kafka’s consumer API describes this recovery case: “In this case the process that took over consumption would consume from last committed offset and would repeat the insert of the last batch of data.” KafkaConsumer API documentation for Kafka 2.8.1.
Commit before work: less replay, but possible loss
If the application commits first and crashes before completing the effect, Kafka can resume after that record even though the business action never happened. That is at-most-once behavior for the committed record. It may be acceptable when losing an event is less harmful than repeating it, but it is usually the wrong choice when every business action must be completed.
For the common case where loss is unacceptable, process successfully before advancing the offset, then make a repeated effect harmless or coordinate the effect and progress in a shared transaction boundary.
Free tools Windows power users keep installed
One-click scans. No signup required.
Which handling strategy fits the destination?
Choose based on the consequences of loss and replay, whether the output system can join a transaction, and how much coordination the application can support. These approaches have different guarantee boundaries; none makes every kind of external side effect exactly once by itself.
| Strategy | Loss and replay behavior | Transaction boundary | Scope and operational considerations |
|---|---|---|---|
| Idempotent business operation | Commit after success; replay may repeat the request, but the final business state remains the same if the operation is truly idempotent. | No shared Kafka-and-sink transaction is required. For a database, a deduplication marker and business mutation can be committed together in the database. | Useful for database and HTTP effects with a stable event or business key. Requires careful handling of non-idempotent operations such as increments or payments. |
| Manual offset commit after success | Failure before commit replays completed work; committing only after success avoids advancing past unfinished work. | Kafka offset commit and external effect are separate unless the sink also participates in a transaction. | Clear at-least-once behavior. Pair with a sink-side idempotency guard or atomic sink transaction to handle the replay window. |
| Sink-side atomic storage of result and offset | The sink commits the business result and progress together, avoiding a gap between those two actions within its transaction boundary. | Both result and offset are stored in one destination transaction. | Appropriate when the destination can durably store both. Kafka’s consumer API documents this approach for an external store. |
| Kafka transactions or Kafka Streams | Kafka input offsets and Kafka output can be committed together, so downstream consumers configured to read committed data do not see aborted output. | Kafka transaction; Kafka Streams can also coordinate its Kafka output and state-store updates. | Fits Kafka-to-Kafka processing and Streams-managed state. It does not automatically include arbitrary database writes or API calls; transaction recovery and consumer configuration are required. |
| Commit before processing | A committed record will not be replayed from that committed position, but a crash before the effect completes can lose the work. | No transaction joining the effect and offset. | At-most-once choice for the affected work; use only when the business cost of loss is acceptable. |
How do you make an external effect safe to replay?
Give each logical event a stable identifier, such as an immutable event ID or a business key whose uniqueness is meaningful for the operation. Then make the destination reject or neutralize duplicate application of that event. The identifier must remain the same across retries and replays; generating a new key each time defeats deduplication.
- For state replacement: use an upsert keyed by the entity or event identity so replay writes the same state rather than creating another row.
- For one-time actions: keep a processed-event record with a uniqueness constraint, and perform the business mutation and marker insert in the same database transaction. If the marker already exists, treat the replay as already handled.
- For increments, payments, and similar effects: do not rely on a blind increment or repeated charge being harmless. Apply an event-key uniqueness guard or an idempotency mechanism provided by the destination.
These are application design patterns rather than Kafka-specific schema requirements. Kafka’s design guide uses overwriting a row by primary key as an example of an idempotent update. Apache Kafka 4.1 Design documentation.
Rank #3
A sink-side transaction can close the gap more strongly when it commits both the business result and the consumed offset. If the sink cannot store Kafka progress atomically with its result, a stable key and idempotent effect remain practical ways to make a replay safe.
What do Kafka’s exactly-once features cover?
Producer idempotence handles a different duplicate source
Producer idempotence prevents duplicates created by producer retries within a producer session; it does not deduplicate a consumer’s repeated database write, and it does not cover application-level resends as the same event. The KafkaProducer 3.9.2 API documents that, from Kafka 3.0, enable.idempotence defaults to true in that API, and describes the producer-session limitation. Do not assume the same defaults for another client language or an older client without checking its documentation. KafkaProducer API documentation for Kafka 3.9.2.
Kafka transactions cover Kafka records and offsets
For Kafka-to-Kafka processing, a transactional producer can write output records and send the consumed input offsets to the same transaction. The consumer must have automatic commits disabled, and downstream consumers should use isolation.level=read_committed so they do not read records from aborted or still-open transactions. A transactional producer configuration includes a stable transactional.id; follow the current Kafka design guide and client documentation for the full workflow and recovery behavior. Apache Kafka 4.1 Design documentation.
Kafka Streams integrates this kind of coordination for its Kafka output topics and state stores. Its 2.1 documentation explains the principle, but it is an older release reference; consult documentation for the deployed Streams version before using exact configuration names or assuming a support minimum. Apache Kafka Streams 2.1 Core Concepts.
Enabling Kafka transactions does not enlist an arbitrary external database or HTTP service. If processing must change such a destination, that system needs its own cooperating transaction mechanism or an idempotent operation. “Exactly once” is meaningful only when the boundary is specified: Kafka output and offsets, Kafka Streams state and output, or a sink’s own result and offset transaction are distinct guarantees.
Recommended Free Tools
How should offsets be committed when processing is parallel?
A committed offset is the next record to read. When manually committing a particular record, commit its offset plus one. In a partition, do not advance the committed position beyond work that has not finished: if records are processed concurrently and a later record finishes first, committing its offset can cause an unfinished earlier record to be skipped after a restart.
If work is moved off the poll thread, keep polling within max.poll.interval.ms and coordinate completed offsets per partition so progress reflects the contiguous work actually finished. KafkaConsumer 2.8.1 documents poll liveness, manual commits, and offset handling; use the API documentation matching the deployed client for exact details. KafkaConsumer API documentation for Kafka 2.8.1.
Can auto-commit cause duplicates or lost work?
Auto-commit is not inherently at-most-once. KafkaConsumer 2.8.1 says automatic commits can provide at-least-once behavior when the application finishes processing all records returned by poll before calling poll again or closing the consumer. If it commits progress for a batch before the application has completed that batch, an interruption can leave Kafka’s committed position ahead of completed work and records can be lost. For work whose completion timing is not tightly controlled, explicit commit-after-success logic makes the intended progress boundary clearer. Check the consumer API version used by your application for current behavior and configuration details. KafkaConsumer API documentation for Kafka 2.8.1.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →
