To preserve session order in Kafka, send every record for a session to the same partition and do not commit past a failed record until it has been handled. This can pause later records in that partition. A separate retry topic may let the source partition continue, but it can also let later records overtake the delayed one unless retries are coordinated by key.
Start with the ordering guarantee you need
Kafka guarantees record order within a partition, not across an entire topic. If a session’s events must be processed in sequence, route them to the same partition—commonly by using a stable session or entity key as the record key. This makes partition order the basis for session order, provided the key remains stable and processing does not allow records for that key to overtake one another. See Apache Kafka’s Kafka 4.0 design documentation.
As an Amazon Associate I earn from qualifying purchases.
Be precise about the scope: Kafka’s guarantee is per partition. A requirement for one global order across multiple partitions is not provided by the partition-ordering guarantee.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsChoose how a failed record should hold up progress
| Approach | What happens to later records in the partition | Ordering trade-off |
|---|---|---|
| Retry in place or replay from the failed offset | They wait while the failed record remains unresolved. | Preserves the straightforward partition sequence, but blocks progress for that partition. |
| Send the failed record to a retry topic and continue the source partition | They may be processed before the retried record. | Improves scheduling flexibility, but a retry topic alone does not preserve original per-key order; coordinate by key if strict order is required. |
The retry-topic trade-off follows from Kafka’s partition and offset model: the source partition can advance while the failed record is waiting on a separate path. It is not an ordering guarantee provided by retry topics.
Retry in place when later records must wait
Kafka consumer records are returned in offset order, and committed offsets determine where a group resumes after a restart. To keep a failed record from being overtaken, do not commit an offset beyond it while it remains unprocessed. A consumer can rewind its position to replay records; the replay may also repeat work already performed after the last committed position. Consult Kafka’s Kafka 4.0 consumer configuration and distribution documentation.
- Identify the failed record’s partition and offset.
- Keep the committed position from advancing beyond that failure while strict order is required.
- Retry processing or rewind and replay from the failure according to your consumer’s offset-handling design.
- Advance the committed position only after the failed record and the preceding work are safely handled.
This preserves the sequence at the cost of holding up later records in that partition. Other partitions can continue independently.
Use retry topics only with an ordering plan
A retry topic separates failed work from the source partition’s normal consumption path. That can be useful when a failure needs a delay or should not stall unrelated work, but continuing to process the source partition means later records may finish before the retried record.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →If strict order is required for each session, coordinate retry and normal processing by key. The design must prevent a later record for a key from being applied while an earlier record for that key remains pending. A retry topic by itself does not provide that coordination.
Protect the producer-side order during retries
Consumer retry handling is only one source of reordering. With producer idempotence disabled, retries combined with multiple in-flight requests can allow a later batch to overtake an earlier batch that failed. Kafka 4.0 documents idempotence as enabled by default when configuration does not conflict with it; it requires acks=all, retries greater than zero, and max.in.flight.requests.per.connection no greater than 5. See the official Kafka 4.0 producer configuration.
When idempotence is disabled, setting max.in.flight.requests.per.connection=1 removes the concurrent-request reordering risk described in the producer documentation, though it can reduce throughput. Idempotence protects the producer-to-broker retry path; it does not ensure consumer business logic runs exactly once or coordinate a retry-topic workflow.
Use Kafka transactions for Kafka-to-Kafka processing
For a consume-transform-produce workflow that reads and writes Kafka, transactions can atomically commit the output records and the consumed offsets. This avoids committing the input progress separately from the Kafka output. Consumers that must not see aborted transactional output should use isolation.level=read_committed. Kafka documents that such consumers return committed transactional records and may wait for an earlier open transaction to resolve before returning later records, stopping at the last stable offset. See the Kafka 4.0 design documentation and consumer configuration.
Free tools Windows power users keep installed
One-click scans. No signup required.
These transaction guarantees cover Kafka records and offsets. They do not automatically include writes to an external database or API; those side effects need their own consistency and duplicate-handling strategy.
Quick Recap
Make the choice against your failure requirements
- Ordering scope: Decide whether order is required per partition, per session key, or more broadly.
- Blocking tolerance: In-place retry keeps the sequence clear but delays later records in the affected partition.
- Retry policy: Set the delay and attempt behavior to fit the failure, and decide when a record is considered resolved.
- Duplicates and side effects: Replays can repeat processing, so make business operations safe to repeat or otherwise manage duplicate effects.
- Output destination: Kafka transactions can cover Kafka output and consumed offsets, but not an external system’s write.
- Throughput and parallelism: Partition-level blocking and producer in-flight settings affect how much work can proceed concurrently.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
