Kafka preserves record order within each partition, not across a topic or consumer group as a whole. To keep a session’s messages in sequence through a rebalance, route every message for that session to the same partition, process that partition in order, and prevent revoked or unfinished work from being committed past.
What a rebalance changes—and what it does not
A rebalance changes which consumer owns a partition. It does not reorder the records in that partition’s log. Apache Kafka’s Kafka 4.1 consumer configuration states, “Messages will always be returned in offset order.” Apache Kafka consumer configuration (4.1)
As an Amazon Associate I earn from qualifying purchases.
That guarantee is narrower than many applications need. Kafka returns records in offset order, but it does not make asynchronous application work finish in that order. Nor does it establish a single order across different partitions. If records for one session land in multiple partitions, Kafka’s per-partition ordering alone cannot preserve a total session sequence.
Route each session to one partition
Use a stable key that identifies the session or entity when producing its records. With consistent partitioning, records for a given key go to the same partition, which gives them a single ordered log. Confirm that producers use compatible partitioning behavior and that partition-count changes or custom partitioners do not undermine the routing assumption.
#1 Best Overall
This establishes Kafka’s ordering boundary; the consumer application must still keep processing and side effects in sequence. If a session’s events must be ordered across several partitions, the application needs an explicit sequencing or coordination design beyond Kafka’s per-partition guarantee.
Keep processing ordered across ownership changes
Use one ordered lane per partition
Process each partition sequentially, or ensure any parallel work preserves both dispatch and completion order before applying effects. If you use worker queues, keep records partition-scoped and do not allow a later record’s side effect to overtake an earlier one. Pause or buffer a partition when its work queue is full rather than continuing to dispatch work that cannot safely complete in order.
Handle revocation before letting work escape the old owner
When a partition is revoked, stop dispatching new records from it and either finish outstanding work or leave unfinished records uncommitted. A new owner may then read and process those records again. Callback details vary by client library and framework, so implement revocation handling according to the documentation for the specific client and version in use; there is no single callback recipe that applies to every Kafka consumer.
Free tools Windows power users keep installed
One-click scans. No signup required.
Commit only the completed prefix
For each partition, advance the committed offset only through the highest consecutively completed record. If record 12 is still being processed, completing record 13 does not make it safe to commit past 12. After a crash or reassignment, an uncommitted record can be replayed. Make downstream effects idempotent or otherwise safe to repeat when the application requires at-least-once processing; ordering alone does not make external side effects exactly once.
Rank #3
Choose a rebalance protocol that matches the deployment
Check the broker and client versions, the group protocol, the assignor, membership settings, and rebalance logs before changing configuration. Kafka 4.x can use either the classic protocol or the newer consumer protocol, and their settings are not interchangeable.
| Option | What changes | Trade-off |
|---|---|---|
| Classic eager assignment | A rebalance can revoke all current partitions before assigning them again. | Simple and broadly compatible, but a larger reshuffle can interrupt more processing. |
| Classic cooperative sticky assignment | Retains eligible assignments and transfers partitions incrementally. | Can reduce unnecessary movement, but all group members must support compatible cooperative assignment and upgrades need care. |
| Kafka consumer protocol | Uses incremental rebalancing with server-controlled assignor and heartbeat/session configuration. | Requires supported Kafka versions and a deliberate migration; classic client settings do not apply in the same way. |
Classic groups: consider cooperative sticky assignment
For a group using the classic protocol, evaluate CooperativeStickyAssignor if reducing unnecessary partition revocation is useful. Apache Kafka’s Kafka 4.3.1 API reference says, “Users should prefer this assignor for newer clusters.” Apache Kafka CooperativeStickyAssignor API (4.3.1)
Use a consistent assignor across group members. Kafka’s API documentation requires every consumer to use that assignor, or a cooperative custom assignor, for cooperative rebalancing. Follow the version-specific migration instructions, especially when upgrading from Kafka 2.3 or earlier. Cooperative assignment reduces some disruptions; it does not eliminate rebalances or make in-flight work safe automatically.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #4
- Metamorphosis: Franz Kafka (Little Clothbound Classics)
Kafka 4.x: distinguish the consumer protocol from classic settings
The newer consumer rebalance protocol became generally available in Kafka 4.0. Kafka 4.3 documentation enables it by setting group.protocol=consumer; the broker controls heartbeat/session settings and assignors in that mode. The same documentation says classic settings such as session.timeout.ms, heartbeat.interval.ms, and partition.assignment.strategy are not usable with this protocol. Apache Kafka consumer rebalance protocol (4.3)
The protocol’s incremental design removes a global synchronization barrier, which Kafka documents as a rebalance-time benefit. That design statement is not a measured performance guarantee for a particular workload. The protocol is not enabled by default in the cited Kafka 4.3 documentation; confirm client and broker support and follow the upgrade path before switching.
Best Value
Keep polling within the configured interval
The consumer must continue calling poll() often enough to remain an active group member. In Apache Kafka’s Kafka 4.1 consumer configuration, max.poll.interval.ms has a documented default of 300000 ms (five minutes). If the interval elapses without a poll, the consumer can be considered failed and its partitions reassigned. This is a version-specific default: check the configuration for the client version actually deployed.
max.poll.records limits how many records a poll returns, while max.poll.interval.ms limits the delay between polls. If processing duration varies, cap records handled per cycle or decouple polling from processing with bounded, per-partition queues. In either design, keep work ordered and ensure backpressure does not lead to concurrent out-of-order effects. Apache Kafka consumer configuration (4.1)
Use static membership only when identities are stable
Static membership through group.instance.id can avoid some rebalances caused by transient unavailability when a consumer has a stable, unique instance identity. Kafka’s Kafka 4.1 documentation also describes a failure-detection trade-off: when a static member exceeds max.poll.interval.ms, it is not immediately reassigned in the same way as a dynamic member; reassignment waits for session timeout. Use static membership only when instance identities are reliably unique and the slower failure detection is acceptable. Apache Kafka consumer configuration (4.1)
Quick Recap
Operational checks before changing settings
- Verify broker and client versions, group protocol, assignor, and whether membership is static or dynamic.
- Inspect rebalance logs and determine whether the trigger is deployment churn, consumer failure, slow processing, or polling delays.
- Confirm every session’s records use the same partitioning key and that processing cannot complete effects out of order.
- Review revocation handling and offset commits: unfinished records must not be skipped by a commit.
- Test crashes and rebalances with the actual client library and downstream side effects; expect replay where work was not committed.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
