To preserve event order for each session in Kafka, publish every event with the same stable session key, route that key consistently to one partition, and process that partition sequentially wherever downstream effects must retain the order. Kafka orders records within a partition—not across a topic’s separate partitions—so keyed partitioning preserves per-session order while allowing different sessions to run in parallel.
What Kafka ordering guarantees—and what it does not
Kafka describes topic partitions as ordered commit logs. A record’s position in its partition determines its order relative to other records in that same partition. A topic with several partitions has several such ordered logs, not one shared sequence. As the Apache Kafka protocol documentation puts it, “Topic partitions themselves are just ordered ‘commit logs’ numbered 0, 1, …, P-1.”
As an Amazon Associate I earn from qualifying purchases.
That boundary shapes the design: records for one session must land in the same partition, and application processing must not let later records’ side effects overtake earlier ones. Keying alone does not establish a total order across sessions or partitions.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose a stable session key and partitioning plan
Define which events belong together
Decide what “session” means in the application, then use its stable identifier as the Kafka record key. Every producer that writes events for that session must use the same key and a consistent partitioning scheme. Kafka’s producer controls partition assignment; semantic partitioning uses a key to route related records together, where they can be processed with local state while retaining partition order.
#1 Best Overall
Balance ordering scope with parallelism
With a session key, different sessions can be spread across partitions and processed independently, while a given session follows one partition’s ordered lane. A single partition can provide one serialization boundary for all of its records, but it also limits parallelism for those records. If the actual requirement is a total order across every event in the topic, keyed partitioning is insufficient; the design needs a single shared ordering boundary instead.
Plan changes to partitioning
Do not assume that changing a topic’s partition count or changing partitioning behavior preserves one uninterrupted sequence for a session across the change. Kafka’s ordering guarantee is per partition; it does not guarantee that a key’s earlier and later records remain in one ordered stream after reassignment. Plan such changes as migrations, including how producers and consumers handle the transition.
Rank #2
Produce keyed events with the Go client
Confluent’s confluent-kafka-go wraps librdkafka. Its producer API uses Produce; producing is asynchronous, so a successful call to enqueue a record is not the same as confirmation that Kafka accepted it. Track delivery reports and handle each message’s success or error. The repository also documents a cloud example, but that does not establish current hosted-service terms or pricing.
Pin the client version in your project and verify configuration names and API details against that version: repository documentation on the mutable master branch may not match the release you deploy. During shutdown, allow in-flight records to reach a delivery result or fail before closing the producer. The repository documents using Flush with a timeout for this purpose; choose and document what the application does if that timeout expires.
Rank #3
Consume each partition sequentially when effects must stay ordered
The Go client’s function-based consumer workflow uses subscriptions and Poll with a high-level consumer group. A group distributes assigned partitions among its members; this allows different partitions to be handled by different consumers, but does not create a total order across them.
If a database update, state transition, or other application-side effect must follow Kafka’s order, ensure work from each assigned partition is completed sequentially. Parallel work within a partition can otherwise finish out of order even when records were read in order. On shutdown, finish processing or safely abandon in-progress work before committing offsets; do not commit progress beyond work the application has actually completed. The exact shutdown and recovery policy depends on the application.
Choose delivery semantics for the processing path
| Approach | What it addresses | What it does not establish | Operational trade-off |
|---|---|---|---|
| Keyed production and sequential partition processing | Per-session order within the session’s partition. | A total order across partitions, or atomicity with an external database or service. | Simpler than transactions, but the application must handle delivery reports, offset commits, retries, and duplicate effects appropriately. |
| Idempotent production | Duplicate log entries caused by producer retries within Kafka’s producer semantics. | Atomicity between consumed input offsets and output records, or arbitrary external side effects. | Requires producer configuration and handling appropriate to the client and broker versions in use. |
| Kafka transactions | Atomic Kafka output records together with consumed offsets in a consume-transform-produce flow. | Atomicity with external systems such as a database. | Adds transaction lifecycle, error handling, and consumer isolation requirements. |
These mechanisms address different concerns. Select the session key and partitioning plan for ordering first; consider idempotence or transactions for retry and processing guarantees separately.
Use Kafka transactions for Kafka input-to-output workflows
Kafka transactions can coordinate the output records from processing with the input offsets being committed. In the Confluent Go workflow, the producer uses a transactional.id, initializes transactions, begins a transaction, produces output, sends the next input offsets with consumer group metadata, and commits. The consumer must have automatic offset commits disabled for this flow. If processing fails, abort and retry according to the error type and application policy. Consumers that must not see aborted transactional records need transaction-aware isolation using read_committed.
Best Value
Transactions do not replace a correct key or partitioning scheme: they do not turn separate partitions into one ordered log. Nor should ordinary keyed processing—or Kafka transactions alone—be described as “exactly once” for a workflow that also writes to a database or performs another external side effect. Those systems need their own coordination or idempotency design. See the Kafka design documentation and the Go client documentation for the transaction workflow and version-specific API details.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
