To preserve order in a Go Kafka consumer, process records sequentially within each partition, and commit only after the required work succeeds. Kafka guarantees order within a partition—not across a whole multi-partition topic—and a consumer that starts handlers concurrently can undo that guarantee at the application level. The examples below use Segmentio’s kafka-go; these APIs and settings are specific to that client, not universal Go or Kafka configuration.
What order does Kafka guarantee?
Kafka records in a partition have an ordered sequence of offsets. A topic with multiple partitions does not have one total order across all its records: records in separate partitions can be consumed independently, and there is no meaningful global ordering guarantee between them.
If a business operation depends on a sequence—such as successive updates to one account—those records must be routed to the same partition. A common design uses a stable record key so related records are partitioned together. The consumer can then preserve their sequence by processing that partition’s records in offset order. Check the producer’s keying and partitioning behavior too; consumer settings cannot restore an order that was split across partitions.
Why receiving records in order is not enough
A consumer may fetch records in offset order and still apply their effects out of order. For example, it can start work on offset 12, then start offset 13 before 12 finishes. If the second handler updates downstream state first, the state change order no longer matches Kafka’s record order.
#1 Best Overall
For each partition whose effects must remain ordered, allow at most one operation to be in flight at a time. You can still process different partitions concurrently when their work is independent. If one partition’s handler fails, do not let a later record from that partition overtake it when the business invariant requires the failed operation to be retried first.
Fetch, process, then commit with kafka-go
In kafka-go consumer-group mode, ReadMessage automatically commits a message and may do so before the application has finished processing it. The project’s Reader source recommends FetchMessage and CommitMessages when the application needs control over commit timing. Confirm the API against the library version pinned in your project; the linked source is the mutable main branch.
A straightforward ordered loop is:
for {
msg, err := reader.FetchMessage(ctx)
if err != nil {
// Handle cancellation, shutdown, and fetch errors.
return err
}
if err := handle(ctx, msg); err != nil {
// Do not commit this message or advance past it.
return err
}
if err := reader.CommitMessages(ctx, msg); err != nil {
// The side effect may have succeeded; handle possible replay safely.
return err
}
}This pattern makes the application’s commit point explicit: the loop commits only after handle reports success. Choose a failure policy that does not continue past failed work. A commit error after a successful side effect creates an important ambiguity: Kafka may not have recorded the commit, so the record can be delivered again. Make side effects idempotent where possible, or otherwise design retries to tolerate duplicates. The kafka-go package documentation describes explicit commits; verify the documentation for the version you use.
Treat commits as per-partition watermarks
Kafka stores a committed offset for each partition. In kafka-go, committing a higher offset for a partition also commits the earlier offsets for that partition. In effect, the committed position is a watermark through the supplied message, not an acknowledgment limited to that one record. The rule is documented in the package documentation and the Reader source.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRank #3
Suppose offsets 1, 2, and 3 have been fetched from one partition. If offset 3 finishes first and the consumer commits it while offset 1 or 2 is still unfinished, the commit advances past that earlier work. A crash can then prevent the unfinished record from being delivered again. Do not commit past unfinished earlier work.
How to add concurrency without reordering a partition
One processing sequence per partition
This is the simplest model: process and commit one record at a time for a partition, while allowing separate partitions to run concurrently. It is usually the easiest design to reason about when each record changes state that depends on its predecessor. Its trade-off is that a slow handler holds up later records in that partition.
Rank #4
- Metamorphosis: Franz Kafka (Little Clothbound Classics)
Per-partition workers or completion tracking
If the application needs parallel work, dispatch by partition and permit no more than one in-flight operation for each ordered sequence. Where work within a partition must run concurrently, maintain a completion tracker and commit only the highest contiguous completed offset. A later record may finish first, but it must not move the commit watermark past an earlier unfinished record.
Concurrency also makes retries, shutdown, and consumer-group rebalances more involved: partition ownership can change while work is in flight. Ensure an old worker cannot commit work after it has lost safe ownership. The exact orchestration and fencing approach depends on the client version and application architecture; the offset rule alone does not provide that coordination.
Free tools Windows power users keep installed
One-click scans. No signup required.
Best Value
Which settings affect ordering—and which do not?
| Setting or choice | Documented value or behavior | What it means for ordered processing |
|---|---|---|
| Processing concurrency | No universal worker count is established. | Limit in-flight work per partition or track contiguous completion. More workers do not themselves preserve order. |
kafka-go QueueCapacity |
The mutable main-branch Reader source documents a default of 100. Verify the pinned release. | Controls the Reader’s internal message queue; buffering is not a limit on in-flight application work per partition and does not enforce side-effect order. |
kafka-go CommitInterval |
The mutable main-branch Reader source documents zero as synchronous commit handling. Verify the pinned release. | Zero makes the commit point synchronous. A nonzero interval enables periodic commit handling; that can reduce commit overhead but may increase successfully processed work replayed after a crash. The setting does not make downstream effects atomic with the Kafka commit. |
Apache Kafka Java consumer max.poll.interval.ms |
Apache Kafka 4.1 documents a Java-client default of 300000 ms (5 minutes). | This is a Java consumer setting describing the maximum delay between poll calls before a consumer can be considered failed and a rebalance can occur. It is not a kafka-go ReaderConfig value. |
Apache Kafka Java consumer max.poll.records |
Apache Kafka 4.1 documents a Java-client default of 500. | This limits records returned per poll in the Java client, not underlying fetch behavior. It is not a kafka-go setting. |
The kafka-go defaults above come from its Reader source, which can change; check the version in go.mod and its corresponding documentation before relying on a default. The poll settings come from the Apache Kafka 4.1 Java consumer configuration reference. Do not copy Java client values into Go configuration as if they were shared settings.
There is no universal queue capacity, commit interval, worker count, timeout, or batch size for an ordered workflow. Tune against the handler’s latency distribution, partition count and key distribution, acceptable replay after failure, and the idempotency of downstream effects. Validate the result under the failure modes that matter to the application.
When does read_committed matter?
Kafka’s read_committed isolation setting restricts a consumer to committed transactional messages up to the last stable offset; records behind an open transaction may remain unavailable until it completes. This can affect visibility and latency, but it does not make arbitrary downstream effects execute in order. Use it when the producer’s transactional isolation requirements call for it. The behavior is described in the Apache Kafka 4.1 consumer configuration reference.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
