Original AiFolder technical editorial cover; symbolic partition-to-consumer routing, not a production trace.
Quick answer: A Kafka consumer group lets several consumers cooperate to process a topic's partitions. Each partition is assigned to at most one consumer within a given group at a time. Rebalancing changes those assignments when membership or subscriptions change. Kafka 4.0 introduced a production-ready, incremental consumer rebalance protocol that avoids the older group-wide synchronization barrier.
Partitions versus consumers
A topic with six partitions and three consumers in the same group can typically give each consumer two partitions. Add a fourth consumer and assignments may move. Add a seventh consumer and at least one consumer has no partition to read in that group. More consumers than partitions do not create more parallel partition readers. Separate consumer groups can each read the same topic independently, using their own offsets.
Follow one partition through a small example
Suppose a topic has six partitions and one consumer group has three members. A possible assignment gives each member two partitions; the exact assignment depends on the configured protocol and assignment behavior. If a fourth member joins, the group must change ownership of some partitions. The key invariant is that a partition has at most one active owner within that group at a time. Another group can read the same topic independently, so consumer groups are also the boundary between separate applications using the stream.
Adding more consumers than partitions does not create additional partition-level parallelism for that group. A member may have no assigned partition. If one partition carries most of the work, a visually balanced assignment can still produce uneven processing load. Distinguish partition count, member count, and traffic distribution when investigating throughput. Confluent's consumer design documentation explains these relationships; the six-partition example here is deliberately small so that ownership changes remain easy to inspect rather than suggesting a measured production layout.
Classic versus Consumer rebalance protocol
| Dimension | Classic | Consumer (KIP-848) |
| Assignment coordination | Group synchronization | Coordinator-driven, incremental |
| Global synchronization barrier | Present | Removed |
| Assignment responsibility | Client-side leader/assignor | Server-side assignor |
| Availability | Legacy supported | Generally available since Kafka 4.0 |
| Activation in Kafka 4.x | Existing behavior | Explicit group.protocol=consumer |
See Apache Kafka's version 4.3 operations guide for supported migration paths and limitations.
What actually happens during a rebalance?
This simplified sequence illustrates coordinator-driven incremental reconciliation under the Consumer protocol. The Classic protocol uses a different synchronization process.
Understand why an apparently healthy member moves
A rebalance can follow membership changes, subscription changes, or a member no longer meeting its liveness requirements. The operational symptom may look like a pause or a sudden change in lag. Start by correlating the membership event with the assignment transition before blaming the broker or the application. A long-running handler can make a consumer appear unresponsive even when it has useful work in progress. Keep handler time, poll behavior, heartbeat health, and deployment timing visible as different signals.
Modern consumer protocols can reduce unnecessary disruption, but a new protocol does not make slow processing disappear. Check the broker and client versions and the selected group protocol together. Confluent documents that the newer consumer rebalance protocol is enabled on Kafka 4.0 servers and requires compatible client-side use. Plan a migration around the actual client stack rather than changing one setting based on a release headline. We have not measured pause duration for this article; the timeline illustrates events to collect during an investigation.
Why it matters for availability
A global pause can amplify latency during deployments. Incremental assignment aims to limit disruption to affected partitions. Application-level pauses, slow processing, offset commit behavior and downstream dependencies can still cause consumer lag.
Reducing unnecessary rebalances
- Keep processing time within the consumer's configured poll and liveness expectations.
- Use graceful shutdown and avoid repeatedly restarting every member together.
- Choose group membership and deployment strategy based on your workload.
- Inspect coordinator and consumer metrics before tuning timeouts.
- Evaluate the Consumer protocol where client, broker and assignor compatibility permit.
- Make message processing idempotent, because replay after failures can happen independently of rebalance optimization.
Common mistakes
More consumers always means more throughput: false once active consumers outnumber partitions. Rebalancing guarantees no duplicates: false; processing and offset commits still require a delivery-semantics strategy. Consumer protocol eliminates all pauses: false; it removes the global synchronization barrier but cannot eliminate all workload interruptions.
Practical test plan — pending
Create a topic with six partitions and three consumers. Record assignment and lag. Add one consumer, then remove one, comparing Classic and Consumer protocols under identical traffic. Measure partition-unavailable time, rebalance duration, processed-message rate and lag recovery. Repeat across several trials and report medians plus worst-case values. Do not claim benchmark improvements without running the experiment.
Protect side effects when ownership changes
A consumer that writes to an external system must consider what happens when work finishes near an ownership transition. Processing and offset management should match the intended delivery semantics, and external writes should tolerate retries where duplication is possible. For an illustrative email job, use a stable event identity to prevent an accidental second send rather than assuming the group assignment alone provides end-to-end exactly-once behavior. The external system is part of the correctness story.
Before changing the group, prepare a test that adds a member, stops a member, and delays a handler while observing assignment, lag, and duplicate side effects. Capture topic partitions, group identifier, client version, protocol, and relevant timeout configuration with the result. A successful consumer count increase should improve the actual bottleneck without losing or duplicating the business effect. If a hot partition remains the limit, adding more members may only create idle capacity. Use the collected evidence to choose between handler changes, partitioning changes, and group configuration changes. Keep assignment snapshots beside the application logs so an operator can connect the business symptom to the ownership event.
Source of truth / References
Primary
- Apache Kafka 4.3 — Consumer Rebalance Protocol
- Apache Kafka 4.3 — Basic Kafka Operations
- Apache Kafka KIP-848 — Next Generation Rebalance Protocol
Secondary / idea sources
- anhthii/kafka-notes — topic discovery only, not the authority for protocol behavior.
Validation
Documentation reviewed October 8, 2026. No AiFolder benchmark has been executed.
Partition assignment diagram
This is an illustrative assignment, not a measured deployment.
Inspect a consumer group
The following commands are read-only. Replace the example bootstrap server and group name with your environment.
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-workers
The output includes current offset, log-end offset and lag for each assigned partition.
Inspect partition ownership
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-workers --members --verbose
Check which members own which partitions before and after a rolling deployment. For the newer Consumer protocol, authorization requirements can differ from Classic; see the official operations documentation.
Rebalance lifecycle
This illustrates the incremental Consumer protocol concept; it is not a packet-level trace.
