Why your Kafka consumer group keeps rebalancing
A rebalance is the process by which a consumer group reassigns topic partitions among its members, and in most production incidents the trigger is a consumer that stops polling, misses its heartbeats or joins and leaves the group too often. While a rebalance runs, partitions are paused, so lag climbs, and when a partition is handed to another member before offsets were committed, the same records can be processed twice. If your group keeps moving partitions, shows lag spikes that recover only slowly, or downstream systems report duplicates after deploys or traffic peaks, the group is not behaving as a stable unit and needs diagnosis before anyone tunes timeouts blindly.
This page is for the platform or application team that owns the consumers, and for the operator who is asked why the group never settles. It walks through the usual causes in order: slow processing that exceeds max.poll.interval.ms, membership churn from container restarts or autoscaling, and the difference between the classic and consumer group protocols introduced with KIP-848. It then shows which metrics and broker and client logs confirm each cause, so you can tell a processing problem from a deployment problem before changing configuration. The default group.protocol value is classic, so the first question to answer is which protocol your clients actually run.
Ask your team three things before you start: which client library and version each consumer uses, whether the application processes records on the polling thread or hands them to workers, and how often instances restart during normal operation. These answers decide which sections apply and how much testing you need. Consult the consumer configuration reference for the exact behavior of each setting in your Kafka version, because defaults and interactions change between releases. DeployOpen can scope a diagnosis and fix as a fixed-scope engagement once you have shared that information, and the time it takes depends on the number of groups and the state of your metrics.
Settings that control rebalance behavior and when each fits
This table compares the consumer settings that decide when a group rebalances and how a member is treated after a pause or restart. Read each row for the symptom it addresses and its trade-off, then confirm defaults and accepted values in the consumer configuration reference for your Kafka version before changing anything.
| Setting | Use it when | Trade-off to weigh |
|---|---|---|
| group.protocol | You want the KIP-848 consumer group protocol (classic or consumer), for example to reduce rebalance pauses in large groups | Confirm client and broker support for your version first; the classic protocol remains the default setting |
| max.poll.interval.ms | Record processing per batch is slow or variable and polls are taking longer than expected | A higher value tolerates slow batches but delays detection of a truly stuck consumer |
| max.poll.records | Each poll returns more work than can finish inside max.poll.interval.ms | Lower values mean more polls and more overhead per record, but shorter batches |
| group.instance.id | Consumers restart often (deploys, pod restarts) and you want the group to keep their assignments | Static members must have unique IDs; a duplicate ID causes fencing errors and the member stays out of the group |
| session.timeout.ms | Consumers lose connectivity briefly and you need the broker to tolerate short silences before evicting a member | Longer timeouts keep dead members assigned longer, so their partitions stall until the timeout expires |
| heartbeat.interval.ms | You need faster or slower membership signals; keep it well below session.timeout.ms | Frequent heartbeats add broker load; infrequent ones delay failure detection |
Matching the symptom to the cause
This pattern appears when a consumer takes longer to process one batch than the allowed gap between calls to poll(). The consumer is then treated as failed, and the group reassigns its partitions while the slow instance is still working. Typical causes include database writes that stall under load, downstream HTTP calls without timeouts, and batches that grow after a traffic spike because the consumer fetches a large number of records per poll.
- Gap between polls: Check whether processing time per batch exceeds the value of
max.poll.interval.ms. A batch that occasionally runs long is the usual trigger. - Batch size: Lowering
max.poll.recordsshortens each batch so the consumer returns to poll() sooner. Test this before raising any timeout. - Timeout change: Raising
max.poll.interval.msgives slow batches more room, but it also delays failure detection for a consumer that is truly stuck. - Confirming metric: Look for rebalance events that line up with the slowest batch durations in your application logs, and for lag rising on the partitions that were moved.
- Confirming log: Search consumer logs for the message that reports the poll interval was exceeded, which names the member that left the group.
To confirm this pattern, compare batch processing time against the poll interval for the affected group and check whether the rebalance follows the slowest batch.
This pattern appears when every restart of an application instance triggers a rebalance, and a rolling deploy causes a series of them. With dynamic membership, the broker treats a stopped consumer as having left the group and reassigns its partitions immediately. The new instance then joins, which triggers another rebalance. A deploy of many instances can therefore produce many rebalances in a short time, each pausing consumption for the group.
- Static membership: Setting
group.instance.idto a stable value per instance lets a restarted consumer reclaim its previous partitions within the session timeout, which reduces churn during deploys. - Session timeout: Check
session.timeout.msagainst how long your application takes to shut down and start up. A restart that exceeds this value is still counted as a departure. - Shutdown behavior: Confirm that consumers close cleanly and leave the group on shutdown, rather than being killed mid-poll.
- Confirming metric: Compare rebalance counts with deployment timestamps in your release tooling. A rebalance that starts within seconds of each pod or instance restart points to membership changes.
- Confirming log: Look for join and leave messages from each member during the deploy window, and check whether the same member identities return after the restart.
Before changing settings, record the restart duration of one instance and check whether the instance keeps its identity when it returns.
This pattern is the hardest to explain because the consumer count stays the same and no deploy or crash is visible. The group still rebalances, which suggests the coordinator is seeing heartbeats or membership changes that the application does not expose. Common causes include heartbeats delayed by long garbage collection pauses or CPU starvation on the consumer host, network interruptions between consumers and brokers, and a change in the group protocol or client version that alters how membership is managed.
- Heartbeat health: Check whether heartbeat timing is delayed by JVM pauses or host contention. Tune the heartbeat settings only after confirming the cause.
- Network path: Look for connection resets or intermittent timeouts between consumer hosts and brokers, especially after load balancer or firewall changes.
- Protocol setting: Confirm the value of
group.protocolon each consumer. Mixing classic and consumer protocol clients, or changing the value during a rollout, can cause unexpected rebalances. - Confirming metric: Track rebalance count alongside consumer host CPU, garbage collection pause time and broker request latency over the same window.
- Confirming log: Read the group coordinator and consumer logs for the reason given at each rebalance, and check whether the same member is repeatedly removed.
Ask the platform team to confirm the consumer host and network history for the rebalance window before changing any group setting, because this pattern is often environmental.
How to find the cause in metrics and logs
This sequence turns a vague complaint that the group keeps rebalancing into a confirmed rebalance rate, a named trigger, and a specific consumer instance and poll loop to investigate. Each step narrows the cause before any setting is changed.
Confirm the rebalance rate from group state
Start by checking whether the group is rebalancing as often as the symptoms suggest. Use the group describe output and watch the group state and member list over a period when the problem is visible, such as processing delays or duplicate work. Record how often the group moves through its rebalancing state and whether the member count changes at the same moments. The step is done when you have a timestamped count of rebalances and can say whether they follow a pattern. Confirm exact command options against the Kafka version you run.
Pull coordinator and consumer logs for the same window
Collect the group coordinator log lines on the broker that owns the group and the log lines from each consumer for the same time window. Look for messages that show a member joining, leaving, or being removed from the group, and for the reason logged with each event. A consumer removed after missing its poll deadline reads differently from one that closed its connection or was fenced by a newer instance. The step is done when every rebalance in your window has a logged cause attached to a member ID.
Compare poll timing with per-batch processing time
For each consumer that left the group, compare how long a batch takes to process with the maximum allowed gap between poll calls set by max.poll.interval.ms. If processing time for a batch regularly approaches or exceeds that value, the consumer is being removed for being slow, not for crashing. Check the client metric for time since the last poll and the metric for average and maximum poll interval if your client version exposes them, and confirm names against your client release. The step is done when you can state whether the gap between polls is the trigger.
Check consumer lag alongside the rebalances
Review consumer lag for the group over the same window. Lag that rises just before each rebalance suggests a consumer stalled on a slow batch or a blocked downstream call, which then fell behind the poll deadline. Lag that stays flat while rebalances continue points toward membership churn rather than slow processing. Use the records-lag-max client metric or the group describe output, and compare per-partition values if one partition dominates. The step is done when lag behavior either matches or rules out the processing-delay explanation.
Check for restarts, deployments and scaling events
Match each rebalance timestamp against deployment logs, container restarts, autoscaling events, and pod or host reschedules. Frequent restarts or scale changes produce joins and leaves that look like storms but are caused by the platform. Also check whether instances are being started with a shared or missing group.instance.id, since that affects how the group treats a returning member. The step is done when every rebalance is either explained by a platform event or flagged as unexplained.
Record which instance left and why
For each unexplained departure, write down the consumer member ID, the host or pod name, the partitions it held, the logged reason, and the last poll time before it left. Keep this as a short table so patterns are visible, such as one instance that always leaves or one partition that always stalls the same consumer. The step is done when you can name a single instance and a single cause for the most frequent rebalances, and you have confirmed the group protocol in use through the consumer configuration, since the classic and KIP-848 protocols differ in how membership changes are handled.
Fixes to test, from smallest change to largest
Rebalance storms usually have one cause, and the fixes below run from configuration changes that touch one consumer to protocol and topology changes that touch the whole group. Start with the smallest change that matches the metrics and logs, and test each one in staging with a replay of production-like traffic before it reaches production.
Shorten poll batches
Lowering max.poll.records reduces how many records each consumer must process between calls to poll(). It is the smallest change because it needs no code or topology change. The risk is lower throughput, since each poll returns less work and more round trips are made. In staging, confirm that the time per batch stays well under max.poll.interval.ms at your slowest partition and that lag still drains at peak produce rate.
Tune the poll interval
Raising max.poll.interval.ms gives slow processing more time before the broker considers the consumer failed. It hides the symptom rather than the cause, so a slow handler will still stall its partitions. The risk is that a truly hung consumer keeps its partitions longer before they are reassigned. In staging, measure the real p99 processing time per batch and set the interval with headroom above it, then confirm that failure detection still meets your recovery target.
Add static membership for restarts
Setting group.instance.id on each consumer lets a restarted instance reclaim its partitions without triggering a full rebalance, as long as it returns within session.timeout.ms. This targets deploys and pod restarts, not slow processing. The risk is that two instances sharing an ID cause fencing errors, and a crashed instance keeps its partitions idle until the timeout expires. In staging, rolling-restart one consumer at a time and check that the group generation does not change.
Move to the consumer protocol after testing
Setting group.protocol to consumer opts the group into the KIP-848 protocol, where the broker coordinates assignment and reduces the stop-the-world pauses of the classic protocol. The default is classic. The risk is behavior and client-side differences, including how assignors and some tooling report group state, so the switch belongs after the lighter fixes. In staging, confirm your client versions support it, run the group under load, and compare rebalance counts and assignment fairness before and after.
Separate slow work from the poll loop
Handing records to a worker pool or queue lets the poll loop keep calling poll() while slow work runs elsewhere. This is a code change and the largest change in this list. The risk is weaker ordering guarantees and offset commits that can run ahead of completed work, which can lose or duplicate records after a failure. In staging, test failure during in-flight work, confirm offsets are committed only after completion, and verify that partition ordering matches what downstream consumers expect.
Review partition counts per group
A group with many more consumers than partitions leaves some members idle, while a group with few partitions per consumer may concentrate load and slow each batch. Reviewing the partition count changes how work spreads across members, but adding partitions to a keyed topic changes which partition each key maps to. The risk is breaking per-key ordering for existing data. In staging, check assignment per member and per-partition processing time, and confirm the key mapping change is acceptable to every consumer of the topic.
Questions teams ask before fixing rebalances
How long does it take to troubleshoot a consumer rebalance problem?
Duration depends on how many groups are affected, how much history your logs and metrics retain, and whether the cause is in application code or in group configuration. A single group with clear logs and a reproducible slow handler is usually a much shorter investigation than a platform-wide storm across dozens of services owned by different teams. No fixed duration applies to every cluster. Ask for a written scope that names the groups, the time window of evidence available and the environments where changes can be tested before anything reaches production.
Can our internal team fix rebalances without outside help?
Yes, many teams do. The work is mostly reading consumer metrics, matching rebalance events to application behavior and tuning settings such as max.poll.interval.ms, max.poll.records and session timeouts, or moving a consumer to static membership. It becomes harder when the cause spans application code, container restarts and broker-side group coordination at the same time, or when no one on the team has tuned a consumer group before. An outside review is most useful when the first pass of tuning does not stop the rebalances.
What do you need from our team to start?
Plan to provide consumer application logs covering rebalance events, the group's describe output over time, consumer lag metrics, deployment and restart history for the consumers, and read access to the relevant cluster and topics. A staging environment that reproduces the consumer load is valuable, because changing max.poll.interval.ms or the group protocol in production without a test run is risky. Also name the owner of each consumer service, since many fixes live in application code and need that team's review.
Is it safe to switch from the classic protocol to the consumer protocol?
It can be, but it is a change to group behavior that should be tested first. The consumer group protocol setting accepts classic or consumer and defaults to classic, and the KIP-848 consumer protocol became generally available in Kafka 4.0. Before switching, check your client library versions, your Kafka cluster version and whether your consumers use features that the new protocol handles differently. Rolling the change out to one group in staging, then to a single production group with monitoring in place, is the usual approach.
What cost components should we expect in a rebalance fix?
The cost comes from analyst and engineering time: reviewing metrics and logs, reproducing the problem, changing consumer configuration, testing in staging and, where needed, changing application code owned by your teams. Kafka itself is licensed under Apache-2.0 with no paid edition from the project, so the software is not a cost line. Ask any provider for a quote that separates the investigation, the configuration changes, the code changes your developers make and any ongoing monitoring or retainer support.
What does a DeployOpen engagement include for rebalance troubleshooting?
DeployOpen works on self-hosted Kafka inside your own infrastructure. A fixed-scope engagement typically covers reviewing consumer group metrics and logs, identifying the cause of the rebalances, recommending and applying configuration changes agreed in writing, and documenting the findings for your team. Scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that scope. Ongoing monitoring and tuning are available as a retainer. DeployOpen does not publish a price list.
Will changing consumer settings affect message delivery or data safety?
Some settings change how long a consumer can go without polling before the group treats it as failed, and that affects when partitions move between consumers. Raising max.poll.interval.ms gives slow handlers more time but delays detection of a truly stuck consumer. Static membership reduces rebalances caused by brief restarts, but it needs stable instance identifiers. Review each change with the owner of the consumer, confirm offset commit behavior in the application, and test redelivery of records after a rebalance so that no duplicate processing surprises the business logic.
How DeployOpen approaches consumer rebalance work
DeployOpen starts every rebalance engagement with a written, fixed scope that names the consumer groups in question, the applications and runtimes that join them, the Kafka version in use, and the evidence the client can provide. Before any change is proposed, the team reviews the group's behavior in the client's own environment: the kafka-consumer-groups output for each group, the broker logs that record group coordinator transitions, and the client-side logs that show when max.poll.interval.ms was exceeded or a member left. The scope also records the current group.protocol setting on each consumer and whether the cluster runs a release where the next-generation protocol is generally available. Confirm the Kafka version and the client library versions with the platform owner first, because the protocol choice depends on both.
DeployOpen then tests the candidate fixes in the client's infrastructure, not on a generic lab. Typical checks include tuning max.poll.interval.ms and max.poll.records against measured processing time, assigning group.instance.id to a canary consumer to confirm static membership behaves as expected during a planned restart, and running a controlled move of a group to the consumer protocol in a non-production environment. Each test is run with the same metrics and logs the client uses in production, so the results can be checked by the client's own operators. Deliverables are agreed in the scope, usually a written findings report, the configuration changes with their rollback steps, and a set of alerts or dashboards for the group's rebalance signals.
Once the group is stable, the client may want help keeping it that way. DeployOpen offers retainers for ongoing support, upgrade testing across Kafka releases, and continuous review of consumer behavior as new applications join the cluster. A retainer is a separate written agreement, so its scope, response expectations and fees are set in writing and are not implied by the fixed-scope work. Ask any provider, including DeployOpen, to state in writing what is included, who runs the changes, and how the client's staff are trained to operate the result.
