Why under-replicated and offline partitions need a diagnosis before a fix
An alert fires for under-replicated or offline partitions, or producers start failing with NotEnoughReplicas errors, and the on-call engineer has to decide what to touch first. This page explains how to find the cause of each state and the recovery step that matches it. The first move is always diagnosis, because the same symptom can come from a dead broker, a full disk, a slow follower or a network path that dropped, and restarting the wrong component can make a partition harder to recover. In Kafka, each partition has a leader and followers, and the in-sync replica set (ISR) is the group of replicas that have caught up to the leader within the allowed lag. A under-replicated partition has fewer in-sync replicas than its assigned replica count, so data is still served but has less redundancy. An offline partition has no leader at all, so reads and writes for it stop until a replica that can lead is elected.
The Kafka documentation describes these states at a high level and points operators to the broker and controller metrics that show them, so start with the metrics your monitoring already exposes rather than with log searches. Ask which brokers are affected, whether the problem is confined to one topic or spans the cluster, and whether the ISR shrank once or keeps flapping. Also check the min.insync.replicas setting on the affected topics, since it determines when a producer with acks=all is refused. Confirm these values in your own environment and on your own topics, because the defaults and your overrides may differ.
Your own runbook should cover single-broker restarts, a known disk replacement or a planned maintenance window. Bring in outside help when an offline partition persists after the leader has been checked, when the controller quorum is unstable, when metadata appears inconsistent across brokers, or when the cause is unclear and production writes are blocked. Before engaging anyone, record the affected partitions, broker IDs, timestamps and the metrics you have already checked, so that the next person can continue the work. Scope any outside engagement in writing, including who has access to the cluster and what changes are permitted during the incident.
What offline, under-replicated and ISR shrink each tell you
An offline partition has no leader, so producers and consumers cannot read or write to it at all. In Kafka the controller elects leaders from the in-sync replica set, and when no replica that is eligible to lead is available, the partition stays offline until one comes back or an operator makes a deliberate choice about unclean leader election. The JMX metric usually watched is kafka.controller:type=KafkaController,name=OfflinePartitionsCount, and it is reported by the active controller, so a non-zero value is a cluster-wide signal rather than a single broker symptom. Offline partitions usually point to several brokers hosting the same partition being down at once, a rack or zone failure, or replicas that fell far enough behind that none of them can be promoted safely.
- Availability impact: Any non-zero count means client traffic to those partitions is failing now, not at some later point.
- Controller scope: Read the value from the active controller, since standby controllers do not report the cluster view.
- Common cause: Simultaneous broker loss, often across a shared failure domain such as a rack, host group or disk type.
- Leader election state: Check whether the partition's replicas are offline or merely out of sync, because the recovery path differs.
- Unclean election: Enabling unclean leader election can restore availability but may lose acknowledged records, so confirm the trade-off with the data owner first.
Before touching configuration, confirm which brokers are down, which partitions are affected and whether any replica still holds the latest data.
A partition is under-replicated when fewer replicas are in sync with the leader than the topic's replication factor. The data is still available and writes continue as long as the in-sync count meets min.insync.replicas, so this signal is usually an early warning rather than an outage. The metric to watch is kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, reported by each broker for the partitions it leads. Because the value is per broker, summing it across the cluster gives the total, and a single broker that stays non-zero often points to a local problem on that broker.
- Reduced redundancy: The partition survives one fewer failure than planned until replicas catch up.
- Per-broker reporting: A stuck value on one broker often isolates a failing disk, network path or overloaded host.
- Slow followers: Followers that cannot keep up with the leader's write rate fall behind and drop out of the in-sync set.
- Planned work: Restarts, reassignments and disk replacements create short, expected under-replication that should clear on its own.
- Persistent cases: A value that does not recover after a restart or a quiet period indicates a fault that needs investigation.
Compare the count against the planned maintenance window and check whether it is confined to one broker or spread across the cluster.
The in-sync replica set, or ISR, is the group of replicas that have recently caught up with the leader. When a follower falls behind for longer than the broker's replica lag allowance, the leader removes it from the ISR, which is a shrink event. When the follower catches up again, it rejoins, which is an expand event. Brokers expose these transitions as rate metrics, commonly kafka.server:type=ReplicaManager,name=IsrShrinksPerSec and IsrExpandsPerSec, and the controller tracks ISR changes in the metadata log. A single shrink followed by an expand is usually harmless, but a steady stream of them is a strong indicator of a replication bottleneck.
- Shrink as a symptom: Shrinks show that a follower missed the replication deadline, not what caused the delay.
- Flapping: Repeated shrink and expand pairs on the same broker often point to network jitter, garbage collection pauses or an overloaded disk.
- Correlation: Compare shrink timing with broker restarts, reassignment jobs, large batch producers and disk latency on the same host.
- Lag configuration: The replica lag allowance decides how sensitive shrinks are, so changing it hides the signal rather than fixing the cause.
- Trend over snapshot: A rate that climbs over hours is more informative than a single spike.
Treat ISR churn as the beginning of an investigation, and confirm which brokers and partitions it touches before deciding on a fix.
Which causes produce these symptoms
Under-replicated and offline partitions usually trace back to one of a few causes, and each leaves a different pattern in broker logs, ISR metrics and producer errors. Start by matching the symptom to the cause, then run the first check before changing any configuration or moving any replicas.
Broker process or host failure
A broker that stops or becomes unreachable drops out of the ISR for every partition it leads or follows. The pattern is a sudden jump in under-replicated partitions across many topics at the same time, with leadership moved to other brokers. Check whether the broker process is running, whether the host rebooted or lost network, and whether the broker rejoins the cluster in its logs. Confirm the broker is down before treating its partitions as a replication problem.
Disk or log directory failure
A failing disk or an unmounted log directory makes a broker unable to write or read segments for the partitions stored there, while the broker process keeps running. The pattern is errors for one log directory, followers on that broker falling out of sync, and partitions that stay offline if the affected replicas are the only copies. Check the broker logs for I/O errors and the disk's health counters, then confirm which log directory the replicas belong to before any move.
Slow followers from I/O or network saturation
Followers that cannot fetch fast enough from the leader fall behind and leave the ISR, even though every broker is healthy. The pattern is one or a few brokers shrinking the ISR repeatedly, with high disk or network utilization on those hosts during the same window. Check disk latency and network throughput on the affected brokers, and compare the follower fetch lag against the leader's produce rate before changing replication settings.
Long GC pauses or overloaded brokers
Long garbage collection pauses or a broker saturated with request handling stall replica fetches and heartbeats long enough for the controller to treat the broker as gone. The pattern is ISR shrink and expand events that repeat on the same broker, with long pauses in the JVM logs and elevated request queue time. Check the GC logs and request handler utilization on that broker, and look for a partition or client load spike that started just before the first shrink.
Misconfigured replication or min.insync.replicas
Some partitions stay under-replicated because the replication factor is lower than intended, or because min.insync.replicas is set equal to or higher than the number of replicas that can be in sync. The pattern is producers receiving NOT_ENOUGH_REPLICAS errors or writes being refused while the broker count looks normal. Check the replication factor and min.insync.replicas on the affected topics against the number of live brokers, and confirm the producer acks setting before changing either value.
Controller or metadata problems
In KRaft mode, the active controller and the metadata quorum decide leadership and ISR changes, so a controller that is unhealthy or out of quorum can leave partition state stale. The pattern is leader and ISR changes that never complete, partitions that show a stale leader, and metadata errors in controller logs. Check the metadata quorum status and the active controller count before restarting brokers, because restarts during a quorum problem can make partition state harder to recover.
How to investigate a shrinking ISR step by step
This sequence moves from the symptom to the likely cause, so an operator can confirm which partitions are affected, which brokers are involved, and whether the fault sits with a broker, its disk, its network or the controller. Each step ends with a check that decides whether to continue on that path.
Confirm scope with topic and partition describe output
Start by running kafka-topics with the describe option against the affected topic, or against all topics filtered to under-replicated partitions. Record each partition's leader, replica list and ISR. A partition is under-replicated when its ISR is smaller than its replica list. Check whether the problem covers one partition, one topic or many. Confirm the exact flags, such as the under-replicated and unavailable filters, in the Kafka 4.3 documentation before you rely on them. Continue only if the shortfall is real and repeatable. If the ISR matches the replica list when you re-check, treat the alert as transient and look at metric history instead.
Identify the brokers that keep dropping out
From the describe output, count which broker IDs appear missing from the ISR most often. A single broker missing from many partitions points to that broker, its disk or its network. Missing brokers spread across unrelated partitions point to a shared cause, such as a controller problem or a cluster-wide network issue. Write down the broker IDs and the partitions they lead or follow. Continue on the single-broker path if one ID dominates. If several brokers are affected, check the network and controller steps first.
Check broker logs for replica fetcher errors
On each affected broker, open the server log in its log directory, which is set by your deployment configuration, and search for messages from the replica fetcher thread and for ISR change entries. Look for repeated fetch timeouts, out-of-range offsets, leader epoch mismatches and errors that name the partitions in step one. Timestamps help you match these entries to the time the ISR shrank. Continue if the logs show fetch failures from one broker toward a specific leader. If the logs are clean, the cause may sit with the controller or the producer configuration.
Check disk and network health on those hosts
Work with the platform or infrastructure team to inspect the hosts you identified. Review disk I/O latency, disk errors in the kernel log, free space on the log directories, and network packet loss or retransmissions between the affected broker and its peers. A failing or saturated disk slows followers, and a flapping network link causes fetch timeouts. Confirm the results with your monitoring history. Continue to recovery if the host is healthy. If you find a failing disk or a link fault, move to the broker or disk recovery path before changing Kafka settings.
Verify the controller is healthy
Check the active controller count and the metadata quorum status with the Kafka tooling, and confirm that exactly one controller is active. Review controller logs for leader election or quorum errors. A controller that has lost quorum or keeps changing leaders can leave ISR updates stalled, so a shrink may persist even after the broker recovers. Continue to the producer check if the controller is stable. If the quorum is unhealthy, resolve that first, because ISR changes cannot be recorded reliably until it is fixed.
Confirm whether producers use acks all
Review the producer configuration for the affected applications and check that acks is set to all, and that min.insync.replicas on the topic is set to a value your durability target requires. If producers use acks set to 1 or 0, a shrinking ISR can lose acknowledged data without any error. If producers report NotEnoughReplicas errors, the ISR has fallen below min.insync.replicas and writes are being rejected. Record the result. This check closes the investigation by showing the business impact, and it decides whether the incident needs a durability review in addition to the technical fix.
Recovery steps for each cause, and when not to force them
Work through recovery in the order a careful operator would: first restore the failed broker or disk and let its replicas catch up, then move replicas off hardware that keeps failing, and only then consider anything that accepts data loss. Most under-replicated partitions clear on their own once the follower is back and fetching from the leader, so the first check is whether the broker process is up, whether its log directories are online, and whether the affected partitions are shrinking back toward full ISR membership. If a disk is failing, do not wait for it to die on its own: plan a reassignment of every replica it holds to healthy brokers, using the same throttled procedure described in the rebalancing guidance for this pillar, and verify that the ISR for each moved partition is complete before retiring the disk.
Restart brokers in a rolling order and wait for ISR recovery before touching the next one. Restarting two brokers that share replicas can take a partition below its minimum in-sync replica count, and the producer errors that follow look like an application fault when the cause is the restart sequence. Confirm after each restart that under-replicated partitions have returned to zero and that the controller is stable before continuing. Do not treat the unclean leader election setting as a shortcut. Enabling it lets an out-of-sync replica become leader, which can drop acknowledged records, and it should only be used when your data owners have accepted that loss in writing for the affected topics.
Changing min.insync.replicas during an incident is a trade-off, not a fix. Lowering it can restore producer writes, but it also lets acknowledged data exist on fewer replicas, so a second failure can lose it. Raising it on a degraded cluster can block writes entirely. Decide the change in writing before executing it: name the topics, the expected duration, the owner who approves the durability risk, and the step that restores the original value. Confirm with your own runbook and the Kafka 4.3 documentation for the exact behavior of the acks and min.insync.replicas combination in your version.
Configuration settings that decide how partitions fail
This table compares the settings that set how a partition behaves when a replica or broker fails, with the trade-off each one makes. Read each row as a choice you make once per topic or cluster, then check the value against your own failure tolerance. The main takeaway is that durability and availability pull against each other, so set these values together rather than one at a time.
| Setting | When it fits | Durability versus availability trade-off | What to confirm in the 4.3 reference |
|---|---|---|---|
| replication.factor | Production topics that must survive a broker loss without data loss or downtime for writes | More replicas protect more copies of each record but increase disk, network and follower fetch load on every broker | Default value, whether it is set per topic at creation, and how it interacts with the number of brokers and racks |
| min.insync.replicas | Topics where a write must reach several in-sync copies before it counts as acknowledged | Higher values block producers with errors when the ISR shrinks, trading write availability for stronger durability | Default value, whether it can be set per topic, and the error producers receive when the ISR is below the minimum |
| acks (producers) | Producers that cannot tolerate losing acknowledged records, paired with idempotence where supported | acks=all waits for in-sync replicas, adding latency and producer errors during shrinks; lower settings return faster but can lose acknowledged data | Default value, the idempotence defaults for the client version in use, and the exact behavior under leader failover |
| unclean.leader.election.enable | Rarely appropriate for business data; considered only when staying offline is worse than losing some records | Allowing a lagging replica to become leader restores availability but can discard committed records, so it should stay off for most topics | Default value, whether it can be overridden per topic, and the cluster-level and topic-level scope |
| replica.lag.time.max.ms | Clusters where brief network or disk pauses should not drop followers out of the ISR | Larger values keep slow followers in the ISR longer, hiding lag and delaying failover; smaller values cause more ISR churn | Default value and how it is read by the broker versus the controller in KRaft mode |
| num.replica.fetchers | Brokers that fall behind leaders on high-throughput partitions while disks and network have headroom | More fetcher threads can speed catch-up but add load on leaders, disks and the network, so raise it only after measuring the bottleneck | Default value, the recommended range in the sizing guidance, and whether it is a static broker setting or dynamically updatable |
Questions teams ask before fixing under-replicated partitions
Can our in-house team fix under-replicated partitions without outside help?
Often yes, if the team already runs Kafka in production, can read broker logs and JMX metrics, and has change windows to act in. The work is mostly diagnosis: separating a failed disk or broker from a slow follower, a network path problem, or a producer configuration that makes ordinary lag look like a fault. Outside help tends to pay off when the team lacks someone who has handled an offline partition before, when the cluster is business critical and the first incident is already underway, or when the fix needs a recovery plan reviewed by a second pair of eyes. Whichever way you go, ask the partner to name what they would check first and what they would not change without your approval.
What drives the effort to troubleshoot and recover a Kafka cluster?
Four factors shape the effort most. The first is cluster size and the number of affected topics and partitions, since each replica state has to be traced to its broker. The second is hardware access: whether your team can inspect disks, controller hosts and network paths directly or has to request them through another group. The third is metrics maturity, meaning whether under-replicated partition counts, ISR shrink events and request latency already exist in a dashboard with history. The fourth is change control, because every restart, reassignment or leader change may need an approved window and a rollback plan. Ask any quote to state which of these it assumes.
What do you need to give a partner to start this work?
A partner needs read access to broker and controller logs, the relevant JMX or exporter metrics for the affected period, and the cluster configuration, including min.insync.replicas, unclean.leader.election.enable and replication settings for the affected topics. It also needs a view of the producer and consumer configuration, acks in particular, and the network and storage layout between brokers. For any step that changes the cluster, you need named change windows, someone who can approve restarts, and a way to reach the on-call team during the work. Confirm in writing who holds credentials, how they are stored, and what is recorded in the change log.
How long does this kind of troubleshooting take?
No duration is given here. The time depends on the written scope: how many clusters and topics are in scope, whether the engagement covers a live incident, a root cause review, a runbook for future events, or all three, and how much access is available on the agreed dates. Incident response is driven by how quickly the replica state can be reached and how many change windows are needed. A review of past events is driven by how complete the logs and metrics are. Ask for the milestones, the dependencies on your team and the acceptance criteria in the scope document before work begins.
Is unclean leader election ever safe to enable on a production topic?
It can be, but only as a deliberate, short-term choice with a known data loss cost. When unclean.leader.election.enable is true, a partition with no in-sync replica can elect an out-of-sync replica as leader, which restores availability but can drop records that were acknowledged on the old leader. For most business data, the safer path is to wait for an in-sync replica to return, restore its disk or broker, or rebuild the replica set. Enabling unclean election should be recorded with the topic, the justification, the expected loss window and the time you plan to switch it back. Treat it as an incident decision that the data owner signs off on, not a default setting.
How do we prevent under-replicated partitions from coming back?
Prevention comes from three layers working together. The first is alerting on the under-replicated partition count, offline partition count, active controller count and ISR shrink rate, so a fault is seen while only one replica is affected. The second is capacity and placement: enough brokers and disk headroom that a single failure does not leave the rest of the cluster saturated, and replicas spread across failure domains. The third is operational discipline: a tested runbook for disk replacement and broker restart, a replication throttle policy for moves, and producer settings that match your durability needs. Verify these with your own failure drills, and ask any partner to show how they would test each control before it is needed in production.
Does DeployOpen issue an audit or certify our Kafka cluster after this work?
No. DeployOpen plans, implements, hardens, documents and supports self-hosted platforms in your infrastructure, and it can prepare systems, controls and evidence for compliance assessments. Audit reports, attestations and certifications are issued by independent auditors and certification bodies. If your organization needs evidence of availability or change control for an assessment, the work can include the runbooks, change records and test results an auditor will ask to see, but the assessment itself is outside what DeployOpen provides.
Security and operational risks during an incident
Incident response on a Kafka cluster often means changing things fast: access rules, listeners, broker settings and leader election. Each of these changes can close one failure and open another, so the cards below cover the risks that most often appear while partitions are offline or under-replicated.
Changing ACLs or authentication mid-outage
Under pressure, teams sometimes loosen ACLs or switch a listener to plaintext so that a client can connect again. In KRaft clusters, the StandardAuthorizer keeps ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so a careless rule change can lock out every client or open a topic to everyone. Before editing, record the current SASL mechanism (such as SCRAM-SHA-512), TLS listener settings and ACLs, and confirm that the change does not remove authentication from any listener.
Exposing JMX and logs during triage
Troubleshooting usually means opening JMX ports, enabling debug logging or copying broker logs to a shared channel. JMX endpoints that are unauthenticated or reachable from a wider network expose broker internals, and broker and controller logs can contain topic names, client addresses and sometimes message fragments. Keep JMX bound to the management network, use TLS and authentication where the deployment supports it, and redact logs before they leave the cluster network.
Unclean leader election and data loss
When every in-sync replica of a partition is unavailable, enabling unclean leader election lets an out-of-sync replica become leader so the partition comes back, but records that existed only on the old leader are discarded. Treat this as a data-loss decision, not a routine restart step. Confirm which replicas were in the ISR before the failure, compare log end offsets, and get sign-off from the data owners before changing the election setting for a topic.
Configuration drift after emergency edits
A broker patched by hand during an incident can run with different replication, listener or log settings than its peers. Drift shows up later as a partition that is under-replicated only on one broker or as a restart that fails on a single node. Compare the effective configuration of every broker and controller after recovery, store the intended values in version-controlled configuration, and remove temporary overrides once the cause is fixed.
Recording every change for audit
Incident changes are often made in a terminal and forgotten by the time the postmortem is written. For each change, record the time, the operator, the command or configuration key, the old and new values, and the reason. Ask your team which audit trail covers broker configuration, ACL changes and controller actions, and confirm that the logs are retained outside the cluster so they survive a broker failure.
Preventing the next under-replicated partition
The most effective prevention is to catch an ISR shrink while it is still a warning. Alert on the under-replicated partitions and offline partitions metrics from every broker and controller, and treat a non-zero count that persists beyond a normal restart window as an incident, not a curiosity. Pair the replica-level alert with a check on min.insync.replicas for each critical topic, since a shrink that drops the ISR below that value is the moment producers start failing with NotEnoughReplicas errors. Agree the thresholds and the on-call response with your team before the first incident, and confirm the exact metric names against the JMX output of the Kafka version you run, because names have changed between releases.
Keep the cluster on a supported release. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and the time-based release plan states that rolling upgrades are supported from each release in the past year to the latest version. Many replication bugs are fixed in bugfix releases, so schedule upgrades on the cadence the project publishes rather than waiting for a failure. Before each upgrade, read the upgrade notes for the target version and confirm that your controllers, brokers and clients meet the Java requirements, since an unsupported runtime is a common cause of a broker that fails to rejoin.
Rehearse broker restarts in a non-production environment that matches your topic configuration, replication factor and throttling settings. Measure how long a controlled restart takes to bring replicas back into the ISR, and record the result so the next maintenance window has a baseline. Finally, spread replicas across failure domains such as racks, availability zones or data halls, and confirm the placement with kafka-topics.sh --describe after every reassignment. Ask your platform team to document which broker sits in which domain, because a placement that looked balanced on paper can concentrate replicas on one rack after a broker is replaced. Related runbooks for disk retention, consumer lag and partition moves are covered in sibling topics in this cluster.
How DeployOpen works on this
DeployOpen approaches partition troubleshooting as a scoped investigation: we collect the evidence first, identify the failing component, and then fix the cause inside your own Kafka cluster. Every step is documented so your team can repeat it during the next incident.
How we scope
- Written scope lists the clusters, brokers, topics and producers covered by the investigation.
- Scope records which metrics, logs and broker configurations we need access to before work starts.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any work begins.
How we build
- We check ISR shrink, offline partition and under-replicated metrics against broker and controller logs to find the root cause.
- We test each recovery step on a non-production copy of the cluster before changing live brokers or topic settings.
- We verify min.insync.replicas and acks settings against producer behavior, then confirm all partitions return to full replication.
How we hand over
- You receive a written incident report covering timeline, evidence, root cause and the exact recovery steps taken.
- Runbooks describe how to check replica health, handle slow followers and recover a failed broker or disk.
- We walk your platform and operations engineers through alert thresholds and the checks to run after each incident.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer for clusters that need regular attention.
- A retainer can include review of alert rules for under-replicated and offline partitions after each Kafka upgrade.
- Ask us to confirm response arrangements and coverage hours in the written scope before you rely on them.
