When decommissioning a Kafka broker or log directory is the safe path
You are retiring a server, replacing a disk that is starting to fail, or shrinking a cluster that has more capacity than it needs. In each case, removal is a planned replica-move exercise, not a shutdown. Every partition replica on the broker or disk has to be moved to other brokers and fully caught up before the broker is unregistered, or the data on it is lost or under-replicated. Stopping the process first and sorting out the gaps later is the failure mode this work exists to prevent.
In Kafka 4.3, a *log directory* is one of the storage paths listed in the broker's log.dirs setting, typically a mounted disk. Each partition replica lives in a subdirectory of one log directory on one broker, so a broker with several disks holds replicas that are spread across all of them. That is why a broker can host replicas you would not guess from the topic list: internal topics such as __consumer_offsets and __transaction_state, topics owned by other teams, and replicas that were moved to the broker months earlier and never cleaned up. Before you plan anything, inventory replicas per broker and per log directory rather than per topic.
Choose the path by scope. A rebalance is the right tool when the cluster is healthy and you only want to even out load. A failed-broker recovery applies when the broker is already gone and its partitions are relying on remaining replicas. Decommissioning applies when the hardware or disk is leaving on purpose, and it needs both the source broker and the destination capacity to be healthy throughout. Confirm the exact cordon and unregistration steps for your Kafka 4.3 build against the Kafka 4.3 upgrade notes, since behavior around KRaft broker registration can change between releases.
Broker removal versus log directory removal
Removing a whole broker changes placement for every partition replica it hosts. Each replica on that broker needs a copy on a remaining broker before the broker stops, and any partition it leads needs leadership moved first. If the broker is removed while copies are still missing, partitions can lose a replica, fall under-replicated, or go offline when the producer acknowledgement settings cannot be met.
- Placement change: Every partition with a replica on the broker needs a new replica set built from the remaining brokers, and leadership must move off the broker before it stops.
- Move first: Generate one reassignment plan that covers all affected replicas, execute it with a replication throttle, and wait until every reassignment reports complete.
- Avoid: Do not shut the broker down or unregister it while its log directories still hold replicas, because the cluster still expects those copies.
- Verify: Confirm with the topic describe output that the broker appears in no replica set or leader position, and that under-replicated and offline partition counts match the pre-removal baseline.
- Record: Save the plan, the throttle value and the completion output so the change can be audited later.
Confirm the broker registration status and the topic-by-broker listing with your own tooling before you stop the process.
A broker can keep running when one of its log directories is retired, provided the replicas on that directory move to other directories or brokers first. The broker keeps its identity and its other partitions stay where they are, so the work is a directory-level move plus a cordon that stops new replicas from landing on the disk. Treat the move as a placement change for only the partitions on that disk.
- Cordon the directory: Stop the broker from placing new replicas on the directory before draining it, so the drain does not refill it.
- Reassign by directory: Build the plan from the directory's replicas and target other log directories on the same broker or on other brokers, using the log directory field in the reassignment JSON.
- Check headroom: Confirm the remaining directories have enough free space for the moved data before starting. Insufficient space on surviving disks is a frequent cause of stalled moves.
- Avoid: Do not remove the directory from
log.dirsor delete its files while any replica still references it, because the broker can fail to start or report offline partitions. - Confirm: Verify that the directory no longer appears in replica placement, then restart the broker with the directory removed from its configuration.
Check the per-directory disk usage on the broker against the expected layout before you retire the disk.
In-place replacement keeps the broker ID and hostname while the server or disk underneath changes. Placement stays the same on paper, so the key decision is whether the broker rejoins with its existing log data or comes back with empty log directories and catches up through replication. Each path carries different durability and traffic costs, and the choice should be made before the maintenance window opens.
- Keep the identity: Preserve
broker.idand the advertised listeners so clients and existing assignments continue to resolve to the same broker. - Choose the data path: Either reattach the existing log directories or start with an empty directory and let replicas rebuild. Rebuilding makes replication traffic and throttle settings the main operational concern.
- Move leadership first: Shift leaders off the broker before the outage so producers and consumers see fewer avoidable errors.
- Avoid: Do not reuse a log directory that carries a different cluster ID, and do not bring back a broker with stale data under a new identity, since either can produce divergent replicas.
- Confirm: Check that every affected partition returns to a full in-sync replica set and that preferred leaders are restored after the broker rejoins.
Ask your hardware or platform team to confirm which disk identifiers and mount paths survive the swap before the window begins.
Options that shape the removal plan
This table compares the settings and tooling choices that change how a broker or log directory removal runs and what it puts at risk. Read each row as a decision to make before the reassignment plan is written, and confirm exact flags against the Kafka version you have installed. The main takeaway is that throttling, replica headroom and target selection decide how long the removal takes and whether the cluster stays protected while it runs.
| Option | What it controls | Trade-off | Confirm before you start |
|---|---|---|---|
| Reassignment throttle | Caps the bandwidth replicas use while they copy data to new brokers or directories during the plan | A low throttle protects producers and consumers but lengthens the move; a high one shortens it but competes with live traffic | Throttle settings and the command that applies and removes them in your installed kafka-reassign-partitions version |
| Replication factor headroom | Number of brokers or directories left to hold each replica after one is removed | Removing from a cluster at its minimum replica count leaves no spare copy during the move | Replication factor of every topic, including internal topics, against the broker count left after removal |
| min.insync.replicas headroom | Number of in-sync replicas required before producers with acks=all can write | If the remaining in-sync count falls below this value during the move, producer writes fail | Topic and broker-level values of min.insync.replicas and the producer acks settings used by each application |
| Target broker selection | Which surviving brokers receive the replicas, and whether they are in the same rack or failure domain | Even spread protects against a single failure; uneven targets create hot brokers and disk pressure | Broker rack assignments, free disk per target and current leader distribution across brokers |
| Target log directory selection | Which surviving data directories on a broker receive replicas when only one disk is removed | Moving to a disk with little space can trigger the disk-full issues covered in the retention and disk pages | Free space and existing partition count on each remaining log directory on the same broker |
| Log directory cordon approach | Whether the directory is excluded from new placement before its replicas are moved | Excluding it early stops new data landing there but still requires a plan for replicas already on it | Whether your version supports the directory operations you intend to use, and the exact command for each step |
| KRaft unregistration tooling | The command path used to remove the broker's registration from the KRaft metadata quorum after its data is gone | Running it too early removes a broker that still holds replicas; running it late leaves a stale registration | Which controller-side command and broker ID your installed version expects, and whether the broker is already shut down |
| Verification method | How you confirm that no partition, replica or leader still references the removed broker or directory | Describe-based checks are quick; a full cross-check of every topic is slower but catches leftover replicas | Describe output for every topic, with the removed broker ID absent from replicas, ISR and leader fields |
How to cordon, reassign and unregister a broker
This sequence moves every replica and leadership role off a broker or log directory before the broker is removed, so the cluster keeps its replication factor and no partition is left depending on the retired hardware. Each step ends with a check that must pass before the next one starts.
Inventory every replica and leader on the target
Before touching configuration, list each partition that has a replica on the target broker or log directory, and note which of those replicas currently hold leadership. Use the topic describe output and your own topic inventory to build the list, and record topic, partition, replication factor and the current replica set. The step is complete when the list matches what the broker reports in its log directory listing, and the team agrees on the total count of partitions to move. Keep this inventory as the baseline for the final check.
Confirm capacity and headroom on remaining brokers
Check that the brokers that will remain can absorb the replicas being moved without exceeding their disk or network budgets. Compare current disk usage per broker and per log directory against the size of the replicas in the inventory, and confirm that the remaining brokers still have room for the expected growth on each partition. Also confirm that the replication factor can still be met on the smaller set of brokers, and that no rack or failure domain loses its last replica. The step closes when the owning team signs off the capacity numbers in writing. If headroom is tight, the decision is to add capacity first, not to proceed on hope.
Cordon the log directory or broker from new placement
Mark the target so that no new partition or replica lands on it. For a single disk, mark that log directory as offline for new placement through your change process and confirm that new topics and reassignments exclude it. For a whole broker, confirm that the broker is excluded from every placement plan you generate. Verify by creating a test plan or reviewing the proposed placement, and checking that the target appears nowhere in it. Check that producers and consumers are not directed to the broker by any static bootstrap list you control, and update those lists. The step is done when a generated placement excludes the target completely.
Generate and review the reassignment plan
Build a reassignment plan that moves every replica in the inventory to brokers that remain, using the reassignment tooling supplied with Kafka. Review the proposed replica lists for each partition, checking that the replicas do not land on brokers that already hold a copy, that rack or failure-domain spread is preserved, and that the total data moved matches the capacity you confirmed. Save the current assignment alongside the proposed one so you have a rollback reference. The step closes when a second engineer has reviewed the plan file and confirmed that its partition count matches the inventory.
Execute the plan with replication throttling
Start the reassignment with a replication throttle so that the move does not starve producers and consumers of network or disk bandwidth. Choose the throttle value with your own network team, based on measured spare capacity, and record it. Run the execute step against the reviewed plan file, then poll the status until the tool reports that the reassignment is complete for every partition. Watch under-replicated partition counts and request latency during the move, and pause if producers begin to see errors. The step is complete when status reports no in-progress partitions and the throttle is removed.
Verify replica placement and leadership
Confirm that the reassignment took effect by describing the affected topics and checking that no replica set still includes the target. Next, verify that the in-sync replica sets are complete and that no partition is under-replicated. Leadership may still sit on the target only if it has been reassigned, so run a preferred leader election for affected partitions and confirm that leaders have moved to brokers that remain. The step closes when the describe output shows zero partitions with the target in the replica list or as leader, and zero under-replicated partitions.
Stop the broker and remove its log directory
With the target holding no replicas, stop the Kafka process on the broker or take the log directory out of service in the configuration, following your change record. Keep the data directory until the final check is complete, so that the state can be recovered if an unexpected replica is found. Confirm from the remaining brokers' logs that no fetch requests from the target are still arriving. The step is done when the process is down and the change record shows the exact time the broker stopped.
Unregister the broker and confirm zero partitions remain
Unregister the broker from the cluster using the procedure documented for your Kafka version, confirm that it no longer appears in the broker list, and check that the controller quorum is healthy. Then repeat the inventory from the first step across all topics, confirming that no replica or leader remains on the removed broker. Compare the final inventory with the baseline and record the result. The decommission is complete only when the cluster reports the expected broker count, no offline or under-replicated partitions, and zero partitions remain on the removed broker or directory. Archive the plan, the verification output and the change record together.
What goes wrong when replicas are left behind
The most common failure in a broker or disk removal is not a crash but a quiet gap: replicas that were never moved. Unregistering a broker before every replica has left it produces under-replicated partitions, and if the removed broker held the last in-sync copy of a partition, that partition goes offline until the broker returns or a replica is recovered. Check this before unregistration with the reassignment plan's verification step, then after it with the describe output for every topic, looking for partitions whose ISR is smaller than the replica list. Also check the under-replicated and offline partition metrics on your dashboards, since a removal can look clean in the CLI while a follower is still catching up.
A second failure appears when the remaining replica count drops below what producers require. A topic with min.insync.replicas set to 2 and a replication factor of 3 will reject acks=all writes once only one replica is in sync, so producers start failing with NotEnoughReplicas errors even though the cluster reports no offline partitions. Before you remove capacity, list the topics and their min.insync.replicas values, confirm the remaining brokers can satisfy each one, and confirm with the application owners that retries and error alerts are in place. Topics with a replication factor already at the minimum needed for their durability target are the first to be affected.
Disk removal has its own trap. When you cordon a log directory, the broker stops placing new replicas there, but replicas already on that directory keep their assignment until you move them. If the disk is removed or fails while those assignments remain, the partitions are reported as offline or stuck with no usable log directory, and the broker logs show repeated errors for the missing path. Verify that the directory holds no replicas in the broker's log directory description, then confirm that no partition in the plan still points to it. Ask your storage team to confirm the mount is quiet before the hardware is touched, because a disk that is still referenced can be the last place a replica lives.
Security and access controls during the move
Decommissioning changes who can move data and which brokers hold the authority to serve it. The cards below cover who runs the work, what must stay locked down on the remaining brokers, how the admin path is protected and how each change is recorded. Each one is a check to complete before the broker or disk leaves the cluster.
Limit who can run reassignments
Reassignment and broker unregistration change where every replica lives, so they should be run by a small named group, not by every operator with cluster access. Before the work starts, confirm which accounts hold admin rights and which hold only read or describe rights. Agree in writing who approves the plan and who executes it. Check that the executing identity is the one the change record names.
Keep ACLs intact on remaining brokers
In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, so removing a broker does not remove its ACLs, but a broken controller quorum or a restored metadata directory can. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that no topic or group loses its ACLs during the move, and verify that setting before and after the change.
Preserve admin credentials and super users
Replica moves and log directory changes rely on the admin client authenticating as a principal that is allowed to alter partitions and brokers. Keep the super user list unchanged on the surviving brokers and make sure the admin credentials used for the move are not tied to the broker being removed. Store them in your secrets system and rotate them after the decommission if they were shared for the work.
Secure the admin path with TLS and SASL
Commands such as reassignment and unregistration travel over the same listeners as client traffic. Kafka supports SSL for encryption and SASL mechanisms including SCRAM-SHA-512 and GSSAPI for authentication, but security is optional, so an unsecured listener can accept admin commands. Point admin tools only at secured listeners and confirm that the --command-config file references the expected truststore and SASL mechanism.
Record every change with an audit trail
Each reassignment plan, throttle change, log directory move and broker unregistration should leave a record that names the operator, the approver, the timestamp and the exact command or JSON file used. Store the plan files and the output of the verify step with the change ticket. Confirm that your platform's audit logging captures authorizer denials, so a failed attempt is visible as well as a successful one.
Questions teams ask before decommissioning Kafka brokers
How much effort does decommissioning a Kafka broker usually take?
Effort depends mainly on how many partition replicas the broker holds and how much data must move off it before it can be removed. A broker with few replicas and small logs is a short job, while a broker that leads many partitions on large disks needs a longer reassignment window, more verification and more rollback planning. Most of the work is preparation: inventorying replicas, confirming the remaining brokers have enough disk and network headroom, and writing a plan that covers every partition. Ask any provider to break its quote into inventory, planning, execution, verification and handover so you can see which parts scale with your cluster.
What does decommissioning cost, and who pays for what?
Cost has two parts: the people doing the work and the capacity you keep paying for while data moves. Internal staff time is the main cost if your team runs the commands itself, and it is usually one-time for a single removal. A partner's fee is based on the written scope, so a quote should list the broker count, log directories, environments, change windows and whether post-change monitoring is included. Recurring costs come from the extra disk and network capacity the remaining brokers need during and after the move. DeployOpen does not publish prices; a quote follows from the agreed scope.
Can our own team decommission a Kafka broker without outside help?
Yes, if your team already runs the cluster, understands partition reassignment and has a tested rollback path. The Apache Kafka tooling covers the steps, and the project documentation describes the operations involved. What usually makes internal work harder is not the commands but the gaps around them: no current replica inventory, no agreed change window, unclear ownership of consumers that depend on the broker, and no one assigned to verify the result. If any of those are missing, an outside review of the plan before execution can be cheaper than an incident during the move. Check your team's experience with the target Kafka version first, since commands and defaults change between releases.
How long does the move take, and what changes the duration?
Duration is driven by the volume of data to replicate, the replication throttle you set and the write rate into the affected topics. More data and a tighter throttle mean a longer move, while a higher throttle can slow producers and consumers if the network or disks are already busy. Reassignment must finish for every replica before the broker is safe to unregister, so one stuck partition extends the whole job. Before committing to a window, ask your team to measure the bytes on the broker, the spare throughput across the remaining brokers and the throttle values they expect to use. No single duration applies to every cluster.
What do we need to provide before the work starts?
Plan to provide four things: a current inventory of every topic partition and replica on the broker or log directory, agreed change windows with the owners of producers and consumers, enough spare capacity on the remaining brokers to absorb the replicas, and access to a test environment or staging cluster where the plan can be rehearsed. You also need named owners for approvals and for the go or no-go decision. The inventory should come from the cluster itself rather than from memory, and it should be refreshed just before the change, because partitions move as the cluster runs.
Can we reverse a broker removal after it has been unregistered?
Reversal is possible while the data still exists on the other brokers, but it is not free. Before unregistering, the replicas should already be reassigned and verified, so a rollback usually means reassigning partitions back to the original brokers, which repeats the replication work and needs the same throttling and verification. Once a broker's storage has been wiped or reused, its old replicas cannot be recovered from it. For that reason the plan should keep the removed hardware or disk intact until the cluster has run stably for a period you agree on, and it should record each partition's original and target placement.
What does a fixed-scope decommissioning engagement cover?
A fixed-scope engagement covers only the work written into the agreement. For a broker or log directory removal, that usually means the replica inventory, the reassignment plan for every partition, a rehearsal in a non-production environment if one is available, execution support during the agreed window, verification that no partitions remain on the removed resource, and a handover document. It does not cover unrelated tuning, consumer redesign or ongoing monitoring unless those are added to the scope. Ongoing support and upgrades are available as retainers. Ask for the deliverables and acceptance checks in writing before the work starts.
How DeployOpen scopes a Kafka removal
Scoping a broker or log directory removal starts with a written agreement on what will be moved, in what order, and under which change windows. No work begins until the scope, deliverables and timeline are agreed in writing, and most engagements are fixed-scope, so the questionnaire is the input that determines the quote. For this topic, the questionnaire needs to capture the cluster topology: the Kafka version in use, whether the cluster runs in KRaft mode, how many brokers and controllers exist, which broker or disk is leaving, and whether the removal is a whole node or a single log directory. It also needs the replica count and minimum in-sync replica settings for every affected topic, since those decide whether the cluster can absorb the loss while replicas move.
The questionnaire also captures consumer impact. We ask which consumer groups read from partitions led by the affected broker, how those groups commit offsets, and which teams must be told before a leadership change. Reassignment plans depend on this information, because throttled replication can lengthen moves and a slow consumer can turn a routine change into a lag incident. Change windows, rollback limits and the owner who approves each step are recorded alongside the technical data. Where the Kafka version matters for the procedure, we check the release line against the Apache Kafka downloads page and confirm the supported versions with your team before planning.
DeployOpen carries out the work inside your own cloud accounts, data centers or restricted networks, using your access paths, your change tooling and your logging. Your team keeps custody of credentials, and the scoping document states which operations DeployOpen performs, which your staff perform, and where evidence of each check is stored. We do not hold your cluster or its data outside your environment. Certifications and partner status are not part of this scope, and any compliance evidence we prepare is for your auditors to assess.
How DeployOpen delivers and supports Kafka operations work
DeployOpen plans and carries out broker and log directory removal on Kafka clusters you run in your own infrastructure. Each step is agreed in writing before work starts, so the reassignment plan, the checks and the handover are visible to your team before anything moves.
How we scope
- Inventory every broker, log directory and topic partition that the removal touches, with replica placement recorded before any change.
- Agree in writing which brokers or disks leave the cluster, the target brokers that will absorb replicas and the change window.
- Confirm with your team the throttle limits, rollback conditions and the owners who approve each stage of the removal.
How we build
- Cordon the target log directories, then generate and review a reassignment plan that covers every replica on the removed resources.
- Run the moves under replication throttling, watch under-replicated partition counts and ISR state, and stop if health degrades.
- Unregister the broker only after a check confirms no partition, leader or replica still references it, and record the results.
How we hand over
- Deliver the reassignment plans as executed, the verification output and a dated log of every command run during the removal.
- Document the cluster state before and after, including the new replica layout and the metadata checks used to confirm completion.
- Walk your operations team through the runbook so the next removal can be repeated without DeployOpen on site.
How we support
- Engagements are fixed-scope, so the written scope defines what is included and what a change request would add.
- Ongoing support, upgrades and continuous testing for the cluster are available as a retainer, scoped separately from the removal.
- Ask any partner, including DeployOpen, to confirm the Kafka version, the supported upgrade path and the access model before work begins.
