When expanding a Kafka cluster means moving partitions
Adding brokers to a running cluster does not, by itself, add capacity for existing data. New brokers only hold partitions that are assigned to them, so expansion is not complete until partitions are moved onto the new brokers and leadership is balanced across the cluster. Many teams that stop after the broker joins find the same hot disks, the same overloaded leaders and the same consumer lag they had before, now spread over one more machine.
Apache Kafka, released under the Apache License 2.0 with no paid edition from the project, provides the mechanics for this work. The kafka-reassign-partitions tool generates and executes reassignment plans, and the kafka-leader-election tool can move leadership to preferred replicas once copies are in sync. Kafka also supports replication throttles so that a move does not starve producers and consumers of network bandwidth. What the project does not provide is a judgment about which partitions to move, how fast, and when. Those decisions, and the consequences of getting them wrong, stay with your team unless a partner takes them on.
The team still owns the planning and the verification. Before generating a plan, confirm the current partition distribution, the disk usage per broker, the topic replication factors and the traffic peaks you must avoid. After execution, check that every reassigned replica is in sync, that the throttle has been removed, and that preferred leaders are where you expect. Confirm the exact flags and behavior for your Kafka release against the Apache Kafka 4.3 upgrade notes and your own change process, since tooling details can differ between versions. Plan a rollback path for moves that saturate the network, and record the plan, the throttle values and the final state so the change is auditable.
Reassignment options and when each fits
This table compares the four ways to run a partition move: generating a plan from topic and broker lists, writing reassignment JSON by hand, setting replication throttles, and running moves in batches. Read each row for the situation it suits and the trade-off it carries. The main takeaway is that generated plans are the fastest starting point, while hand-written plans and batching give the most control over load during the move.
| Option | When it fits | Main trade-off |
|---|---|---|
| Generate a plan from topic and broker lists | Moving a known set of topics onto new brokers when you accept the tool's placement and want a quick, reviewable starting point | The generated layout may not account for uneven partition sizes or traffic, so you must review the proposed placement before executing |
| Hand-written reassignment JSON | Placing specific replicas on specific brokers, racks or log directories, or correcting a layout the generator produced | Every replica assignment is manual, so mistakes are easy to make and must be checked against the current assignment before execution |
| Replication throttles during the move | Any move that copies significant data while producers and consumers are still active on the same brokers | Throttles slow data movement, so the move runs longer, and a throttle set too low can leave partitions under-replicated for longer than planned |
| Batched moves | Large expansions where moving everything at once would saturate network or disk on the brokers, or where you want to verify each step | Batches stretch the overall schedule and need a record of which partitions moved in each batch and which remain |
| Single unthrottled move | Small moves on idle or lightly loaded brokers during a planned window, with the team watching replication closely | Replication competes directly with client traffic, which can raise produce and fetch latency for the duration of the copy |
| Moving leadership after data moves | Restoring balanced leadership across the expanded cluster once replicas have caught up | Leadership changes cause brief client pauses for affected partitions, so schedule them with consumer and producer owners |
How to expand a Kafka cluster step by step
This sequence takes a running KRaft cluster from a confirmed healthy state to new brokers that carry their share of partitions and leaders, with a record of each change. Each step closes with a check, so a stalled move is caught before the next one starts.
Confirm version and KRaft health
Start by recording the Kafka version on every broker and controller, and confirm it is one of the supported releases on the Apache Kafka downloads page. Check that the metadata quorum has a healthy leader and that all voters are caught up, using the quorum status tool. Confirm there are no offline partitions and no under-replicated partitions before anything changes. Done means you have a saved snapshot of the version, quorum state and partition health that you can compare against later.
Add and register the new brokers
Provision the new broker nodes with the same Kafka version, the same listener and security configuration, and a node ID that is not already in use. Start them with the cluster ID and their role settings so they register with the controller quorum. Your platform or infrastructure team usually owns the servers, while the Kafka operator or engineer owns the configuration. Done means each new broker appears in the broker list and reports as registered, but it holds no partitions yet.
Capture the current partition assignments
Before generating any plan, export the current replica assignment for every topic you intend to move, or for all topics if you are balancing the whole cluster. Store the output in version control or another location with change history, because it is your rollback reference. Record the current leaders as well. Done means you have a file that lists each partition, its replicas and its leader, and someone other than the author has confirmed the topic list is complete.
Generate and review the reassignment plan
Build a proposed assignment that includes the new broker IDs, scoped to the topics you captured. The generation step produces a proposal but does not move anything, so treat the output as a draft. Review it for replication factor changes, rack or failure-domain placement, and any partition that would land on a broker that is being decommissioned or is still catching up. Done means a second reviewer has signed off on a plan file that matches the topics and target brokers agreed in the change record.
Set the replication throttle
Apply a replication throttle before execution so that data copying does not saturate broker network and disk and starve producers and consumers. Choose the throttle rate with the platform team based on measured headroom on your network, and apply it to the brokers and topics involved in the move. Confirm the throttle values on each broker after applying them, since a missing setting on one broker means that broker copies at full speed. Done means the throttle is visible in the broker configuration for every participating broker.
Execute the reassignment
Start the reassignment using the reviewed plan file. Replicas begin copying from current leaders to the new locations, and the old replicas are removed only after the new ones are in sync. Monitor client latency, broker disk and network use, and under-replicated partition counts while the move runs, and pause if producers begin to see errors or consumer lag climbs. Done means the execution tool reports the reassignment as in progress on the expected partitions, and you have a log of the start time and the plan used.
Verify completion with the verify step
Run the verify mode of the reassignment tool against the same plan file. It reports whether each partition has moved to its target replicas and whether the throttle is still applied to any of them. Also confirm that under-replicated partitions are back to zero and that the ISR matches the new replica list. Done means the verify output shows every partition at its target assignment with no pending work, and you have saved that output as evidence of the change.
Remove the throttle and rebalance leadership
Remove the replication throttle from every broker and topic it was applied to, and confirm the removal on each broker. Then trigger preferred leader election so leadership returns to the preferred replicas across the expanded cluster, which spreads client traffic onto the new brokers. Watch request latency and partition leader counts while this runs. Done means no throttle settings remain, each broker leads its expected share of partitions, and the final version, quorum and partition health match the snapshot you took at the start.
What goes wrong during expansion and how to prevent it
Partition moves copy real data while the cluster keeps serving traffic, so a mistake shows up as client latency or lost headroom rather than an error message. Each risk below has a check to run before the move starts, during it and after it finishes.
Moves saturate network and disks
A reassignment copies every replica of every moved partition across the network and writes it to the new broker's disks, competing with produce and fetch traffic. Before starting, measure the spare network and disk throughput on the source and destination brokers, and move a small batch first to see the real rate. Confirm that the disks on the new brokers have room for the incoming replicas plus normal retention growth.
Unthrottled replication hurts clients
Without a replication throttle, follower fetches during a move can consume enough bandwidth to raise produce and consume latency for every application on the cluster. Set the throttle explicitly for the move, and watch request latency and consumer lag while it runs. Agree on a rate limit with the teams that own latency-sensitive topics before the window opens.
Plans that overload new brokers
A plan generated only for even partition counts can still place most leaders or replicas on the newly added brokers, which then serve more traffic than the older brokers. Review the generated plan per broker for replica count, leader count and data size before executing it. Confirm that the new brokers have the same hardware class as the existing ones, or adjust the plan for the difference.
Incomplete verification of moves
A reassignment can report progress while some partitions are still catching up or have stalled on a slow follower. Check that every target replica is in sync for every moved partition, not just that the command finished, and confirm that no partitions remain in the old reassignment state. Record the partition list and final replica sets so the result can be audited later.
Leaders left unbalanced
Replica placement can be correct while leadership still sits on the old brokers, so the new capacity stays idle for reads and writes. After the data moves, run a preferred leader election for the affected partitions and check leader count per broker. Confirm that clients reconnected to the new leaders and that no partition now has a leader outside its preferred replica list.
Throttles left in place
A replication throttle that is not removed after the move keeps limiting follower traffic, so later failures recover slowly and the cluster looks healthy while it is still catching up. Remove the throttle configuration once verification passes, and make removal a named step in the runbook. Check the broker configuration afterwards to confirm that no throttle entries remain.
Questions teams ask before expanding a Kafka cluster
What drives the cost of adding brokers and moving partitions?
The cost is mostly people's time, plus the environment work needed to run the move safely. Internal staff spend time on capacity analysis, writing the reassignment plan, reviewing it with the owners of affected topics, running the move in a change window, and verifying the result afterward. Infrastructure costs for the new brokers, disks and network capacity are separate and usually recurring. A fixed-scope quote should list who does each task, which environments are covered (test, staging, production), how many topics and partitions are in scope, and whether leadership rebalancing and monitoring updates are included. Ask for each of these to be written out, so the quote can be compared with your own estimate.
How long does a partition move take?
Duration depends on how much data must be copied, how fast the network and disks are, how aggressively replication is throttled, and how many moves run at once. Moves that copy large partitions take longer, and a throttle that protects producers and consumers also slows the copy. Duration also depends on how many maintenance windows the change board allows and how many validation passes your team requires. No general figure applies, so ask for a duration estimate tied to your measured data volumes, throttle settings and test results from a non-production cluster, and confirm which of those inputs the estimate assumes.
What do we need to provide before the work starts?
The work needs access and data before it needs anything else. Typically this means:
- Administrative access to the brokers and controllers, plus the cluster's configuration and TLS and SASL settings
- Current topic and partition layouts, replication factors and the retention settings per topic
- Historical disk, network and CPU metrics for at least one representative period, including peak load
- Named owners for each important topic and consumer group, and the change windows your organization accepts
- A non-production cluster or a documented way to test the plan first
Confirm these with your platform team early, since missing capacity data is the most common reason a move is re-planned.
Can our internal team expand the cluster without a partner?
Yes, if the team has Kafka operations experience and a tested process. The core commands are documented by the Apache Kafka project, and the work is well understood. A partner adds value mainly when the team lacks recent experience with reassignment, when the cluster is business critical with tight windows, or when the team needs an independent review of the plan and the rollback path. Ask yourself who will own the plan after handover, who will be on call during the move, and whether you have a rehearsal environment. If those answers are clear, an internal team can usually do the work alone.
Should we upgrade Kafka before adding brokers?
Often yes, because the upgrade path can change how you expand. Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate to KRaft before moving to 4.x. The Kafka upgrade notes also recommend that KRaft clusters older than 3.3.x first move to 3.9.x. Rolling upgrades are supported from each release in the past year to the latest version. Check the supported releases on the Kafka downloads page and the upgrade notes for your current version before planning expansion. Running a new broker on a version your existing cluster does not support can block the move entirely.
Is Cruise Control a replacement for manual reassignment?
Cruise Control automates much of the work, but it does not remove the need for planning. It can generate balancing proposals from goals, detect anomalies and run self-healing actions, which reduces hands-on reassignment. You still need to define the goals, review proposals, set throttles and decide what it may do unattended. Running and tuning it is itself a project with its own operations burden. Manual reassignment with the built-in tools is simpler to reason about for a small, stable cluster. Decide after you know how often you expand and how many topics change. A separate page covers the Cruise Control approach in detail.
What are the main risks during a move, and how do we check it worked?
The main risks are replication traffic that starves producers and consumers, followers that fall out of sync, and a plan that leaves some replicas on the wrong brokers. Limit risk with a replication throttle, moves in small batches, and monitoring of under-replicated partitions, request latency and consumer lag during each batch. Verify completion by confirming that every reassignment has finished, that replicas sit on the intended brokers, that the in-sync replica sets are complete, and that leadership is balanced afterward. Record the before and after state so the change can be audited and rolled back if needed.
Does DeployOpen certify that our expanded cluster is compliant?
No. DeployOpen plans, implements and documents the work and can prepare evidence such as change records, configuration baselines and verification results for your compliance assessments. Independent auditors and certification bodies issue audit reports, attestations and certifications. If your data flows fall under a framework, ask the auditor early which evidence they expect from a capacity change, such as approvals, test results and rollback records, and make sure the expansion process produces it.
Limits to confirm in your environment before you move partitions
Partition moves depend on facts that change between releases and between environments, so the cards below list what to confirm before a plan is written. Each item affects whether a reassignment finishes on schedule, whether it can be rolled back, and whether the cluster stays within its safe operating margin during the move.
Tool behavior on your release
Reassignment tooling, its flags and its output can differ between Kafka releases, so confirm the exact behavior of kafka-reassign-partitions.sh on the version you run rather than copying commands from older runbooks. Check how the tool reports progress, how it handles throttles and whether a cancelled move leaves replicas in a state you expect. Run the same steps on a non-production cluster first and record the output you will compare against during the real move.
Supported release status
Confirm that the release you plan to move partitions on is still a supported release on the Kafka downloads page before you schedule work. A move that spans a release change or a planned upgrade adds risk, so decide the order of operations explicitly. If your cluster runs an unsupported version, the fix may need to come before the expansion, not after it.
Java 17 on every broker
Since Kafka 4.0, brokers require Java 17, so any new broker added to the cluster must run that JVM before it joins. Check the runtime on every host in the plan, including hosts provisioned from older images. A broker on the wrong JVM can fail to start after it receives replicas, which leaves partitions under-replicated during the move.
Network capacity between brokers
Replication traffic during a move competes with producer and consumer traffic on the same links. Ask your network team for the sustained throughput between racks, sites and the new brokers, and compare it with the bytes each moved partition replicates. Plan the move in batches sized so that the spare capacity covers the replication load with margin, and confirm the figures with measurements on your own network rather than vendor specifications.
Throttles and your monitoring
Replication throttles limit how fast replicas copy, but they also change how long under-replicated and lag metrics stay elevated. Confirm which dashboards and alerts will fire during a throttled move, and whether alert thresholds need a temporary silence or a change window. Agree in advance who watches the metrics, what counts as a stall, and when to lower or lift the throttle.
Rollback and change approval
Decide before the move what rollback means for your cluster: cancelling a reassignment, restoring the original replica list or reversing a batch. Confirm your change process allows the window, the approvers and the communication to application owners. Record the original assignment so that every step can be reversed with evidence.
What a DeployOpen expansion engagement includes
An expansion engagement is fixed-scope: the written scope defines which brokers are being added, which topics and partitions will move, what throttling and change windows apply, and what evidence DeployOpen hands over at the end. Nothing starts until that scope is agreed in writing. The engagement is delivered in the client's own infrastructure, whether that is their cloud accounts, data center or air-gapped network, and DeployOpen does not take over the cluster or its credentials beyond what the scope requires.
The work begins with an inventory and plan review. DeployOpen collects the current broker count, partition distribution per broker and per log directory, replication factors, the min.insync.replicas values in use, disk headroom, and the KRaft controller layout, then compares them with the target state. The output is a reassignment plan for each batch of partitions, with throttle values, the order of batches, rollback steps and the checks that gate each batch. Ask for the plan to be reviewed by the owners of the most sensitive topics before execution, since a move that looks balanced on paper can still saturate a link during business hours.
Execution runs batch by batch in the agreed windows, with replication throttles applied and removed as specified in the scope. Verification evidence is recorded at each gate: reassignment status, under-replicated and offline partition counts, ISR state, and leadership distribution before and after preferred leader election. The handover package covers the plan and what was executed, the metrics snapshots, a runbook for repeating the process, and the open items. Effort and duration depend on cluster size, the number of partitions to move, the number of change windows available and how many topics need individual sign-off, so ask for a quote built from those inputs rather than a generic estimate.
How DeployOpen scopes, delivers and supports Kafka expansion
DeployOpen treats Rebalancing partitions and expanding a Kafka cluster as a planned change to a running system inside your own infrastructure. We agree the written scope first, then build and test the reassignment and throttling runbooks against your cluster before any production move, and hand over the procedures with your team.
How we scope
- Inventory brokers, topics, replication factors and current partition placement across the cluster before proposing any move
- Agree in writing which brokers are being added, which topics are in scope and the throttle limits that apply
- Confirm the maintenance window, rollback conditions and who approves each stage, with engagements fixed-scope from the start
How we build
- Generate reassignment plans from current placement, then review each plan for leader balance and rack spread before execution
- Test throttled moves on a non-production cluster that matches your topology, recording the observed replication impact
- Verify completion with partition, ISR and leader checks, then document the balancing steps and the verification commands used
How we hand over
- Deliver a written runbook covering plan generation, throttle removal, verification and rollback for your operators
- Walk your platform team through a live dry run so they can repeat the expansion without DeployOpen present
- Provide the monitoring checks, alert follow-ups and change records that your auditors or change board may request
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the expansion is complete
- Review the next capacity plan with your team when new brokers, topics or partition counts are proposed
- Confirm support terms, response expectations and any retainer scope in writing before work begins
