Why teams automate Kafka rebalancing with Cruise Control
If some brokers carry more partition replicas, leadership or disk than others, or if every broker addition and removal means someone hand-builds a reassignment plan, your cluster is paying a continuing operational cost. Kafka Cruise Control produces and executes balancing proposals for a Kafka cluster, so the work of deciding what to move and in what order can be generated from the cluster's current state rather than assembled by hand each time. It is a separate component that you run beside the cluster, not a feature built into the brokers, which means it has its own version, configuration, deployment and failure modes that your team must own.
Cruise Control does not decide what a balanced cluster looks like on your behalf. The platform owner still chooses the goals that a proposal must satisfy, such as how strictly replica distribution and disk usage should be balanced, and which of those goals are hard requirements. Those choices should be written down before the first proposal is generated, because a proposal is only as sound as the goals behind it. Confirm the Cruise Control release and the Strimzi release you intend to run against your Kafka version, since Cruise Control compatibility is tied to specific versions and the combination you choose determines which features and deployment options are available to you.
Before you enable any automated execution, plan how proposals will be reviewed, who may approve them, and how a move will be stopped if replication slows production traffic. Ask the team that operates your Kafka brokers to confirm the replication throttle and the maintenance windows they accept, and test the full cycle of proposal, review, execution and verification on a non-production cluster that matches your topology. Treat the first automated run as a supervised change, with the same rollback planning you would apply to a manual reassignment.
Which Kafka Cruise Control goals to enable first
Cruise Control evaluates a cluster against an ordered list of goals, and each goal you enable adds a constraint that every rebalance proposal must respect. The cards below cover the main goal families a platform team chooses between, with the protection each one gives and the cost it adds to the proposals it generates.
Replica distribution goals
These goals spread replicas of each topic across brokers so that no single broker carries a disproportionate share of data. They protect against one broker running out of disk or network capacity after a broker is added or a topic grows. The trade-off is data movement: a replica distribution goal can propose moving large volumes of data, so check how much replication traffic your network can absorb during a move before you enable it on a busy cluster.
Leader distribution goals
Leader distribution goals balance which broker serves reads and writes for each partition, separate from where the data lives. They protect against a few brokers handling most client traffic even when disk use looks even. The trade-off is that leadership changes are cheap to execute but can shift producer and consumer load abruptly, so confirm that your clients tolerate brief leader elections during business hours.
Disk capacity goals
Disk capacity goals keep each broker below a configured utilization threshold so that log segments do not fill a volume. They protect brokers from the disk-full failures described in the retention and disk troubleshooting material. The trade-off is that a tight threshold on a cluster with uneven topic sizes can produce many proposals that move data to free space, so set the threshold with your retention settings in view.
Network capacity goals
Network capacity goals limit inbound and outbound traffic per broker, which matters when replication and consumer fan-out saturate network interfaces. They protect latency-sensitive producers from a broker that is saturated by a neighbor's traffic. The trade-off is that these goals depend on accurate network metrics from your brokers, so verify that the metrics feed reports correctly before relying on the proposals they produce.
Goal list and order
The goal list and its order are configured per deployment in Cruise Control, and the names, defaults and precedence can change between versions. Confirm the exact goal names and behavior against the Cruise Control documentation for the release you run. Starting with a short list is easier to explain to application owners, who can then see which constraint caused a proposal to change.
How Cruise Control turns goals into proposals and execution
Cruise Control works in a fixed loop: it collects metrics from the cluster, builds a model of load per broker, disk and partition, evaluates the goals you have configured against that model, and produces a proposal of replica and leadership moves. The model is the centre of the design. Cruise Control does not look at live traffic in a single moment; it estimates what each broker is carrying from the sampled metrics over a window, then checks whether the current assignment violates a goal such as replica distribution, leader distribution, disk capacity or network capacity. Goals are ordered by priority, so a hard goal that fails blocks the proposal, while soft goals are optimized where possible. Before you act on any proposal, confirm which goals your build enables and how they are ordered, because the same cluster can yield different moves under different goal sets.
A proposal is a plan, not a change. You can review the moves, compare the expected load per broker before and after, and decide whether to execute. This review step is where most teams catch problems, such as a proposal that moves large partitions into a period of peak traffic or that relies on a broker you intend to retire. Execution is different. Replica moves copy full partition data across the network, which competes with producers and consumers, so execution depends on replication throttling to limit the bandwidth used by the move. Confirm the throttle values with your network team before the first run, and plan the window in which moves may happen. Expect that the proposal reflects the cluster at generation time, so a proposal that sat unused for a long period should be regenerated.
Every proposal is only as good as the metrics feeding it. If broker metrics are missing, sampled at the wrong interval, or distorted by a restart or a traffic spike, the model will steer moves toward a picture of the cluster that does not exist. Check that samples cover every broker and every disk, that the sample store is retained long enough to see weekly patterns, and that partition sizes are measured rather than assumed. Treat a proposal that looks surprising as a signal to check the model before the cluster. Scope of this page is the planning loop and execution behavior. The Kafka-level mechanics of generating and running reassignments, including how throttles are applied to the underlying reassignment, are covered on the sibling rebalancing page.
Cruise Control options and when each fits
This table compares the main Cruise Control decisions, from how proposals are approved to how aggressively moves are throttled. Read each row as a starting position: the right setting depends on how sensitive your workloads are to replication traffic and how much operator review your team can sustain. Exact configuration keys and defaults must be confirmed in the Cruise Control and Strimzi documentation for the version you run.
| Option area | What it controls | When it fits | Trade-off |
|---|---|---|---|
| Proposal review versus automatic execution | Whether a generated rebalance plan is applied only after a person approves it, or applied by Cruise Control as soon as it is produced | Review fits first deployments, regulated change control and clusters where partition moves affect latency-sensitive topics | Review adds a delay before balancing happens, while automatic execution moves replicas without a human checking the plan |
| Goal set size | How many balancing goals Cruise Control optimizes for, such as replica distribution, leader distribution, disk and network capacity | A small goal set fits clusters that only need basic spread; a larger set fits clusters with uneven disk or network use | More goals produce more moves and longer proposal computation, and conflicting goals can make some proposals hard to satisfy |
| Self-healing per anomaly type | Whether Cruise Control acts on its own when it detects a specific anomaly, such as a broker failure, a disk failure or a goal violation | Enable for anomaly types you have tested in a non-production cluster; keep others on notify-only while you learn their behavior | Automatic repair reduces manual toil but can move large amounts of data during a bad moment, and a wrong detection can trigger unwanted moves |
| Throttle value during moves | The replication bandwidth limit applied while partitions move between brokers | Lower throttles fit clusters close to network or disk limits or serving strict latency targets; higher throttles fit maintenance windows with spare capacity | Low throttles lengthen the time to finish a rebalance and leave the cluster unbalanced for longer, while high throttles can starve producers and consumers |
| Dry-run or proposal-only mode first | Whether you generate and inspect proposals without executing any moves | Use it first on every new cluster, after changing goals, and before enabling any automatic execution | Dry runs cost little but do not reveal runtime effects such as throttle impact, so you still need a supervised first execution |
| Anomaly detection interval and notifier | How often Cruise Control checks for anomalies and where it reports them | Short intervals fit clusters where fast detection matters; longer intervals fit clusters with noisy metrics and teams who prefer fewer alerts | Frequent checks raise sensitivity and alert volume, while infrequent checks delay response to real failures |
| Scope of excluded topics or brokers | Which topics or brokers Cruise Control must leave alone during a rebalance | Exclude internal or highly sensitive topics, and brokers under maintenance or being decommissioned | Exclusions reduce the balance Cruise Control can achieve, and forgotten exclusions leave skew that operators may not notice |
Self-healing by situation: broker failure, disk failure and goal violation
When a broker stops responding for longer than the configured detection window, Cruise Control raises a broker failure anomaly that lists the missing broker IDs. Kafka itself handles the immediate failover by electing new leaders from in-sync replicas. Cruise Control's role is to restore replica placement afterwards, by moving the replicas that lived on the failed broker to live brokers that still satisfy the active goals.
- What the detector reports: the broker IDs that are missing and when the absence was first seen. Confirm the detection delay and the exact fields in the anomaly state of your version.
- What self-healing proposes: a reassignment plan that moves the failed broker's replicas onto remaining brokers while respecting rack and capacity goals.
- Before you enable it: confirm the surviving brokers have enough free disk and network headroom to absorb the replicas, and that rack-aware goals can still be met with the broker gone.
- Transient outages: a broker that restarts quickly may not need any moves. Decide how long an outage must last before automatic action is acceptable, and set
self.healing.broker.failure.enabledonly after that decision (confirm the key name for your version). - After the event: check under-replicated and offline partition counts, confirm the failed broker holds no replicas, and verify it rejoins cleanly if it returns.
Before going live, confirm that broker failure anomalies reach the on-call rotation and that someone reviews every proposal that ran.
Disk problems are a different signal from broker problems. On brokers that use multiple log directories (JBOD), a failed directory can take partitions offline while the broker process stays up. Cruise Control can report this as a disk failure anomaly and, where supported and enabled, move the affected replicas to healthy directories. Confirm disk failure handling is available for your Kafka and Cruise Control versions and storage layout before relying on it.
- What the detector reports: the broker and the failed log directory, which tells you the problem is local to one disk rather than to the whole broker.
- What self-healing proposes: moves of the replicas on the failed directory to other directories, which may be on the same broker or elsewhere, depending on the goals.
- Before the failure: check that each broker has spare capacity across its remaining directories, so one disk loss does not overload the others.
- Disk filling is not disk failure: a disk that is merely full appears as a capacity goal violation, which is handled in the goal tab. Keep the two alerts separate so the response matches the cause.
- After the event: confirm the failed directory is out of the placement of every partition, check for offline partitions and log errors on the broker, and plan the physical replacement with your storage team.
Before enabling automatic moves, agree who replaces the disk, because the self-healing plan only rebalances data and does not repair hardware.
Goal violations appear after load shifts, such as a new high-volume topic, uneven leadership, or traffic growth on a few partitions. Cruise Control evaluates its configured goals on a schedule and reports which ones are no longer met. This is the situation where self-healing moves the most data, and those moves compete with production traffic for disk and network bandwidth.
- What the detector reports: the goals that are violated and the state of the cluster that caused it, so you can see whether the cause is leadership skew, disk usage or replica distribution.
- What self-healing proposes: a rebalance plan that changes replica and leader placement until the violated goals pass, executed in the same way as a manual proposal.
- Before enabling it: review the proposal size on a non-production cluster or in dry-run mode, confirm replication throttling is configured, and confirm which goals are hard requirements and which are soft preferences.
- Alert first: many teams start with alerts only for goal violations, then allow self-healing after they have reviewed several proposals and understand their effect on consumer lag.
- After the event: verify the goals pass in a new state check, watch consumer lag and produce latency during the move, and record which partitions moved.
Before allowing automatic rebalancing, decide which goal violations are acceptable to act on without review, because this is the anomaly type most likely to change partition placement during busy hours.
Setting up Cruise Control through Strimzi on Kubernetes
This sequence takes a Kafka cluster from version and mode checks through a verified Cruise Control deployment, a reviewed dry-run proposal and a staged set of enabled goals and self-healing types. Each step closes with a check you can record before moving on.
Confirm Kafka version and KRaft mode
Your platform team checks which Kafka release the cluster runs and confirms it operates in KRaft mode with no ZooKeeper dependency. The Kafka downloads page lists the supported releases, and the upgrade notes state that Kafka 4.0 supports only KRaft and that ZooKeeper-based clusters must migrate before upgrading to 4.x. Clusters older than 3.3.x are advised to move to 3.9.x first. Confirm the Java runtime requirement for brokers and tools in the same notes. Done means you have a written record of the running version, the controller mode and the target release you will test Cruise Control against.
Match Strimzi and Cruise Control versions
Before installing anything, confirm with the Strimzi documentation which Strimzi operator release supports which Kafka and Cruise Control versions. The bundled Cruise Control image is tied to the operator release, so do not pick these versions independently. Record the operator version, the Kafka version it manages and the Cruise Control version it ships in your change record. Done means a compatibility table that your reviewers have approved and that matches the Kafka version confirmed in the previous step.
Enable metrics reporting on brokers
Configure each Kafka broker to expose metrics in the form Cruise Control reads, using the metrics reporter setup and the metrics configuration that your Strimzi version documents for Kafka resources. Cruise Control builds its load model from these metrics, so without them it has nothing to balance. Roll the change through the brokers using the normal Strimzi reconciliation so that no more than one broker restarts at a time. Done means each broker reports the expected metric series to your metrics store and the Kafka custom resource reconciles without errors.
Deploy Cruise Control with resources
Declare Cruise Control in the Kafka custom resource with its configuration, including the capacity file that gives each broker's disk, CPU and network limits, and set CPU and memory requests and limits for the Cruise Control pod. Keep the capacity values sourced from your hardware inventory, not from defaults. Apply the change through your GitOps pipeline and watch the pod start. Done means the Cruise Control pod is running and ready, and its logs show it connected to the Kafka cluster and the metrics source without authentication errors.
Verify the load model covers all brokers
Query the Cruise Control state endpoint, for example through a port-forward and the monitor state view, and check that the model reports a valid load snapshot covering every broker in the cluster. Count the brokers in the output and compare the number with the broker list from Kubernetes. If a broker is missing or the model stays incomplete, check the metrics reporter on that broker before going further. Done means the model lists all expected brokers with valid metrics and is marked as having enough data to generate proposals.
Generate and review a dry-run proposal
Request a rebalance proposal in dry-run mode so that Cruise Control computes moves without executing them. Review the output for the number of partition replica moves, leadership changes and the brokers involved, and compare these against your understanding of current skew. Confirm that the proposal does not move partitions onto brokers you intend to keep idle and that the replication volume it implies fits your throttle settings. Done means you have saved a reviewed proposal file that lists expected moves, and your team has signed off on it in writing.
Enable goals one at a time
Turn on one goal or one self-healing type in the Cruise Control configuration, apply it, and generate a new dry-run proposal to confirm the effect. Add the next goal only after the previous one behaves as the proposal predicted in a staging cluster and then in production. Keep self-healing disabled until your team has reviewed several anomaly reports and trusts the detection thresholds. Done means each enabled goal produced a proposal matching the expected moves, and you have recorded which goal and anomaly type are active, with the date and the approver for each change.
Security and operational considerations for Cruise Control
Cruise Control acts on live cluster state, so it needs the same access discipline as the brokers it moves partitions across. The cards below cover the Kafka-side controls it depends on and the operational controls your team must add around its REST API, proposals and settings.
Authenticating Cruise Control to Kafka
Cruise Control connects to brokers as a Kafka client, so it needs either SSL or SASL credentials configured for its connection. Kafka's security overview lists GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as the SASL options, and TLS for encryption in transit. Confirm which mechanism your brokers expose, store the client secret in your secrets tooling rather than in plain configuration, and rotate it on a schedule you can test.
Service account ACLs
The account Cruise Control uses needs enough permission to read cluster metadata and to alter partition assignments, and nothing beyond that. In KRaft clusters the StandardAuthorizer stores these ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so check that setting explicitly and confirm the account's grants with a dry run before enabling self-healing.
Access to the REST API
Cruise Control exposes an HTTP API that can request proposals and trigger execution, so anyone who reaches it can move partitions. Restrict the endpoint to the operators and automation that need it, authenticate every caller, and separate read-only status queries from calls that change the cluster. Decide in writing who may approve a proposal before execution, and record that approver in the request.
Network exposure between namespace and brokers
Brokers, Cruise Control and its metrics reporter need network paths to each other, but those paths should not be open to the wider cluster network. Limit traffic to the broker listeners and ports Cruise Control uses, and keep the API port off public routes. In Kubernetes, express these rules as network policies and verify them from outside the namespace, since a permissive default will not show up as an error.
Audit trail for executed proposals
Every executed proposal moves replicas and changes load across brokers, so you need a record of who requested it, what the proposal contained, who approved it and what the cluster looked like before and after. Keep Cruise Control logs and Kafka audit output in a store your operators cannot quietly edit, and tie each execution to a change ticket so a later latency spike can be traced to a specific move.
Change control for goals and self-healing
Goal lists, thresholds and self-healing switches decide what Cruise Control does without a person in the loop. Treat them as configuration under version control, with peer review and a tested rollback. Enable self-healing for one anomaly type at a time, and confirm the expected behavior in a non-production cluster before widening it to production.
What teams ask before automating Kafka rebalancing
What does automating Kafka rebalancing with Cruise Control cost, and who pays for what?
The cost is mostly effort, not licenses, because Cruise Control is open source and the Kafka project itself has no paid edition. Your team pays for the people who build and run it, the test environments, and any partner time you buy. The main cost drivers are the size of the cluster, how many optimization goals you define, how many environments need the same setup, and how familiar your engineers already are with Kubernetes and Strimzi. Ask any partner for a written scope that lists each deliverable, the environments covered, and what happens after handover. A quote should separate the one-time build from any recurring support or review of proposals.
How long does a first Cruise Control rollout take, and what makes it longer?
No fixed duration applies to every cluster, so treat any number you see as a starting point to verify. Timing depends on how many brokers and topics you run, how many goals you need, whether you already have reliable metrics in place, and how many approval steps your change process requires. Teams that are new to Kubernetes operators or Strimzi usually need more time for the first rollout than teams that already run them. Ask for a plan with phases such as read-only proposals, supervised execution in a test environment, and then production, each with an exit condition.
Can our own engineers run Cruise Control without outside help?
Yes, many teams do, provided they have people who understand Kafka partitions, replication and broker resources, and who can read a rebalance proposal before approving it. The harder part is the operating routine: reviewing proposals, watching for failed or stalled moves, and tuning goals after the cluster changes. If your team has not run Kafka in production before, an outside engagement is more often used for the first build and handover than for daily operation. Whichever path you choose, confirm in writing who owns each runbook step after go-live.
What access, data and skills does the work need from our side?
You need to provide the following before work starts:
- Administrative access to the Kafka cluster and, for Strimzi, to the Kubernetes namespaces where it runs
- Working Prometheus or equivalent metrics for broker load, disk and partition state, so proposals rest on measured data
- A test cluster or staging environment that resembles production closely enough to validate goals
- A change window for the first production execution and a named owner who can approve moves
- Engineers with Kubernetes, GitOps and Kafka operations experience to review changes and take over operation
Confirm the exact access model with your security team before the scope is signed.
When should self-healing stay off in Cruise Control?
Keep self-healing off until your team has reviewed several rounds of proposals and trusts the goals and thresholds. Self-healing lets Cruise Control act on detected anomalies without a person approving each step, so a misconfigured goal or a noisy metric can trigger repeated data movement during business hours. It is also reasonable to keep it off for clusters with uneven hardware, unstable networks, or topics with strict throughput needs until those conditions are understood. Decide the criteria in writing, such as a period of clean proposals in staging, and record who may enable it in production.
How does Cruise Control fit with Kafka upgrades and version planning?
Plan Cruise Control alongside your Kafka version, not after it. Kafka's upgrade notes and time-based release plan define which upgrade paths are supported, and any Cruise Control or Strimzi version you run must support the Kafka release you move to. Test the Cruise Control configuration and goals against the target Kafka version in staging before upgrading production, and check whether metric names or APIs changed. Ask any partner how the upgrade path for the rebalancer is included in the scope and whether it is covered by a support arrangement.
Limits to confirm in your environment before you rely on Cruise Control
Cruise Control acts on your cluster without a human in the loop, so its assumptions matter as much as its features. The cards below list the points to confirm with vendor documentation or a test cluster before you enable automated rebalancing or self-healing. Each one maps to a place where a default, a version pair or a workload trait can quietly change the outcome.
Which Kafka and Cruise Control pairs are supported
Start by asking which Cruise Control release is certified against which Kafka release, because the two projects do not move in lockstep. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported Kafka releases, so check that your Cruise Control build and any Strimzi operator version target one of them. Confirm the pair on a test cluster before the production upgrade, and record it in your change plan. Ask your operator or distribution vendor for the tested matrix in writing.
Whether the Cruise Control service runs on your Java runtime
The Kafka upgrade notes require Java 17 for brokers, Kafka Connect and tools, but they do not settle the runtime for Cruise Control itself. Check the Cruise Control release notes and build files for its required JDK, then confirm the container image or host package you deploy uses that version. Do not assume a broker image carries the right runtime for a separate Cruise Control deployment. Test startup and a dry-run proposal after any JDK change.
How it behaves on KRaft clusters
Cruise Control was built around the metrics and metadata model of ZooKeeper-era clusters, and your KRaft setup may differ in how it discovers controllers and reads metadata. Confirm that your Cruise Control release reads cluster state correctly on KRaft, including broker counts, leader placement and the controller quorum. Test a proposal and an execution against a staging cluster with the same controller layout as production. Since Kafka 4.0 supports only KRaft, the answer determines whether you can run it at all on newer releases.
Topics with custom placement or cross-site replication
Confirm how the goals treat topics that rely on rack awareness, explicit replica assignments or MirrorMaker 2 replication to another site. A balancing proposal that ignores placement constraints can move replicas out of the failure domains you designed for. Check whether mirrored topics and their consumer groups are excluded or included in optimization, and whether proposals respect replica placement requirements. Exclude topics you manage by hand until you have verified the behavior in a test environment.
Metrics coverage gaps
Self-healing decisions depend on the metrics Cruise Control collects, and some broker or topic signals may be missing or delayed in your setup. Compare the metrics reporter configuration on each broker with the signals your alerts already use, such as under-replicated partitions and request latency. Confirm that the metric sampling interval and the window length suit your traffic patterns. Where coverage is thin, keep human review on proposals until the gap is closed.
Whether the chosen goals fit your workload
Goals express what the balancer optimizes for, and the default set may not match your priorities for disk, network, or leadership distribution. Ask which hard goals block a proposal and which soft goals only shape it, because a hard goal you did not intend can stall a rebalance. Test the goal list against a recent traffic snapshot and review the proposed moves for partition counts and data volume. Document the chosen goals and their thresholds so they can be reviewed after each Kafka upgrade.
How DeployOpen scopes and delivers Cruise Control work
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Apache Kafka clusters. For Automating Kafka rebalancing with Cruise Control, we start from your cluster's actual load, failure history and access model, then agree a written, fixed scope before any change is made to a running cluster.
How we scope
- We review your brokers, topics, partition layout and current load pattern before defining the Cruise Control scope.
- The written scope names which goals, anomaly detectors and self-healing behaviors are in and which stay manual.
- We agree how Cruise Control reaches the cluster, including Strimzi or other deployment method and the access controls it needs.
How we build
- We deploy Cruise Control in a non-production environment that mirrors your broker count and topic configuration.
- We test rebalance proposals in dry-run mode and review every proposed partition move against your throttle and retention settings.
- We enable anomaly detection and self-healing in stages, recording the behavior observed at each stage before widening it.
How we hand over
- You receive the deployed configuration, goal definitions, runbooks for proposals and rollback, and documented self-healing limits.
- We walk your operations team through reading proposals, approving or rejecting them, and checking the result afterwards.
- Deliverables are listed in the written scope, so you can verify each one against the agreed list at handover.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, and can cover Cruise Control version changes.
- Retainer scope, including how often configurations are reviewed after Kafka upgrades, is agreed in writing before work begins.
- Quotes follow from the written scope. Ask us to itemize one-time build work separately from recurring support.
