Why a KRaft cluster on a static quorum needs a planned move to dynamic
If your KRaft cluster fixes its controller set in controller.quorum.voters, you can only change that set by restarting the cluster with new configuration, and that is the situation this page addresses. Moving to a dynamic controller quorum lets you add and remove controllers while the cluster keeps running, but the switch itself is a deliberate upgrade of the cluster's metadata layer and has to be planned as one. Before you change anything, confirm the current state: a cluster is on a static quorum when the kraft.version feature level is 0 or absent, and on a dynamic quorum when it is 1 or above. You can read that level with kafka-features.sh describe on a running cluster.
The timing depends on the release you are running. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and the time-based release plan supports rolling upgrades from each of the last three releases to the latest version. If your cluster is on an older 3.x line, the upgrade notes recommend first moving KRaft clusters older than 3.3.x to 3.9.x before going to 4.x, and ZooKeeper-based clusters must migrate to KRaft before any 4.x upgrade. Confirm your current version and its supported upgrade path with the vendor documentation before you schedule the work, because the order of the controller change relative to the broker upgrade matters.
Treat the quorum change as a separate maintenance item with its own rollback thinking. Once the feature level is raised, the cluster no longer runs the static membership model you started with, and some steps cannot be undone by simply restarting old binaries. Decide in advance which controllers will be added or retired, confirm that each new controller can catch up before it is counted, and agree who signs off on the metadata version change. The Apache Kafka 4.3 KRaft documentation describes the add and remove controller procedure that the rest of this page follows.
What changes when the quorum becomes dynamic
This table compares a KRaft cluster on a static quorum with one on a dynamic quorum, row by row, from the kraft.version feature level through to the controller commands and the operational steps they change. Read each row as a question to answer about your cluster before you start, and note that the main takeaway is that controller membership stops being a configuration file edit and becomes a metadata operation.
| Area | Static quorum | Dynamic quorum |
|---|---|---|
| kraft.version feature level | 0 or absent | 1 or above |
| How the controller set is defined | Set by controller.quorum.voters in each node's configuration | Tracked in the cluster metadata, with controller.quorum.voters no longer the source of truth |
| Adding a controller | Edit controller.quorum.voters on every node and restart them in a planned order | Start the new controller, let it catch up, then run kafka-metadata-quorum.sh add-controller |
| Removing a controller | Remove it from controller.quorum.voters everywhere and restart the remaining nodes | Run kafka-metadata-quorum.sh remove-controller before shutting the controller down |
| Change risk during membership edits | Configuration drift between nodes can leave the quorum inconsistent until every node agrees | The change is recorded in metadata, so the controller set is managed through one command path |
| How you verify the current state | Compare configuration files across controller nodes | Check kraft.version with kafka-features.sh describe, then review quorum state with kafka-metadata-quorum.sh |
| Action required before the switch | Confirm all controllers share identical controller.quorum.voters values | Confirm every controller and broker runs a release that supports the target feature level (confirm the exact requirement in the upgrade notes) |
| Action required after the switch | Keep configuration files in sync with any future membership change | Update runbooks and automation so membership changes go through the metadata commands, not file edits |
How to confirm whether your cluster still uses a static quorum
The first step is to establish which quorum model your cluster runs today, because every later step depends on it. A KRaft cluster uses a static quorum when the kraft.version feature level is 0 or absent, and a dynamic quorum when the level is 1 or above. Run kafka-features.sh with the describe subcommand against any broker or controller endpoint, and look for the kraft.version line in the output. If the line reports level 0, or the feature does not appear at all, the controller set is still defined by controller.quorum.voters in each node's configuration file and cannot be changed with kafka-metadata-quorum.sh.
Before you change anything, record what the cluster looks like so you can compare it after the upgrade. Capture the output of the describe command, the kraft.version and metadata.version levels, the Kafka release every node runs, and the value of controller.quorum.voters on each controller. Also note the node IDs, the listener addresses and the directory log paths for each controller, since a dynamic quorum identifies controllers by ID and directory ID. Store this record with your change ticket. A written baseline makes it possible to prove, later, which controllers were voters before the change and which were added after it.
The exact output layout of kafka-features.sh describe can differ between Kafka releases, so confirm the format against the release your cluster runs before you build any parsing or runbook step around it. Check the KRaft documentation for Kafka 4.3 for the version your nodes match, and run the command on a non-production copy if one exists. If the output does not show a kraft.version line for a cluster you expected to be current, treat that as a finding to resolve with the vendor or your own platform team, not as proof that the cluster is static.
What to prepare before touching controllers
Changing the controller quorum touches the metadata that every broker depends on, so the preparation is as much about records as about commands. The six items below each have an owner and an artifact you should be able to show before the first change.
Verified metadata and broker backup
Take a backup of the metadata log and snapshots from the controllers, and keep broker data backups for the topics you cannot rebuild. The platform or database owner runs the backup, but the evidence that matters is a restore test into an isolated environment, with the restored cluster reporting the same metadata state. Record the backup time, the source controller, and the checksums. Without a tested restore, a failed quorum change has no clean way back.
Controller host and listener inventory
List every controller and combined broker-controller host with its node ID, controller listener address, port, and certificate identity. Compare that list with controller.quorum.voters on each node so no node disagrees about the current membership. The infrastructure team owns this inventory, and the evidence is a dated export from each node's configuration, stored with the change ticket.
Staging cluster that mirrors production
Build a staging cluster with the same controller count, listener layout, TLS and SASL settings, and Kafka version as production, then run the full sequence there first. Staging with three nodes on one host does not reproduce network or disk behavior, so the team should confirm which differences remain. The platform team owns the environment, and the evidence is a written record of each step run and the result.
Client and tooling compatibility checks
Inventory the scripts, automation, monitoring checks and admin clients that call kafka-metadata-quorum.sh or kafka-features.sh, and confirm they run against the target version. Also confirm that any tool reading controller endpoints from static configuration will be updated when membership changes. The owners are the teams that maintain those tools, and the evidence is a list of tested commands with their outputs.
Change window and rollback decision
Agree a change window with the service owners, the on-call lead and the people who can approve a stop. Write down who decides to halt, what condition triggers a halt, and who executes the rollback steps from the backup. Keep the approval, the named contacts and the stop criteria in the change record. Confirm the window length with your own change board, since no timing is set by the project.
Operator access and separation of duties
Confirm that the operators performing the change have shell access to every controller, the credentials needed for TLS and SASL, and read access to monitoring during the window. Where the change requires a second person to review commands, assign that role before the window opens. Security owns the access grants, and the evidence is the access list with the date each grant was verified.
The upgrade sequence from static to dynamic quorum
This sequence takes a KRaft cluster from a static controller quorum defined by controller.quorum.voters to a dynamic quorum where controllers can be added and removed while the cluster stays online. Each step ends with a check, so you know whether to continue, pause, or roll back before the next change.
Confirm the running release and kraft.version
Record the Kafka version on every broker and controller, then run kafka-features.sh describe against the cluster to read the current kraft.version feature level. A level of 0, or no entry, means the cluster uses a static quorum. Save this output with your change record. If any node runs a release older than 3.3.x, plan a move to 3.9.x first, as the upgrade notes recommend. If the output does not match your inventory, stop and resolve the difference before changing anything.
Upgrade the cluster to a supported 4.x release
If the cluster is not already on a supported 4.x release, perform a rolling upgrade to the latest supported version. The time-based release plan supports rolling upgrades from each of the last three releases to the latest version. Upgrade controllers and brokers one node at a time, waiting for each node to rejoin and for under-replicated partitions to clear before moving on. Confirm client compatibility beforehand. Until you finalize in the next step, the release can still be rolled back to the previous binaries, so confirm the rollback limits in the upgrade notes for your starting version.
Finalize the kraft.version feature level
Raise kraft.version to 1 with the feature upgrade command in kafka-features.sh, which turns on dynamic quorum support in the metadata log. Finalizing is the point where downgrade becomes limited, so treat it as a planned change with a named owner and a backout plan. After the command completes, run kafka-features.sh describe again and confirm that kraft.version reports level 1 on the cluster. Do not add or remove controllers until this check passes.
Add new controllers after they catch up
Start each new controller with its node ID, storage directory and controller listener configured for the cluster, but do not yet give it a voting role in your plan. Run kafka-metadata-quorum.sh add-controller for the new node, and use describe with the replication option to confirm it has caught up to the leader's metadata log. Add controllers one at a time. If a new node does not catch up, leave it out of the quorum, fix the cause, and check again. The result is a larger voter set with every new member replicated.
Verify the quorum before removing controllers
Check the quorum state with kafka-metadata-quorum.sh describe --status and confirm that a leader is elected and that the expected voters appear in the voter set. Run a test that writes and reads a topic to confirm metadata operations succeed. This step is your last clean rollback point for the membership change. Keep the retired controllers running and healthy until the verification passes, because removing a voter before confirming the new set is the most likely way to lose quorum during the change.
Remove retired controllers before shutdown
For each retired controller, run kafka-metadata-quorum.sh remove-controller before stopping the process. Remove one controller at a time and repeat the status check after each removal to confirm a leader and the expected voter count. Only shut down a controller after its removal is confirmed. If you need to stop a node earlier, the dynamic quorum procedure is the supported route, so do not stop a voter first and clean up later. Record the final voter set in your change record.
How to add and remove controllers without downtime
Add controllers one at a time, and only after each new controller has caught up with the metadata log. Once kraft.version shows 1 or higher, the quorum is dynamic and membership changes are made with kafka-metadata-quorum.sh. Start the new controller with its own node ID, directory and listener, and let it replicate. Then run add-controller against the active quorum, using the bootstrap server and the command configuration file for your security settings. Do not run add-controller before the new node is ready, because the controller will be counted toward the quorum while it still lacks data.
Remove a controller with `remove-controller` before you stop its process. The sequence is the reverse of adding: confirm the controller you are retiring is not the only path to the leader, run remove-controller with its node ID and directory ID, and only then shut it down. Stopping a voter first leaves the quorum counting a member that no longer answers, which is the situation that can cost you majority. Keep the controller's directory ID from the describe output, since removal depends on it. Plan the number of controllers so that each step still leaves a majority of voters online, and confirm that number with your own sizing before you begin.
Verify every step with describe output, not with the command's exit status alone. Run kafka-metadata-quorum.sh describe --status to see the leader, the current voters and the highest log offset. Run describe --replication to confirm each voter's log end offset is close to the leader's before you proceed. After each change, repeat both commands and confirm the voter list matches your plan. Record the output at each step so that a failed change can be rolled back to a known membership. Confirm the exact flags and output fields against the KRaft documentation for the version you run, since the output format can change between releases.
Testing and rollback by environment
Staging is where the team proves the sequence works on a copy of production that matches its controller count, listener configuration and authorization setup. Record the current state with kafka-features and kafka-metadata-quorum first, then run the full sequence end to end, not only the individual commands. The KRaft documentation is the reference for each command's expected output.
- Starting state. Confirm
kraft.versionreads 0 or is absent on staging, and record the voter list fromcontroller.quorum.votersso you can compare results after each step. - Controller add and remove. Add a new controller only after it has caught up, then remove a retired controller before it shuts down. Time each step against the runbook.
- Client behavior. Run producers, consumers and admin tooling through every stage and watch for metadata errors during leader changes.
- Failure injection. Stop one controller during the membership change and confirm the cluster elects a leader and keeps serving traffic.
- Rollback drill. Practice the rollback steps you intend to use in production, and record any step you could not complete.
Record timings, errors and deviations from the runbook before production approval is requested.
The production window should change one controller at a time, with a go or no-go decision before each step. Agree the rollback decision point in advance, because the available options narrow once the cluster has been finalized. The goal is no client downtime, but the team should still plan for the case where a step behaves differently from staging.
- Pre-checks. Verify that all controllers and brokers are healthy, that configuration and certificate backups exist, and that the quorum has its full expected membership before the first step.
- One change at a time. Do not start a second membership change until the first has finished and the quorum reports healthy.
- Decision gates. Name the person who can halt the change and the criteria that trigger a halt, such as unexpected leader churn or rising client errors.
- Communication. Tell application owners when controller changes start and end, so they can correlate any client symptoms with the window.
- Change record. Capture the output of
kafka-features describeandkafka-metadata-quorumbefore and after each step.
Confirm that the window, decision owner and rollback point are signed off before the change begins.
Monitoring after the change confirms the quorum stays stable under normal load and during routine restarts. Some problems appear only when a controller restarts later, so the monitoring period should include at least one planned restart of each controller. Compare the results with the pre-change baseline rather than with general thresholds alone.
- Quorum state. Check
kafka-metadata-quorumoutput on a schedule, confirming one leader and the expected voter set. - Restart behavior. Restart each controller in a planned way and confirm it rejoins with the correct configuration.
- Configuration drift. Remove references to
controller.quorum.votersfrom automation and templates, so a rebuilt node does not rejoin with stale settings. - Election and metadata alerts. Alert on repeated leader elections or repeated fetch errors in controller logs.
- Client signals. Watch producer error rates and consumer lag against the pre-change baseline.
Keep the change record and monitoring baseline until your team has completed a full cycle of normal operations, including planned maintenance.
Rollback options depend on whether the cluster has been finalized and which features have been enabled. Before any membership change, the safest path is to stop the procedure and leave the cluster in its previous state. After the metadata version is raised, reversal is limited, and the team should not assume a step can be undone until the supported path is confirmed in the 4.3 upgrade notes.
- Before any membership change. Stop the procedure, keep the static voter configuration in place, and confirm
kraft.versionstill reads 0 or is absent. - Between controller additions. Remove a newly added controller only if the runbook covers that case, then confirm the quorum returns to full health.
- After `kraft.version` is raised. Check which downgrade paths the upgrade notes support for this feature level, and do not assume a downgrade is available.
- Binary rollback. Confirm the supported release and downgrade path before reverting any broker or controller binary.
- Irreversible steps. Treat any step that enables new features or changes the metadata format as one-way until your own testing shows otherwise.
Write down the last point at which rollback is possible and who approves it before the change window opens.
What makes a controller quorum change take longer
This table lists the factors that change how much work a move from a static to a dynamic controller quorum involves. Read each row as a direction: the factor adds effort or time as it grows, and the right column names what to check before you schedule the change. The main takeaway is that the controller count matters less than the tooling and configuration that still points at controller.quorum.voters.
| Driver | Why it changes effort | How to keep it manageable |
|---|---|---|
| Number of controllers and brokers | More controllers mean more add-controller and catch-up checks, and more nodes to restart with updated configuration. Broker count adds rollout time because each broker's configuration must be verified. | Plan the controller sequence in a written runbook and confirm each controller is caught up before moving to the next. |
| Prior version | Clusters on older KRaft releases may need an intermediate upgrade before the feature level can be raised. Confirm the release path in the Kafka upgrade notes before planning the quorum change. | Confirm your current release and the supported upgrade path first, then schedule the quorum change as a separate step. |
| Customizations and tooling that reference controller.quorum.voters | Scripts, health checks, Helm values, Ansible roles and monitoring rules that hard-code the voter list must be found and changed. Missed references often surface only after a controller is added or removed. | Inventory every file and dashboard that names controller IDs or endpoints, and test each change in a non-production cluster. |
| Automation and configuration management | Configuration drift across hosts makes it harder to confirm that every node uses the intended quorum settings. Automation can also re-render the old static voter list and undo changes. | Pause or update configuration management before the change, then reconcile the rendered files against the running cluster. |
| Network and TLS setup for controller listeners | Controller listeners must be reachable from every controller that joins, and any TLS configuration must cover the new nodes. Firewall rules, certificate SANs and DNS entries are common delays. | Verify connectivity and certificates on each new controller endpoint before running add-controller. |
| Size of the metadata log | A larger metadata log means new controllers take longer to catch up before they can be added. Check the replication state and lag before each step. | Check the metadata log status with the quorum tooling before each add or remove, and do not proceed while a new controller is still catching up. |
Questions teams ask before changing controllers
What does a controller quorum change cost, and what drives the price?
DeployOpen does not publish a price list, and a quote follows from the written scope agreed before work starts. Apache Kafka is licensed under Apache-2.0 with no paid edition from the project, so the cost sits in people, rehearsal environments and change support. The one-time work covers reviewing the current controller.quorum.voters setup, checking kraft.version, writing the change runbook, rehearsing in a non-production cluster, executing the controller changes in your window and verifying the result afterward. A recurring cost appears only if you add a retainer for upgrades or support. The main drivers are the number of clusters, the number of controllers to add and remove, the number of environments, your change approval requirements and the size of the client estate that needs checking. Ask for line items for planning, rehearsal, execution, verification and handover so you can compare them with your own team's time.
How long does moving to a dynamic quorum take?
The written scope should give the timeline, and it should be treated as something to confirm rather than assume. The hands-on steps for each controller are short, but elapsed time depends on how long each new controller takes to catch up with the metadata log. That depends on log size, disk speed and the network between nodes and sites. Other drivers include the number of controllers to change, whether you rehearse in staging first, your change calendar and approval cycle, and whether the Kafka version needs an upgrade before the quorum switch. Multi-site layouts add time because each controller placement needs its own network and latency checks. Ask the partner to break the timeline into phases with entry and exit checks, and ask which of those phases your change board must approve.
What must our team provide before the controller change starts?
The team needs to provide access and configuration details so the change is planned against your actual cluster. Expect to supply:
- Administrative access to the controller and broker hosts, and the Kafka version running on each.
- The current
server.propertiesfiles for every controller and broker, includingcontroller.quorum.votersand listener settings. - Output from
kafka-features.sh describeandkafka-metadata-quorum.sh describetaken while the cluster is healthy. - The Java runtime on each node, since brokers require Java 17 from Kafka 4.0 onward.
- A non-production cluster or rehearsal window, along with your change calendar and named approvers.
- Named owners for the client teams that must be warned about restarts or endpoint changes.
Missing any of these items usually moves work from planning into discovery during the change window, which is the costliest place to find problems.
Can our own platform team make this change without a partner?
Many teams can, provided they have someone who has operated KRaft in production and a tested rollback plan. The documented sequence is short: confirm the feature level with kafka-features.sh describe, add new controllers with kafka-metadata-quorum.sh add-controller after they have caught up, and remove controllers with remove-controller before shutting them down. The risks lie in the details, such as removing the wrong node, dropping below a safe number of voters, mismatched node IDs or configuration errors on a new controller. A partner typically adds a written runbook, a rehearsal in a non-production cluster, a second pair of hands during execution and written evidence for your change record. Decide based on your team's Kafka experience, how many people can be on call during the window and whether a failed step can wait until business hours.
Is a rolling upgrade from recent Kafka releases supported?
Yes. The Kafka time-based release plan says rolling upgrades are supported from each release in the past year, which is the last three releases, to the latest version. The supported releases listed on the downloads page are 4.3.1, 4.2.2 and 4.1.2, so a cluster on one of those can move to the current release by rolling upgrade. Confirm the exact path for your starting version in the upgrade notes. Clusters that are KRaft-based but older than 3.3.x are advised to move to 3.9.x first, and ZooKeeper-based clusters must migrate to KRaft before moving to 4.x. The move to a dynamic quorum is a separate step. Plan it after the cluster is on a stable, supported version and after you have checked the upgrade notes for that version.
What happens if adding a new controller fails partway through?
The existing controllers keep serving the quorum, so a failed add should not by itself stop production writes, but confirm this against the documentation for your version before you rely on it. Do not run remove-controller on existing controllers to clear the failure. Instead, check the quorum status with kafka-metadata-quorum.sh describe to see whether the new controller has caught up. Then review the new node's logs for causes such as network reachability, disk space, a wrong node ID, a listener or TLS misconfiguration or a mismatched directory. Fix the cause and retry add-controller only after the new node is ready. Remove a controller only when the quorum status shows the remaining voters are healthy. Ask the partner to document this decision path in the runbook before the change window opens. The KRaft operations documentation is the reference for the commands.
Will producers and consumers see downtime during the switch?
The goal is no client outage, but whether clients notice depends on the order of steps, your network and your client settings. Controller changes do not usually touch client traffic directly, yet a few conditions can cause visible impact. Removing too many voters can break the quorum, so the number of controllers in each step matters. Configuration changes that need a restart can move leadership and briefly disrupt partitions. Clients whose bootstrap lists point only to retired nodes will fail to reconnect, and DNS or load balancer changes can take longer to propagate than expected. Ask for a rehearsal that records producer and consumer behavior during each step, and compare the results against the client-side retry and timeout settings your applications use.
How DeployOpen scopes and delivers a quorum migration
DeployOpen treats a move from a static to a dynamic controller quorum as a fixed-scope change to a live metadata layer, planned in writing before any controller is touched. The work covers the feature check, the upgrade sequence, controller replacement and the runbook your operators will use afterwards.
How we scope
- We confirm the current kraft.version level and the controller.quorum.voters list on every controller before work starts.
- The written scope names each controller host, its storage path, its network path and the change window agreed with your team.
- We record rollback limits, the finalization decision and who approves each step, so the boundaries are clear before the first change.
How we build
- We rehearse the feature upgrade and controller additions on a non-production cluster that mirrors your topology and configuration.
- We add new controllers with kafka-metadata-quorum only after each one has caught up, and we check quorum status after every change.
- We test removal of each retiring controller before shutdown and confirm the broker and client connection strings no longer depend on the old voter list.
How we hand over
- You receive a written record of the versions, feature levels, controller IDs and configuration files as they stand at completion.
- The runbook covers add-controller, remove-controller, quorum status checks and the steps to take if a controller fails to catch up.
- Your operators walk through the procedure with us, and the runbook is kept in your own repository and change system.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers for clusters that need help after the migration.
- Retainer scope for controller changes, Kafka upgrades and periodic quorum health reviews is agreed in writing with your team.
- Questions about future 4.x upgrades that depend on the dynamic quorum are scoped as separate written work before they begin.
