Can you still undo a ZooKeeper to KRaft migration?
Yes, while the cluster is still in migration mode and before you finalize the migration, you can revert to ZooKeeper mode. The steps you follow depend on the last migration step the cluster completed, so the first task is to confirm exactly which phase the controllers and brokers have reached. Treat that confirmation as a documented check, not an assumption made from the runbook. The rollback path narrows with every completed phase, and once you pass the point of no return, the answer changes to no.
The effort is mostly planning, rehearsal and verification, not a single command. Write the revert procedure for each phase you might stop in, rehearse it on a non-production cluster that has the same ZooKeeper and KRaft settings, and record what state you expect after each step. Ask your team to confirm the exact revert steps for your Kafka version against the Apache Kafka 3.9 KRaft operations documentation, because the steps are version specific and the migration guide is the authoritative source for them.
Plan for one known risk: the revert can cause problems if the KRaft controller epoch has moved past the last ZooKeeper controller epoch. Before you start any rollback, compare the two values from your cluster state, and if the KRaft epoch is ahead, stop and review the workaround for that issue (tracked as KAFKA-20488) before changing anything. Also avoid changing the metadata version or inter.broker.protocol.version during the migration, since that change is not supported at this stage.
Revert steps by migration phase
This is the earliest point at which a revert is simplest. The cluster still has its ZooKeeper-based controller doing the work, and KRaft controllers have not yet taken over cluster metadata. The steps for this phase are the ones in the Kafka 3.9 KRaft documentation for reverting before controllers are migrated, and they depend on exactly which step you last completed.
- Confirm the last completed step: Check the migration state and the steps you have run against the procedure. Do not assume the phase from the plan you wrote.
- Stop the KRaft controllers you started: If controller processes were launched for migration, confirm they are stopped before reverting the configuration.
- Restore the ZooKeeper settings on brokers: Brokers should return to their original configuration that points at ZooKeeper. Verify each broker individually.
- Confirm the cluster elects a ZooKeeper controller: Check that topics and partitions remain available and that the ZooKeeper controller is active before proceeding.
- Record the state: Keep the configuration and logs from the attempt for your change record.
Before you start, confirm the exact commands and order with the Kafka 3.9 KRaft documentation for your version, and test the revert in a non-production environment that mirrors your configuration.
Once controllers have been migrated but not every broker has been moved to migration mode, the cluster is mixed. Some brokers receive metadata from KRaft controllers and others still follow ZooKeeper. The revert path here is more involved because you must return each broker and the controllers to a consistent state. The Kafka 3.9 documentation describes the procedure for this phase.
- Map which brokers are in migration mode: Build a list of each broker and its current configuration before you change anything.
- Revert brokers in the documented order: Follow the sequence in the procedure rather than the order your automation happens to run.
- Watch the controller role: Confirm which controller is active after each change, since leadership moves between ZooKeeper and KRaft during this phase.
- Check partition leadership and ISR: Verify that leaders and in-sync replicas are healthy after each broker is reverted.
- Avoid metadata version changes: Changing the metadata version (
inter.broker.protocol.version) is not supported during a ZooKeeper to KRaft migration, so do not attempt upgrades in the middle of a revert.
Work through the broker list one step at a time and only proceed when the cluster reports healthy state after each step.
When every broker runs in migration mode, the cluster is close to completing the move, and reverting is more constrained. Your options depend on the last migration step completed, and the KRaft controller epoch can create problems. Kafka 3.9 documentation tracks this as KAFKA-20488, and reverting may cause issues if the KRaft controller epoch exceeds the previous ZooKeeper controller epoch. Confirm the procedure for this phase in the Kafka 3.9 documentation before you attempt any revert.
- Check the controller epoch first: Compare the KRaft controller epoch with the last ZooKeeper controller epoch recorded before the migration started.
- Plan for the epoch issue: Treat the workaround described for KAFKA-20488 as a planned step with a written rollback check, not an ad hoc fix.
- Keep a full backup: Capture the metadata and configuration state before you begin, so you can compare after the revert.
- Run a test cluster rehearsal: Practice the revert on a copy of production metadata where possible.
- Escalate before finalization: If you are unsure of the state, stop and get Kafka project guidance or your support provider's input before any irreversible step.
Do not proceed to finalization while you are still deciding whether a revert is needed, because the next phase has no return path.
The controller epoch problem and how to work around it
Reverting from migration mode to ZooKeeper can fail when the KRaft controller epoch has grown beyond the last ZooKeeper controller epoch. The project tracks this as KAFKA-20488, and the Kafka 3.9 documentation warns that a revert may cause issues in exactly this case. Before you schedule any revert window, record the current controller epoch from the KRaft controller side and the last epoch the ZooKeeper controller wrote, then compare the two. The ZooKeeper value is visible in the controller znode and the KRaft value is exposed through the controller's metadata and logs. Confirm the exact paths and commands for your version in the 3.9 operations documentation, because the locations are not the same across every release.
A mismatch rarely announces itself as a clean failure. The usual symptom is a controller that refuses to take leadership or brokers that disagree about which controller is current after you have reverted the configuration. Treat any unexpected controller election, stale leader metadata or brokers stuck on an old epoch during a rehearsal as a stop signal, not as something to retry. Rehearse the revert on a non-production copy of the cluster with the same migration step reached, and keep the controller logs from that rehearsal so you can compare them with production behavior if you have to act.
Workaround options should be evaluated in writing, not chosen from memory. Start with the KAFKA-20488 issue itself, including its comments, linked pull requests and the release in which any fix was merged. Then check the Kafka 3.9 KRaft documentation for the revert steps that apply to your last completed migration step. Do not assume a fix is present in your build because the issue is old or the issue was discussed. If you need a fix that is not in your release, ask a support provider to confirm whether a backport exists, and plan the rollback and the change window around the answer. Record that decision and the evidence behind it in your change ticket.
What you cannot change during the migration
While a cluster runs in migration mode, a few changes are off limits and others can undo the rollback path you are relying on. The cards below cover the metadata version, broker configuration, upgrade timing and client changes. Keep each of these frozen until the migration is finalized or the rollback decision is closed.
Metadata version stays fixed
Changing the metadata version, which the broker protocol setting inter.broker.protocol.version controls, is not supported during a ZooKeeper to KRaft migration. Do not attempt a version bump to test the new cluster. Confirm the setting is identical on every broker before the first migration step, and record it in the change log so that a rollback starts from a known state. Ask your platform team whether any configuration management role or automation could rewrite this value during the window.
Broker settings changed mid-migration
Each migration step expects a specific set of controller and broker settings. Editing unrelated settings during the window makes it harder to tell whether a later failure came from the migration or from your change. Freeze the configuration files, deploy them from version control, and compare running values against those files after every rolling restart. Ask for the exact keys each phase needs from the official documentation, and do not rely on older blog posts or copied snippets from other clusters.
Upgrades timed around migration
Kafka releases from the 4.x line support only KRaft mode, so a ZooKeeper-based cluster must finish migrating before it moves to any 4.x release. Do not schedule a broker binary upgrade, a Java runtime change or a Kafka version move inside the migration window. Your runbook should state which release each cluster runs at the start and which release is planned after finalization. Check the upgrade notes for your starting release before you approve any date.
Client changes outside the window
Client libraries, Java runtimes and application deployments belong in separate change windows from the cluster migration. Since Kafka 4.0, clients and Kafka Streams applications require at least Java 11, so a runtime change on the application side can get bundled into a deploy that later needs to be rolled back. Keep producer and consumer changes out of the rollback window. List every client version in use so you can verify compatibility before cutover, and ask each application owner which client version they pin.
Why finalization is one-way
Finalizing the migration is the point of no return. Once a cluster is finalized, ZooKeeper is no longer the source of metadata and the revert path to ZooKeeper mode closes. Everything before that point can still be undone, with steps that depend on the last migration step completed. After it, the only way forward is to keep running KRaft. This makes finalization a go or no-go gate, not a routine step to run when convenient.
The reason it matters is the release line. Apache Kafka 4.0 supports only KRaft mode and removes ZooKeeper mode, and ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. A cluster that still depends on ZooKeeper cannot follow the 4.x path, so finalization is also the step that commits you to the KRaft upgrade track. Confirm the target version in the Kafka upgrade notes before you schedule the gate, since the upgrade path depends on the release you are moving to.
Before passing the gate, confirm three things in your own environment: that every broker and controller reports healthy under KRaft, that clients and tools have been tested against the KRaft endpoints, and that you have a restore plan for the period after finalization. Also confirm that no metadata version change is pending, because changing the metadata version (inter.broker.protocol.version) is not supported during a ZooKeeper to KRaft migration. Record the gate decision and its evidence so the team can show why the revert window was closed.
Checks to run before removing ZooKeeper
This sequence confirms that the cluster has run stably in KRaft long enough to trust its metadata, that access and identity still work, and that a rollback path is recorded before anyone deletes ZooKeeper. It ends with a written go or no-go decision that names who signed it.
Confirm the cluster has run stably in KRaft
The platform team watches the cluster in KRaft mode for a period it sets in its own change plan, covering normal produce and consume traffic, controller failover during a planned broker restart, and partition leader elections. Check broker logs for repeated controller errors, under-replicated partitions that do not recover, and metadata fetch failures. The step is done when the team can point to a clean observation window in its monitoring with no unexplained controller or metadata errors.
Verify controller epochs and controller health
Compare the active KRaft controller epoch with the last ZooKeeper controller epoch recorded before migration, using the controller metrics and logs your version exposes. Confirm which nodes are active controllers, that quorum is healthy, and that no broker is repeatedly re-registering. Because a KRaft controller epoch above the previous ZooKeeper epoch is the issue that complicates reverting, record the values now and treat any unexplained jump as a blocker. The step closes when the values and the quorum state are written into the change record.
Validate ACLs after the migration phases
Run a access test matrix for every principal, topic, consumer group and transactional ID that the cluster serves, after each migration phase and not only at the end. Compare the ACLs listed on the KRaft side with the ACLs recorded before migration, and confirm the authorizer change from AclAuthorizer to StandardAuthorizer has carried every rule across. Check that resources without ACLs still behave as your policy requires, since the default restriction to super users applies unless allow.everyone.if.no.acl.found is set. The step is done when every matrix row matches its expected result. The sibling page on moving ZooKeeper ACLs and SCRAM credentials to KRaft covers the detailed procedure.
Validate SCRAM credentials and client authentication
Confirm that every SCRAM-SHA-256 or SCRAM-SHA-512 user can authenticate against the KRaft listeners and that credentials exist in the metadata log, not only in the old ZooKeeper store. Test one account per application, including service accounts used by connectors and stream processors, and record which accounts are still on the legacy path. Also confirm that any Kerberos, PLAIN or OAUTHBEARER listener still accepts its expected clients. The step closes when each application owner has confirmed a successful login from their own environment.
Confirm client connectivity and monitoring coverage
Inventory every producer, consumer, Kafka Connect worker, Kafka Streams application and admin tool that connects to the cluster, and confirm each uses bootstrap addresses and listeners that will survive ZooKeeper removal. Check that dashboards and alerts were rebuilt for KRaft controller and metadata metrics and that ZooKeeper-specific alerts have been reviewed rather than silently left firing or silenced. Run a produce and consume check on representative topics from each client group. The step is done when the inventory has no unknown clients and every alert in the new set has a named owner.
Back up metadata and configuration
Take a backup of the KRaft metadata log directories on every controller and broker, together with the broker and controller configuration files, the ZooKeeper data and its configuration as they stood before removal, and the ACL and SCRAM exports. Store the copies outside the cluster hosts, record checksums, and perform a restore test on a non-production copy to confirm the files are readable. Confirm the backup location and the retention period with your own backup policy. The step is complete when a restore test has succeeded and the backup manifest is signed off by the platform owner.
Record a written go or no-go decision
Assemble the results of the earlier steps into one decision record that lists each check, its evidence, any open defects, and the rollback procedure that applies at the current migration state. Name the people who must approve, typically the Kafka platform owner, the security owner and the owners of the most important consuming applications. Mark the decision as go only when every check passes or has a documented, accepted exception. The step ends with a dated, signed go or no-go, and the ZooKeeper removal steps begin only after a go is recorded.
Rollback readiness by migration state
This table compares what can still be undone at each point in a ZooKeeper to KRaft migration, the main risk at that point and what to verify before moving on. Read it top to bottom as the cluster progresses, and treat any row you are unsure about as a reason to stop and confirm the exact revert steps for your Kafka version before changing anything.
| Migration state | Rollback still possible | Main risk | What to verify |
|---|---|---|---|
| ZooKeeper mode, migration not started | Yes, nothing to revert | Controller and metadata settings are not yet changed, so the risk is low but planning gaps remain | Confirm the Kafka version supports the migration path and that ZooKeeper-based clusters are on a release that can move to KRaft |
| Migration mode, before the last migration step | Yes, with steps that depend on the last completed step | Reverting may fail if the KRaft controller epoch has moved past the previous ZooKeeper controller epoch | Record the last completed migration step and compare controller epochs before any revert |
| Migration mode, after the last migration step completed | Limited, and only with the revert steps for that state | Controller epoch divergence is the main issue to work around, and recovery options narrow as traffic runs on KRaft | Check the controller epoch, broker health and metadata consistency, and confirm the documented revert path for your version |
| Finalized KRaft | No, finalization is one-way | ZooKeeper cannot be restored from the cluster, so any problem must be fixed forward | Run the pre-removal checks, confirm ACLs and SCRAM credentials work in the metadata log, and keep verified backups before removing ZooKeeper |
What goes wrong during a rollback attempt
Most failed rollbacks come from acting on an assumption about where the cluster stands. The cards below cover the risks that appear while a cluster is in migration mode, and each one names a check to run before you change anything.
Rolling back past the last completed step
The revert steps depend on which migration step finished last, so a runbook written for one phase can damage a cluster sitting in another. Before any change, confirm the last completed step from the controller and broker logs and match it to the revert sequence for that phase. Record the result in the change ticket so the on-call engineer works from the same state.
Controller epoch mismatch
A revert can fail if the KRaft controller epoch has moved past the last ZooKeeper controller epoch. Compare the two values before reverting, and treat any gap as a reason to stop and escalate rather than force the revert. Also confirm the ZooKeeper ensemble is healthy and reachable before you begin, because a revert depends on it.
Changing the metadata version mid-migration
Changing the metadata version, which is governed by inter.broker.protocol.version, is not supported during a ZooKeeper to KRaft migration. An upgrade bundled into the same change window can leave you with a cluster that is neither cleanly on ZooKeeper nor cleanly on KRaft. Freeze version changes until migration is complete and verified, and review the deployment manifests for any version bump.
Losing ACL and credential state
Access control lists and SCRAM credentials are stored in different places before and after the migration, so a revert or a partial move can drop permissions or lock out clients. Export the current ACLs and SCRAM users before each phase, then test producer and consumer access on a sample of topics. Compare the results against the export after every change.
Treating a partial migration as safe to finalize
Finalization removes ZooKeeper from the picture and cannot be reversed, so a cluster that still has brokers or controllers outside the expected state should not be finalized. Before removing ZooKeeper, confirm that every broker reports the expected role, that no broker still depends on ZooKeeper settings, and that clients have been tested on the new metadata path. Make finalization a separate, approved decision.
Questions teams ask before attempting a rollback
Can we revert a ZooKeeper to KRaft migration after it has been finalized?
No. Finalization is the point of no return, so a cluster that has been finalized cannot be moved back to ZooKeeper mode. Revert steps only exist while the cluster is still in migration mode, and they depend on which migration step was last completed. Before you finalize, confirm the exact revert procedure for your Kafka version in the Kafka 3.9 KRaft documentation, and make sure your team agrees that the cluster has been stable long enough to commit. Treat finalization as a separate, approved change with its own rollback plan for the data and clients that depend on the cluster.
How long does the rollback window stay open during migration?
The window stays open only while the cluster remains in migration mode and has not been finalized. The project documentation does not give a fixed duration, so do not plan around one. Ask the vendor or your own platform team to confirm which migration states are reversible for your version, and how long your cluster can safely remain in dual-mode operation given your monitoring, your change calendar and your recovery needs. Record the decision point in the change plan so the window closes on a known date rather than by default.
What drives the effort of a rollback rehearsal?
The main drivers are cluster size, the number of brokers and controllers, the number of topics and partitions, how many clients and consumer groups depend on the cluster, and whether ACLs and SCRAM credentials were moved during the migration. Rehearsal also takes time to build a staging copy that matches production configuration, to script each revert step by migration phase, and to record the controller epoch before and after each step. Ask for the rehearsal scope in writing, including which phases are tested, what evidence is captured, and who signs off the result.
Can our own team run the rollback without outside help?
A team with strong Kafka operations experience and a tested runbook can do it internally. The work is still demanding because the controller epoch issue can make a revert unsafe, and a mistake in the wrong phase is hard to undo. Check whether your engineers have already run a KRaft migration in a non-production environment, whether they can read controller logs and metadata state, and whether they have a documented escalation path to the Kafka community or a vendor. If any of those are missing, a partner can review the plan, run the rehearsal or sit alongside your team during the change.
What does the rollback work need from our side?
You need to provide access to the staging and production clusters, the current server and controller configuration, the migration state of each node, and the monitoring that shows broker and controller health. You also need client owners who can confirm which applications produce and consume, and a change window approved by the people who own those services. Give the team a named decision maker who can call a stop or a go at each phase. Without those inputs, the rehearsal results will not match what happens in production.
What should we ask a partner to include in a quote for this work?
Ask for a written scope that names the Kafka version, the cluster topology, the migration phases covered, and the revert steps to be rehearsed. Request the deliverables, such as a rollback runbook, a rehearsal report, a pre-finalization checklist and a post-change verification plan. Confirm which activities are one-time (assessment, runbook, rehearsal) and which are recurring (retainer support, if any). Ask how the partner will handle access, change control and out-of-hours support, and what happens if the rehearsal finds a blocker. DeployOpen does not publish prices, so the quote follows from that written scope.
Which checks must pass before we remove ZooKeeper?
Before removing ZooKeeper, confirm that the cluster has been finalized, that all brokers and controllers run in KRaft mode, and that no component still reads ZooKeeper-specific settings. Verify that producers, consumers and admin tools work after each phase, that ACLs and SCRAM credentials resolve from the metadata log, and that the controller epoch behaves as expected under a controlled failover test. Keep a backup of the ZooKeeper data until the cluster has run stably for an agreed period. The exact list depends on your version, so cross-check it against the Kafka 3.9 KRaft documentation before the change.
How DeployOpen scopes and runs a migration rollback plan
DeployOpen treats a ZooKeeper to KRaft rollback as a planned, phase-by-phase procedure agreed before the migration starts. We map where each cluster sits in the migration, which revert steps apply at that point, and which checks must pass before any ZooKeeper removal is approved.
How we scope
- Inventory each cluster's current migration phase, Kafka version and controller setup before writing any rollback steps
- Document which revert path applies at each migration step and where the one-way point sits
- Record the controller epoch comparison and the workaround agreed for it in the written scope
- Confirm change windows, rollback decision owners and go or no-go criteria in writing before work starts
How we build
- Write a runbook per migration phase with exact commands, configuration keys and expected log output
- Rehearse each revert step in a non-production copy of the cluster before the production change
- Verify metadata health, controller state and client connectivity after every phase, not only at the end
- Test ACL and SCRAM access after each phase so security state is confirmed alongside cluster health
How we hand over
- Deliver the phase runbooks, rollback decision tree and pre-removal checklist as documented artifacts
- Record the test results from rehearsals and production checks so your team can audit each decision
- Walk your operators through the rollback path and the conditions that end the reversible window
- Provide a written close-out stating which checks were run and which items remain your team's to confirm
How we support
- Offer ongoing support and upgrade work as a retainer after the migration closes
- Review rollback readiness again before any Kafka version change that touches the migration path
- Run follow-up checks on ACLs, credentials and controller health after ZooKeeper removal
- Keep runbooks current with the Kafka documentation for the versions your clusters run
