Why an old Kafka cluster needs a multi-step path to Kafka 4
If your cluster runs on a 2.x or early 3.x release and still depends on ZooKeeper, you cannot move straight to Kafka 4. Kafka 4.0 supports only KRaft mode, and ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. The route runs through the 3.9.x line, which is the bridge release for the ZooKeeper to KRaft migration. The Kafka upgrade notes also recommend that KRaft clusters older than 3.3.x move to 3.9.x before moving to 4.x, so a cluster that is already on KRaft but predates 3.3.x follows the same bridge.
The current supported releases listed on the Kafka downloads page are 4.3.1, 4.2.2 and 4.1.2. Your target should be one of these, and the exact patch level should be confirmed on the Apache Kafka downloads page on the day you plan the change, since bugfix releases ship as needed. Kafka is licensed under the Apache License 2.0 and the project publishes no paid edition, so the cost of this path is the engineering, testing and change-window time your team or a partner spends, not a license fee. Any commercial support you buy from a vendor is a separate decision and should be scoped on its own terms.
This page covers the jump from an old ZooKeeper-era cluster to Kafka 4. Routine moves between 4.x releases, such as 4.1 to 4.3, are covered on the sibling pages for rolling upgrades and for finalizing the metadata version, and the client compatibility checks for Kafka 4 have their own page. Read this page first to decide which bridge steps your cluster needs, then use the sibling pages for the detailed procedure at each stage.
What changes between old Kafka releases and Kafka 4
This table compares what each change means for a ZooKeeper-era or early KRaft cluster heading to Kafka 4, and what you need to do about it. Read it top to bottom as a checklist, since several rows set the order of steps. The main takeaway is that ZooKeeper must be gone and the 3.9 bridge taken before any 4.x upgrade.
| Change | Impact on your cluster | Action needed |
|---|---|---|
| ZooKeeper mode removed in 4.0 | Kafka 4.0 runs only in KRaft mode, so a ZooKeeper-based cluster cannot move to 4.x as it is. | Plan a ZooKeeper to KRaft migration first, and treat it as a separate project with its own rollback plan. |
| KRaft-only operation | Metadata lives in the KRaft controller quorum rather than ZooKeeper, which changes how controllers are deployed and monitored. | Confirm controller placement, quorum size and the tooling your team uses against ZooKeeper before migrating. |
| 3.3.x minimum for KRaft clusters | The upgrade notes recommend that KRaft clusters older than 3.3.x move to 3.9.x before 4.x. | Record the exact broker and controller versions in each cluster and step through 3.9.x if you are older than 3.3.x. |
| 3.9.1 required before migrating | Brokers must be on 3.9.1 before the ZooKeeper to KRaft migration starts, since 3.9 is the last bridge release. | Upgrade every broker to 3.9.1 and confirm it is healthy before scheduling the migration. |
| Java 17 for brokers and Connect | Brokers, Kafka Connect and tools require Java 17 from Kafka 4.0 onward. | Check the JVM on every broker, Connect worker and admin host, and upgrade any that run older Java. |
| Java 11 for clients and Streams | Clients and Kafka Streams applications require at least Java 11. | Inventory the Java version of each application and plan upgrades for any running below Java 11. |
| Rolling upgrade window | Rolling upgrades are supported from each of the last three releases to the latest version, so older 2.x or 3.x releases need intermediate hops. | Map your current version to the supported path in the release plan and schedule each hop as its own change. |
| Client protocol and client versions in use | Old clients and protocol versions may stop working on Kafka 4, and this is often the hardest part to see. | Confirm which protocol versions and client libraries are still in use, then review the client compatibility page for detail. |
How the ZooKeeper to KRaft migration fits between old releases and 4.x
The sequence has a fixed order: the cluster must first be on a release that supports the migration, the brokers must be on Kafka 3.9.1 before the migration starts, and the controllers move to KRaft before any move to 4.x. Kafka 4.0 supports only KRaft mode and ZooKeeper mode has been removed, so a ZooKeeper-based cluster cannot skip the migration and go straight to 4.x. The Kafka upgrade notes also recommend that KRaft clusters older than 3.3.x move to 3.9.x before 4.x. Treat 3.9 as the bridge release for the whole path, not as an optional stop.
Each stage has a different owner on your side. The metadata stage belongs to the platform team that holds the cluster state: confirm where ZooKeeper-held metadata will live after the migration and who signs off on the cutover. The controller stage covers the KRaft controller quorum, its controller.quorum.voters settings and the hosts that run it, and must be planned before any broker is switched over. The broker stage covers the rolling restart of each broker onto 3.9.1 and then into migration mode. Assign a named owner to each stage and record the rollback point for it before the first change is made.
Do not rely on this summary for exact steps, property names or the order of operations inside the migration. Read the official Kafka 3.9 KRaft documentation for the migration procedure, the required broker settings and the conditions for finalizing the migration, and confirm each one against the exact version you run. Confirm the controller and broker configuration in a non-production copy of your cluster first. Ask your vendor or implementation partner to put the confirmed steps, the checks and the rollback limits in writing before the change window is booked.
What to prepare before touching a production broker
Most failed upgrades are lost before the first broker restarts. The six areas below set the baseline you will roll back to, the evidence you will test against and the people who decide whether to proceed. Each one has to be settled in writing before the migration window opens.
Inventory every component and runtime
List every broker, controller, client application and Kafka Connect worker with its exact Kafka version, client library version and Java runtime. Since Kafka 4.0, brokers and Connect need Java 17, and clients and Kafka Streams applications need at least Java 11, so any host still on an older JVM is a blocker to find now. Record which clients are on ZooKeeper-era settings, because those will be reviewed again at the migration step.
Back up topic data and configuration
Capture topic configurations, partition counts, consumer group offsets, ACLs and the broker and controller configuration files before any change. Confirm that the backup can be restored into a separate cluster, not only read back from disk. Agree on who holds the copy and how long it is kept, since it is the fallback if the migration needs to be abandoned.
Build a representative test environment
Stand up a staging cluster that matches production topic layout, replication factors, listener security and client traffic patterns. Replay realistic produce and consume load through the full path, including the ZooKeeper to KRaft migration and the stop-and-restart of each broker. A staging run that skips the security settings or the client mix will not show the failures you are likely to meet in production.
Review authentication and ACL behavior
Document every SSL and SASL listener, mechanism and principal in use. In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Confirm the ACLs you migrate into the KRaft metadata match what clients need, because a missed ACL shows up as a production authorization failure.
Confirm bridge and rollback points
Decide which release each cluster must reach before the next step, such as 3.9.1 on brokers before the KRaft migration, and record the point after which rollback is no longer practical. Write down what a rollback involves in your environment, including the restore steps and the expected duration you have measured in staging. Treat each stop as a checkpoint, not a single window.
Name the go or no-go owner
Assign one named person with authority to halt the migration, and define the criteria they check at each stop: test results, backup verification, client inventory gaps and on-call coverage. Name the people who run each step and who is paged during the window. A single accountable owner prevents a partial migration from continuing because no one was willing to call it off.
The upgrade sequence from ZooKeeper to Kafka 4
The sequence takes a ZooKeeper-based Kafka cluster through a supported 2.x or 3.x release, the 3.9 bridge line, a KRaft migration and client and Connect updates before the final move to Kafka 4.x. Each stage ends with a check that tells you whether the next stage is safe to start.
Inventory brokers, clients and Connect workers
Your platform or Kafka team records the exact broker versions, the listeners and authentication settings, the topics and their configuration, and every client, Kafka Streams application and Connect worker that talks to the cluster, with the library or framework version of each. Run the inventory against the current Kafka upgrade notes and the Kafka 4.3 upgrade page so you know which items need changes before the cluster moves. The step is done when every client has a named owner and a version, and the team has a written list of what must change. Take a configuration and data backup before any change to the cluster. Use the outputs as the baseline for rollback.
Test the sequence in a non-production cluster
Rehearse every stage on a copy of the production topology with representative topics, security settings and client traffic. Your team runs the commands exactly as the official upgrade and KRaft pages describe them for the versions in use, and records each command, its output and any deviation. Confirm that producers, consumers and Connect tasks keep running through each rolling restart in the test. The step is done when the rehearsal completes without data loss, and the team has a signed-off runbook that names the rollback point for each stage.
Bring brokers to a supported 2.x or 3.x release
Upgrade the brokers one at a time to a supported 2.x or 3.x release that can reach the 3.3.x line or later, as the upgrade notes recommend for clusters that will move to KRaft. Check that each broker rejoins the cluster, that under-replicated partitions return to zero, and that client error rates stay within the levels you recorded before the change. Keep the previous package and configuration available until the next stage is verified. The step is done when all brokers run the target version and the cluster has been stable through a full business cycle that your team defines. Confirm the exact version path against the official notes.
Move the cluster to 3.9.x and then 3.9.1
Roll the brokers to the 3.9 bridge release, then to 3.9.1, which the KRaft documentation requires before the ZooKeeper to KRaft migration begins. Use the same one-broker-at-a-time pattern and verify replication health after each restart. The 3.9 line is the last bridge release for this migration, so this is the most important version checkpoint in the path. The step is done when every broker reports 3.9.1, partition leadership is balanced, and no broker has pending log or metadata errors. This is the last point where a return to the previous release is a routine rollback.
Run the ZooKeeper to KRaft migration with its checks
Provision the controller quorum and follow the 3.9 migration procedure in the KRaft documentation to switch metadata from ZooKeeper to KRaft. Run each validation the procedure lists before advancing, including the migration state checks and the controller and broker registration checks. Keep ZooKeeper available until the procedure says it can be decommissioned, because it is your main rollback path for this stage. The step is done when the migration reports complete, all brokers are registered with the KRaft controllers, and the team has recorded the rollback limits for this stage. Ask your Kafka platform owner to confirm the specific steps against the version you run.
Confirm the cluster is healthy on KRaft
Verify that the cluster runs with KRaft metadata end to end. Check controller quorum status, broker registration, partition leadership, consumer group operation, ACL and authorization behavior, and TLS or SASL listeners for every client group. Run a produce and consume test on representative topics and confirm Connect tasks and Streams applications still process records. Hold the cluster on this state for a period your team defines before any client or Java change. The step is done when the health checks pass, and the team has a written record that the cluster is stable on KRaft.
Upgrade clients and Connect to supported Java versions
Update producers, consumers, Kafka Streams applications and Connect workers to client versions that the compatibility checks marked as safe, using the client compatibility guidance for the target line. Brokers and Connect must run Java 17 from Kafka 4.0 onward, and clients and Streams applications need at least Java 11, so update the JVMs on each host before its client upgrade. Roll each application group separately and confirm its lag, error rate and throughput match the baseline. The step is done when every client group reports the target version and Java level, and no group needs a rollback.
Move brokers and metadata to Kafka 4.x
Upgrade the brokers to the Kafka 4.x release your team has chosen, using the official Kafka 4.3 upgrade notes for the exact sequence and version rules. Roll the brokers one at a time, then finalize the metadata version only after you have confirmed the rollout is stable, because finalizing limits some rollback options. Confirm that the cluster reports the expected version, that the health checks from the previous step still pass, and that no client errors appear after finalization. The step is done when the target release is finalized and the team has closed the rollback window in the runbook. Confirm the finalization steps and rollback limits with the official pages for the version in use.
Where client upgrades fit in the path
Client upgrades are not a separate project you can finish later. Clients, Kafka Streams applications and Kafka Connect workers often have to move on the same schedule as the brokers, because the Java floors have risen: since Kafka 4.0, brokers and Connect require Java 17, while clients and Streams applications require at least Java 11. A cluster that moves from a ZooKeeper-era 2.x or early 3.x release to 4.x will therefore expose any application still running on Java 8 or an older runtime. Before the first broker is touched, the team should know which runtime each service uses, because a Java upgrade can take longer than the broker work itself.
Start with an inventory. List every producer, consumer, Streams application and Connect worker, the Kafka client library and version each one bundles, and the JVM it runs on. Include non-Java clients, since librdkafka-based and other third-party libraries follow their own release lines. Owners of services that are rarely deployed are the usual gap, so the inventory should come from deployment manifests and dependency files rather than memory. Once the list exists, the team can decide which clients to upgrade before the rolling broker work starts, and which can wait because they will keep working across the broker versions in between.
Sequence the changes so producers and consumers keep working while brokers restart one at a time. Clients built on a client library that supports the broker versions in the rolling window can stay in place, and upgrading them first gives the team a stable target. Upgrade Connect workers and Streams applications in a planned window, since a rolling Connect cluster that loses tasks during a restart causes lag and reprocessing. Confirm each client's compatibility with the broker versions along the path, including the 3.9 bridge release, on the Apache Kafka 4.3 upgrade notes. The detailed matrix for 0.x, 2.x and 3.x clients is covered on the client compatibility page of this group.
How long each stage takes depends on the environment
A small cluster with a handful of brokers, a few topics and a limited set of owning teams usually has the shortest stage times. The main work is the ZooKeeper to KRaft migration itself, the broker rolls and the final validation, and there are fewer owners to coordinate with and fewer clients to test. Stage duration still depends on how many clients you must check and how much test time your team can schedule before production windows open.
- Small client inventory: Fewer applications mean a shorter compatibility review, but every client version still needs an entry in the inventory.
- Simple topology: Few brokers and no multi-site replication reduce the number of rolling steps and the verification scope.
- Single owning team: One team can make decisions and approve change windows quickly, which shortens waiting time between stages.
- Limited custom settings: Default listeners and simple security reduce what must be re-verified after each stage.
- Test copy availability: A test cluster that mirrors production lets you measure each stage before committing to a date.
Before you rely on any schedule, measure each stage in a test environment and ask for a quote based on the written scope.
A shared multi-team cluster with many Kafka Connect connectors takes longer because every team's clients, connectors and topics must be verified, and some owners may not be reachable during the same window. Connect workers and connector plugins often lag behind broker versions, so the migration plan has to account for them as separate work streams with their own tests.
- Connector plugin compatibility: Each connector plugin must be checked against the target version, and plugins with no current release may need replacement or a planned exception.
- Number of owning teams: Each team needs a test window, sign-off and a rollback plan, which adds coordination time.
- Connect worker rolling: Workers must be restarted without losing tasks, and the order of worker and broker changes affects how long each stage runs.
- Topic and ACL volume: Large numbers of topics and ACL entries increase the time needed to confirm that access still works after the migration.
- Change freeze calendars: Shared platforms often have overlapping freezes, which can lengthen the calendar even when the technical work is short.
Ask the team for a connector and client inventory with owners and versions, then plan the Connect stage separately from the broker stages.
A cluster that uses custom authentication or an external authorization service takes longer because the security configuration has to be rebuilt, tested and proven after each stage. Kafka supports SSL and several SASL mechanisms, and authorization is pluggable, so a non-standard setup needs confirmation at each step rather than a single check at the end.
- Non-standard SASL setup: Any mechanism beyond the common ones needs its own compatibility confirmation with the target release.
- External authorizer integration: Ask the vendor or project whether the authorizer you use supports the KRaft metadata model before planning the change.
- ACL migration and testing: In KRaft clusters, ACLs live in the metadata log, so existing ACLs must be exported, reviewed and confirmed after the move.
- Super user and default access rules: Confirm how resources without ACLs are treated, because the default restriction affects clients that depend on open access.
- Client credential changes: Rotating or reissuing credentials for many clients adds time and needs coordinated cutover.
Confirm your authorization design in the test environment before any production date is set, and treat it as a separate sign-off item.
What makes this upgrade longer or riskier
This table lists the factors that most change how long a move from Kafka 2.x or early 3.x to Kafka 4 takes and how much can go wrong. Read each row as a question to answer about your own cluster before estimating the work. The main takeaway is that client and connector inventory usually decides the schedule more than broker count does.
| Driver | Why it changes effort | What to check |
|---|---|---|
| Number of brokers and topics | More brokers lengthen each rolling restart and the migration window. More topics raise the number of configurations and partition assignments to verify afterward. | Count brokers, controllers and topics, and note which topics carry critical traffic or have unusual settings. |
| Custom authorizers or plugins | Custom authorizers, interceptors or broker plugins may depend on ZooKeeper-era internals or APIs and must be rebuilt or replaced before the cluster runs on Kafka 4. | Check each plugin against the Kafka 4 upgrade notes and confirm whether the vendor or internal team still maintains it. |
| Kafka Connect connectors | Connectors without a Kafka 4 compatible version block the move or force a replacement, and each replacement needs its own test cycle. | List every connector class and version, and confirm with each vendor which release supports the Kafka version you plan to run. |
| Old client libraries and frameworks | Clients built on pre-2.x or older frameworks may use protocol versions removed in Kafka 4. Each team's application may need a library upgrade with its own release. | Inventory client libraries, Kafka Streams versions and framework versions per application, and record the owning team for each. |
| Security configuration (SSL, SASL, ACLs) | Listener, certificate and SASL changes must be applied consistently across brokers and clients. ACL migration errors can block production traffic. | Document listeners, TLS certificate chains, SASL mechanisms and the current ACL set, and export them before the first change. |
| Data volume and replication load | Large partitions and high replication traffic lengthen leader movement and catch-up time after each broker restart. Tight disk headroom raises the risk of a failed restart. | Measure under-replicated partitions, disk usage and typical catch-up time during normal operations, and confirm the replication factor for critical topics. |
| Number of teams that must test | Each owning team must validate its producers, consumers and pipelines against the new cluster, and coordination across teams often sets the calendar. | Name an owner for each application and agree on test environments, sign-off criteria and a freeze window in advance. |
| Rollback expectations | Rollback after a KRaft migration or version change is limited, so a longer test phase before the cutover is usually needed. | Confirm with the Kafka upgrade notes which rollback options apply to your starting version and plan the pre-cutover test window accordingly. |
Questions teams ask before upgrading an old Kafka cluster
How much does upgrading an old Kafka cluster cost?
The cost depends on the written scope, not on a list price, and DeployOpen does not publish one. Kafka itself is licensed under the Apache License 2.0 with no paid edition from the project, so the spend is labor, infrastructure and testing. The main drivers are the number of clusters, how many brokers and controllers run in each, how many client applications must be inventoried and changed, the state of your monitoring and backups before work starts, and how many environments must be rehearsed. Ask any quote to separate one-time work (inventory, the migration, client changes, runbooks) from recurring work (support, patch intake, and future 4.x upgrades), and to state which environments and rollback rehearsals are included.
How long does the upgrade from Kafka 2.x to Kafka 4 take?
Duration is set by the number of hops, the cluster size, and how many clients need work. The path runs through the bridge release, since the upgrade notes recommend that KRaft clusters older than 3.3.x move to 3.9.x before 4.x, and ZooKeeper clusters must migrate to KRaft on 3.9.1 or later before they can reach 4.x. Each hop needs a rolling restart, a verification window and a rollback plan. Clusters with many topics, large partition counts or heavy replication usually need longer validation. Ask for a schedule with a named gate for each hop, rather than a single end date.
What does the upgrade require from our own team?
Your team provides access, knowledge of the business owners for each topic, and the change windows that match your service levels. In practice you need people who can approve production changes, grant access to brokers, controllers, monitoring and client repositories, and confirm which applications depend on which topics. Expect your engineers to own the client version inventory, since they know which services use which libraries. Agree on who signs off each gate before work starts, and who can stop the rollout if consumer lag or error rates change.
Can our internal team perform this upgrade without a partner?
Yes, a capable internal team can run the upgrade if it has Kafka operations experience, a tested rollback plan, and time to rehearse in non-production environments. The hard parts are the ZooKeeper to KRaft migration, the Java 17 requirement for brokers and Connect since Kafka 4.0, and finding every client that will break. A partner can take over planning, client inventory, rehearsal and documentation, or act as a second pair of hands on the day. Whichever route you choose, write down who owns each step and how you will check it.
Can we roll back to ZooKeeper after the migration to KRaft?
Plan on not rolling back to ZooKeeper once the cluster is running in KRaft mode, because Kafka 4.0 removed ZooKeeper mode entirely and the migration is a one-way change for the cluster. Before the migration, confirm the exact rollback conditions in the official notes for your starting version, and rehearse the steps in a copy of production. Keep verified backups of configuration and topic data, and define the point after which a return to the prior state is no longer supported. Treat this as a decision to confirm with the upgrade notes and your own test results, not an assumption.
Which parts of this upgrade must be confirmed against the official notes?
Several details change between releases and must be checked against the notes for your exact versions before you commit to a plan:
What should we check in clients before upgrading to Kafka 4?
Client compatibility is the most common surprise in these upgrades, because Kafka 4.0 removed older protocol support and requires at least Java 11 for clients and Kafka Streams applications. Inventory every producer, consumer, Streams application and Connect plugin, record the library and framework versions, and test each against a 4.x broker in staging. Also confirm authentication settings, since SASL mechanisms and TLS configuration may need updating during the rollout.
Testing and rollback limits for each gate
Each gate in this path needs a tested rollback point before the team moves on, and the team should know in advance how far back it can go at every stage. The rollback window narrows once a metadata or feature version is finalized, so the point of no return has to be written into the plan before the first broker is touched. Rehearse the full sequence in a test cluster that matches production in broker count, controller layout, listener and TLS configuration, client mix and topic settings. Run the ZooKeeper to KRaft migration, the hop to 3.9 and the client changes there first, and record how long each step took and where it stalled.
After each step, verify four things before proceeding. Check broker health: every broker should rejoin, under-replicated partitions should return to zero, and logs should show no repeated errors. Check the controller quorum: confirm a leader is elected and the metadata log is advancing on all controllers, for example with kafka-metadata-quorum. Measure client throughput against a baseline taken before the change, including producer error rates and request latency. Watch consumer group behavior for unexpected rebalances, growing lag or committed offsets that stop moving. Keep these checks in a runbook so every gate uses the same criteria.
The exact rollback limits for each version step must be confirmed against the official documentation for that version, because they change between releases and depend on which metadata or feature versions have been finalized. Read the upgrade notes for each target release and the KRaft migration documentation before setting a gate, and ask your partner or vendor to confirm the rollback path in writing for your starting version. Do not assume that a rollback that worked in the test cluster will be available after production has finalized a feature version.
How DeployOpen scopes and delivers a Kafka upgrade
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including cloud accounts, data centers and air-gapped networks. For a Kafka 2.x or early 3.x cluster, we start by agreeing in writing which brokers, controllers, clients and migration steps are in scope, then work through the ZooKeeper to KRaft path in a sequence your team can review.
How we scope
- Inventory brokers, ZooKeeper ensembles, Kafka versions and the clients that connect to the cluster.
- Confirm the target path, including the 3.9 bridge release and any required 3.9.1 step before migration.
- Agree written deliverables, exclusions and acceptance criteria before any change is made to production.
How we build
- Rehearse the migration and upgrade steps in a non-production environment that mirrors your topology.
- Prepare the client upgrade plan, including Java 11 and Java 17 requirements, and test client compatibility before cutover.
- Run each stage with a documented rollback point and verify broker health, replication and consumer offsets.
How we hand over
- Deliver runbooks for each step, with the commands used, checks performed and observed timings from your environment.
- Record the final configuration, KRaft settings, authentication and authorization state for your operations team.
- Walk your engineers through the rollback limits and the conditions under which each stage can be reversed.
How we support
- Engagements are fixed-scope, and the written scope defines what is covered after handover.
- Ongoing support, upgrades and continuous testing are available as a retainer if your team wants continued help.
- Ask us to confirm support terms, response expectations and effort for any follow-on work in the written quote.
