Apache Kafka 3.9 to 4.0 upgrade

What the Apache Kafka 3.9 to 4.0 upgrade changes first

If your team runs a 3.9 cluster and must reach Kafka 4.x, the first thing to know is that Apache Kafka 4.0 supports only KRaft mode, and ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. The upgrade is therefore not a version bump on an existing cluster. Confirm first that your 3.9 cluster is already running in KRaft mode, and that the controller quorum, broker metadata and admin scripts all operate without ZooKeeper. If any part still depends on ZooKeeper, that migration becomes the first project and the 4.0 upgrade waits behind it.

The change touches more than brokers. Controllers, brokers, Kafka Connect workers, command-line tools and client applications each have their own compatibility conditions in the Kafka upgrade notes. Brokers, Connect and tools now require Java 17, while clients and Kafka Streams applications need at least Java 11. Admin scripts that still pass --zookeeper must move to --bootstrap-server. Inventory each of these components before you schedule a window, because a broker-only rollout can leave Connect or client fleets broken.

The supported route is set out in the Kafka time-based release plan, which says rolling upgrades are supported from each of the last three releases to the latest version. Check the Kafka upgrade notes for 4.3 against your exact starting version before you choose a sequence, and confirm the current supported releases on the Kafka downloads page. Clusters older than 3.3.x should move to 3.9.x first. Plan the sequence as controllers, then brokers, then clients, and treat any gap outside the supported range as a separate upgrade step.

Breaking changes and deprecations in Kafka 4.0

This table lists the Kafka 4.0 changes that most often stop an upgrade from 3.9, with the impact of each and the action to take before the first broker restarts. Read it as a pre-flight checklist: any row with an impact on your clusters or clients needs an owner and a finished action before you schedule the rollout.

Change	Impact	Action needed
ZooKeeper mode removed	Kafka 4.0 runs only in KRaft mode, so a ZooKeeper-based cluster cannot move to 4.x as it is	Confirm the cluster is already on KRaft and that the migration finished before planning the 4.x step
Java 17 for brokers and Connect	Brokers, Kafka Connect workers and command-line tools fail to start or run on older JVMs	Check the JVM on every broker, Connect worker and tool host, and upgrade it before the rolling restart
Java 11 minimum for clients and Streams	Clients and Kafka Streams applications on Java 8 will not run against the new stack	Inventory Java versions for applications and Streams jobs, and plan their runtime upgrades separately
Pre-0.10.x protocols removed (KIP-896)	Clients built for 0.x, 1.x and 2.0 are not compatible with Kafka 4.0 brokers	Find these clients in your inventory and upgrade them to a supported client version before the brokers move
--zookeeper option removed from admin tools	Scripts and runbooks that pass --zookeeper to kafka-topics or similar tools will fail	Replace --zookeeper with --bootstrap-server in automation, runbooks and monitoring jobs
KIP-848 consumer protocol becomes available	Consumer group behavior and rebalancing can change for groups that opt in	Confirm the default and any opt-in steps in the Kafka 4.0 upgrade notes, then test group behavior in staging

Which clusters must move to KRaft before they can upgrade

If your cluster still runs in ZooKeeper mode, you cannot move it to Kafka 4.0 directly. The Kafka upgrade notes state that Apache Kafka 4.0 supports only KRaft mode and that ZooKeeper has been removed, so ZooKeeper-based clusters must migrate to KRaft before upgrading to 4.x. That makes the migration a prerequisite decision, not a step inside the 3.9 to 4.0 change window. Decide early whether your cluster is still on ZooKeeper, because that answer determines whether this upgrade is a version change or a platform migration with its own project plan, owners and rollback procedure.

The upgrade notes also recommend that KRaft clusters older than 3.3.x first upgrade to 3.9.x before moving to 4.x. Treat this as a bridge requirement: a KRaft cluster on an early release should go to 3.9.x, be verified there, and only then start the 4.x path. The ZooKeeper-to-KRaft migration is a separate project. It has its own preparation, cutover, validation and rollback plan, and its rollback window is not the same as the rollback window for a broker-version upgrade. Plan for both, and do not assume a rollback of one reverts the other.

Before you schedule any work, confirm your exact starting version, your metadata state and whether any cluster still depends on ZooKeeper, using the Apache Kafka 4.3 upgrade notes. Record the broker and controller versions, the metadata mode, and any tooling that still expects ZooKeeper connection strings. Check that the path you intend is documented for your starting point, since the upgrade notes are the authority for which hops are supported and in what order. A cluster that is older or in an unexpected state needs a written upgrade path agreed with the owners before any broker is touched.

Preparing the Kafka 3.9 to 4.0 upgrade

Preparation for a Kafka 3.9 to 4.0 upgrade is mostly inventory and rehearsal. The cards below cover what runs against the cluster, how the configuration compares with the upgrade notes, what you can restore if something goes wrong, and whether a staging environment matches production security closely enough to trust the result.

Inventory every client and library

List each producer, consumer, Kafka Streams application and Connect worker, along with the Kafka client library version it bundles. Kafka 4.0 removed pre-0.10.x protocols, so 0.x, 1.x and 2.0 clients will not work after the upgrade. Include Connect plugins and any framework that embeds a Kafka client, because those versions are easy to miss. Application owners usually hold this list, and platform teams should confirm it against broker request metrics before signing off.

Review configuration against the upgrade notes

Compare every broker and controller property file with the Kafka 4.x upgrade notes, looking for removed or renamed settings and defaults that changed. Confirm that the cluster already runs in KRaft mode, because Kafka 4.0 supports only KRaft and ZooKeeper-based clusters must migrate first. Check that admin scripts and automation use --bootstrap-server rather than the removed --zookeeper option. Platform engineers usually own this review, with the Kafka administrators signing off.

Confirm the Java runtime on every host

Brokers, controllers, Kafka Connect and tools require Java 17 from Kafka 4.0 onward, while clients and Kafka Streams need at least Java 11. Verify the JVM on each broker, controller and Connect host, and confirm that container base images and startup scripts point to the intended runtime. Record the JVM flags in use so they can be reapplied unchanged. Infrastructure or operations teams usually own the host and image changes.

Back up metadata and topic data

Take backups of the KRaft metadata log and the controller and broker configuration before the first restart. Decide, from your recovery plan, whether topic data also needs a copy, since a broker rollback is limited once the metadata version is finalized. Store backups off the cluster and test a restore on a separate environment. Operations or database-style recovery owners usually run this work, and the recovery time expected from it should be agreed with the business owner of each topic.

Build a staging environment that matches production

Rehearse the upgrade on a staging cluster with the same SASL mechanisms, TLS listeners, ACLs and quotas as production. Authorization for resources without ACLs depends on allow.everyone.if.no.acl.found, so a staging cluster without matching ACLs can pass tests that production would fail. Replay representative client traffic, including Streams and Connect workloads, and record the results. Security engineers and the platform team usually share ownership of this environment.

Agree the order and the go or no-go criteria

Write down the upgrade sequence for controllers, brokers and clients, and the conditions that stop the rollout, such as client errors, lag growth or failed health checks. Confirm which Kafka release each step targets and check the supported upgrade path in the release plan. Name one decision owner who can pause the rollout. Change managers and the Kafka platform owner usually agree this list before the maintenance window is booked.

Order of the Kafka 3.9 to 4.0 upgrade

This sequence moves a KRaft cluster from Kafka 3.9 to 4.0 with the controllers, brokers, Java runtime and client-facing components changed in an order that keeps the cluster available and leaves a clear rollback point at each stage.

Confirm KRaft mode and clean metadata state
Before anything changes, the platform owner confirms that every cluster in scope already runs in KRaft mode on 3.9, since Kafka 4.0 supports only KRaft and ZooKeeper-based clusters must migrate first. Check that the metadata quorum is healthy, that no controller or broker is in a failed or lagging state, and that no unfinished partition reassignments are pending. Record the current version and metadata version for every node. The step is done when the cluster reports a healthy quorum and the inventory of versions is saved. If a cluster is still on ZooKeeper or an older KRaft release, stop here, because the upgrade notes recommend reaching 3.9.x first.
Run client and tool compatibility checks
Inventory every client, Kafka Streams application, Connect plugin and admin script that talks to the cluster, and compare their versions with the Kafka 4.0 compatibility notes. Clients older than the pre-0.10 protocol generation are no longer compatible, and any script that still calls the --zookeeper option must be found now. Owners of each application confirm the version they will run after the upgrade. The step closes when every client and tool has an owner and a version that is listed as compatible, or a dated plan to replace it. Your own team usually holds this list, so confirm the details against the compatibility page for your target release.
Move controllers, brokers and tooling hosts to Java 17
Install Java 17 on every controller, broker, Kafka Connect worker and host that runs Kafka command-line tools, while the cluster stays on its current Kafka version. Java 17 is required for brokers, Connect and tools in Kafka 4.0. Restart each node only as part of its normal rolling restart so the runtime change does not stack with a version change. Verify the JVM version on each host with the process still running and confirm the broker starts cleanly on the new runtime. The step is done when every node reports Java 17 and the cluster passes its health checks. Confirm the supported runtime options and JVM settings in the upgrade notes before you change any node.
Roll controllers and brokers one at a time
Upgrade the Kafka binaries on controllers and brokers one node at a time, starting with the controllers, then the brokers. Before each restart, confirm that under-replicated partitions and offline partitions are at their baseline, and after each restart wait for the node to rejoin the metadata quorum and its partitions to catch up. Only move to the next node when the health check passes. Keep a record of each node's version and restart time. This is the stage where rollback is most practical, because the metadata version has not been finalized and the cluster can still run mixed versions. The step is done when every controller and broker runs 4.0 and no partitions are under-replicated. Confirm the exact restart order and any mixed-version rules in the 4.0 upgrade notes.
Switch admin scripts to bootstrap-server
Update every runbook, cron job, automation and admin script that used the --zookeeper option so it uses --bootstrap-server with the broker endpoints. The old option was removed from admin tools in Kafka 4.0, so these scripts fail once the cluster runs 4.0. Run each updated command against a non-production cluster or a read-only query first, and confirm that it returns the same output as before. The step is done when a test run of each script succeeds and the change is recorded in your configuration management. Rollback is not affected by this step, since the changed scripts also work on 3.9 with the same endpoints, so you can make this change before the broker restarts.
Upgrade Connect and Streams applications
Upgrade Kafka Connect workers, connector plugins, Kafka Streams applications and other clients to the versions confirmed in the compatibility check. Roll Connect workers one at a time so tasks move to healthy workers, and confirm that connectors return to the running state after each restart. Deploy Streams applications in the order your team has tested, and watch consumer group rebalances and lag during the change. The step is done when every connector and application reports running on its target version and lag returns to its normal level. Each client can usually be rolled back independently, but confirm the rollback limits for each component in the upgrade notes before you deploy.
Confirm final state and record the rollback point
Run a final health review covering controller quorum status, broker versions, under-replicated partitions, consumer group lag, connector status and the output of the admin scripts. Then record the state in your change record, including the metadata version the cluster currently reports and the date of the last point where a rollback to 3.9 is still practical. Finalizing the metadata version is a separate step that can change what a rollback allows, so do not run it until the rollback window is agreed. Confirm the finalization command and the features it enables in the 4.0 upgrade notes before you run it. The step is done when the review is signed off and the rollback point is written down.

What each team must do during the upgrade

Platform operators own the controllers, brokers, JVM runtime and the security configuration that governs access. Kafka 4.0 supports only KRaft mode, so the first question is whether every cluster you run is already on KRaft. Clusters still on ZooKeeper must migrate before the 4.x upgrade begins, and the upgrade notes recommend that KRaft clusters older than 3.3.x move to 3.9.x first. Brokers and controllers must run Java 17.

Metadata and mode: Confirm each cluster runs in KRaft mode and record its current release, including any controller quorum settings.
Runtime: Install Java 17 on every broker and controller host, and check the JVM options in your service units against the upgrade notes.
Authorization: Review ACLs under the StandardAuthorizer, which stores them in the metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so confirm that setting before you roll brokers.
Security settings: Check the SSL and SASL listeners, the mechanisms in use and any mixed secured and unsecured listeners.
Admin tooling: Replace any scripts that pass --zookeeper with --bootstrap-server.

Provide the release inventory, JVM versions, listener and authorizer configuration, and an ACL export to your DeployOpen contact before scheduling the rollout.

Application and client teams own the producers, consumers and Kafka Streams applications that connect to the cluster. Kafka 4.0 removed pre-0.10.x protocols under KIP-896, so 0.x, 1.x and 2.0 clients are not compatible with the new brokers. Clients and Kafka Streams applications must run at least Java 11, which is a lower bar than the broker requirement but still requires an inventory.

Client inventory: List every application, the client library and its version, and the runtime it uses.
Protocol age: Identify any client that predates 0.10.x, since those clients will fail after the upgrade.
Admin scripts: Confirm that automation uses --bootstrap-server instead of the removed --zookeeper option.
Consumer protocol testing: Test the KIP-848 consumer protocol in a non-production environment before enabling it for any group, and confirm the client libraries you use support it.
Rollback path: Keep the previous client build available so teams can revert without a broker change.

Provide the client inventory, library versions, Java runtime versions and the test results for the consumer protocol to your platform team before the broker rollout starts.

Effort drivers in a Kafka 3.9 to 4.0 upgrade

This table lists the factors that most often expand a Kafka 3.9 to 4.0 upgrade, why each one adds work, and what to confirm in your environment before asking for a fixed-scope quote. Use it to inventory your estate first, since the count and age of clients, connectors and security settings usually matter more than broker count alone.

Driver	Why it lengthens the work	What to confirm
Number of clusters and environments	Each cluster needs its own controller, broker and client sequence, plus a staging rehearsal before production. Multi-region or air-gapped copies repeat the same steps.	How many clusters, which are production, and whether non-production environments mirror production versions and configuration.
Client applications and library versions	Clients on 0.x, 1.x or 2.0 cannot talk to 4.0 brokers, so they must be upgraded first. Each application may bundle its own Kafka library version.	Inventory of every producer, consumer and Streams application, with its Kafka client library version and owning team.
Kafka Connect workers and custom plugins	Connect requires Java 17 for workers and tools. Custom connector plugins may need rebuilding or testing against the new runtime.	Connect cluster count, connector and plugin versions, and whether any plugin is built in-house or unmaintained.
ZooKeeper-era configuration still in use	Scripts, runbooks and automation that call the --zookeeper option must move to --bootstrap-server. Old broker settings may also need review.	Whether any cluster still runs in ZooKeeper mode, and a search of scripts, Helm values and tooling for --zookeeper.
Security setup and ACL volume	SASL mechanism choices, TLS and mTLS listeners, and large ACL sets in the metadata log all need verification after each restart. Mixed secured and unsecured listeners add test cases.	Enabled SASL mechanisms per listener, whether mTLS is required, the approximate number of ACLs, and the state of allow.everyone.if.no.acl.found.
Cross-cluster replication	MirrorMaker 2 pipelines depend on both source and target versions. Mirror topologies must be checked in an order that keeps replication running.	Number of replication flows, their direction, and which side upgrades first in each pair.
Data volume and backup or restore time	Larger topics and longer retention increase the time needed for a verified backup and a realistic rollback rehearsal. Log size also affects broker restart time.	Total stored data per cluster, retention settings, and how long a verified restore takes today.
Number of teams who must test	Each owning team validates its own clients, connectors and topics against the new version. Coordination adds calendar time that is separate from technical work.	Named test owners per application, their availability windows, and sign-off criteria agreed before the change window.

Testing the upgrade and deciding where rollback stops

Test the move from 3.9 to 4.0 in a staging cluster that matches production before any production broker changes. The staging cluster should use the same KRaft topology, listeners, TLS and SASL settings, authorizer configuration and a representative set of topics, partition counts and retention settings. Connect it to clients at the versions you inventoried, including any Java 11 or Java 17 applications and any clients that still depend on the removed pre-0.10 protocols, because those are the clients most likely to fail. Rehearse the rolling restart in the order the upgrade notes describe: controllers first, then brokers one at a time. At each step, watch under-replicated partitions, leader elections, producer errors and consumer lag, and record how long each restart took. Those timings give you a production window based on evidence rather than estimates.

Rollback is simplest before the metadata version is finalized. Until that step runs, you can generally return a node to the previous software version, but the exact boundaries depend on your starting version. Confirm them in the Apache Kafka 4.3 upgrade notes and the notes for the release you are upgrading from, and do not assume the 3.9 behavior carries over. Rehearse the rollback in staging as well: stop one broker, reinstall the earlier version, start it, and confirm it rejoins the cluster and catches up. Write down which configuration changes, if any, must be reverted with it, so the team does not improvise during an incident.

Once you run the feature upgrade step to finalize the release version, the picture changes. Finalizing enables features that older binaries may not understand, so going back is limited to what the finalization guidance allows. Treat the finalization step as a separate, scheduled decision with a named owner and a stated rollback window, and do not combine it with the broker rollout. Use the finalizing and downgrading guidance for the metadata version to decide which features can be reverted afterward and how long the window stays open. Before production begins, agree in writing who can approve finalization, what evidence must be in hand, and at what point the team accepts that the 4.0 cluster cannot return to 3.9.

Questions teams ask before a Kafka 3.9 to 4.0 upgrade

What does a Kafka 3.9 to 4.0 upgrade cost?

No price is estimated here. The cost comes from who does the work and how much of it there is. Your team is paid in staff time for the inventory, staging tests, change windows and rollback planning. A partner is paid for the scope it agrees to in writing. Costs that recur include the staff hours needed to keep clients, Connect plugins and JVM versions current after the upgrade. Drivers that raise effort are the number of clusters, the number of client applications still on old versions, Connect deployments with custom plugins, and how many environments need a full dry run. To compare quotes, ask each provider for line items covering inventory, environment work, the broker and controller rollout, client support, testing, documentation and handover.

How long does a Kafka 3.9 to 4.0 upgrade take?

The duration depends on the cluster, not on a fixed number of days. The main drivers are how many clusters and environments you run, how many client applications and Connect workers must be checked or changed, whether your cluster already runs KRaft, and how many change windows your organization allows. Clusters that are still on ZooKeeper cannot upgrade to 4.x until they migrate to KRaft, which adds a separate project. Ask any provider for a written scope that lists the environments, the client inventory assumptions, the change windows and the acceptance tests, along with the timeline that follows from them.

What does our team need to provide for the upgrade?

Your team needs to supply the facts the upgrade depends on and the people who can make decisions. Expect to provide the current Kafka version and metadata mode for every cluster, an inventory of clients and their Kafka library versions, Connect workers and connector plugins, Streams applications, admin scripts, and the Java version on each host. You also need a staging environment that resembles production, named owners for each application team, approved change windows, and a rollback plan that the business accepts. Without these, any upgrade plan stays theoretical.

Can our own team perform the Kafka 3.9 to 4.0 upgrade?

Yes, a team can do this itself if it already runs Kafka in production and has JVM operations experience. Before starting, confirm that every cluster runs in KRaft mode, that brokers and Connect hosts run Java 17, and that the clients and Streams applications meet the minimum Java 11 requirement. Check that the Kafka upgrade notes for 4.0 are reviewed against your own configuration, that old admin tool options such as --zookeeper are removed from scripts, and that every client has been tested against a 4.0 broker in staging. Also confirm that you can roll brokers one at a time and that monitoring shows under-replicated partitions and client errors during each step.

Can a 3.9 cluster skip directly to 4.0?

A 3.9 cluster is the release the Kafka upgrade notes name as the step before 4.x, but you should confirm the exact path before you plan the change. Check the upgrade notes for the 4.0 release and the time-based release plan, which describes which releases support rolling upgrades to the latest version. Also check the supported releases list on the downloads page to see which versions receive fixes. If your cluster is older than 3.9, the notes recommend moving through the intermediate releases first, so you should confirm the chain with the official documentation before scheduling work.

What happens to clients that still use old protocols?

Clients that use protocol versions removed in Kafka 4.0 cannot connect to a 4.0 broker, so they must be upgraded before the brokers move. Kafka 4.0 removed the pre-0.10.x protocols under KIP-896, which means 0.x, 1.x and 2.0 clients are not compatible. Clients from 2.1 and later need a separate compatibility check against the Kafka compatibility page before you change the brokers. Admin scripts that still use the --zookeeper option also need to switch to --bootstrap-server. Build a client inventory first, upgrade the affected applications in staging, and only then roll the brokers.

How DeployOpen scopes and delivers a Kafka upgrade

DeployOpen plans and carries out the move from Apache Kafka 3.9 to 4.0 inside your own infrastructure, starting from a written scope. The work covers KRaft readiness, Java 17 for brokers and Connect, client inventory, and the controller, broker and client order, with documented handover at the end.

How we scope

Inventory current Kafka brokers, controllers, Connect workers and client versions against the 4.0 prerequisites.
Confirm that every cluster runs in KRaft mode, since ZooKeeper-based clusters must migrate first.
Record the agreed scope, deliverables and timeline in writing before any change is made to your clusters.

How we build

Prepare Java 17 runtimes for brokers and Connect workers, and test the upgraded images in a non-production environment.
Sequence controllers, brokers and clients in a rehearsal run, then document each step and its rollback point.
Check admin scripts and automation for removed options such as --zookeeper, replacing them with --bootstrap-server.

How we hand over

Deliver a runbook covering the rolling order, health checks between steps and the agreed finalization point.
Document the client compatibility inventory, including any pre-0.10 or 0.x, 1.x and 2.0 clients that must be replaced.
Walk your platform and security teams through configuration, access controls and evidence collected during the upgrade.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after the handover is complete.
Retainers can cover later 4.x releases, which the project supports on a rolling basis from the last three releases.
Confirm response expectations and support coverage in the written agreement, since these are set per engagement.