Apache Kafka KRaft migration to Kafka 4

What a Kafka 4 migration involves

Moving to Kafka 4 is an ordered path of version steps, and your starting point sets the order. Three triggers are common: a ZooKeeper-based cluster that has to leave ZooKeeper, an older 3.x cluster that has to reach a supported line, or a move off another messaging platform, which adds topic, schema and client mapping on top of the version work. Each trigger produces a different first step, so the plan starts with an inventory of the current cluster version, its metadata mode and every client that connects to it.

The version constraints come from the project's upgrade notes. Kafka 4.0 removed ZooKeeper mode, so a ZooKeeper-based cluster must become a KRaft cluster before any 4.x broker runs. The notes recommend that KRaft clusters older than 3.3.x move to 3.9.x before they move to 4.x. Brokers, Kafka Connect and tools need Java 17 from Kafka 4.0 onward, while clients and Kafka Streams applications need at least Java 11. Confirm the current supported releases on the Kafka downloads page at engagement start. At the time of writing, 4.3.1, 4.2.2 and 4.1.2 are listed as supported, and the target should come from that list.

Whether to run this in-house depends mostly on operating experience. A team that has already run Kafka in production, including broker replacements, controller quorum changes and rolling restarts, can often run the path itself. Teams without that experience usually bring in a partner for the rehearsal and the cutover, which is where a failed step costs the most. Ask any partner to show the rehearsal plan, the rollback criteria and the client Java inventory in the written scope before work starts.

What the migration includes

Each work package is scoped and agreed in writing before work starts. The engagement ends with a target architecture, validated runbooks, updated security configuration and handover notes.

Inventory of brokers, topics and clients

Record every broker, topic, partition count, ACL, Kafka Connect worker, client application and admin script before choosing a path. ACLs matter most because in KRaft clusters the StandardAuthorizer keeps them in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is enabled. Confirm the inventory covers Connect and tooling, not only brokers.

Migration path decision

Choose between moving from ZooKeeper to KRaft, moving a KRaft cluster from 3.x to 3.9.x and then to 4.x, or moving to a different messaging platform. Kafka 4.0 supports only KRaft, so ZooKeeper-based clusters must migrate before upgrading to 4.x. The upgrade notes recommend moving KRaft clusters older than 3.3.x to 3.9.x first.

Java 17 runtime for brokers and tools

Since Kafka 4.0, brokers, Kafka Connect and command-line tools require Java 17. Plan the JVM upgrade on every host, including Connect workers and any scripts, cron jobs or container images that call the tools. Check that monitoring agents, configuration management and base images support the new runtime.

Client compatibility checks

Clients and Kafka Streams applications need at least Java 11 to run against Kafka 4.x. Test each producer, consumer and Streams application against the target broker version, including serializers, interceptors and any custom authentication or authorizer plugin. Applications that cannot move to Java 11 will block the upgrade, so identify them early.

Rehearsal in a non-production copy

Run the full sequence on a non-production copy with similar topic counts, ACLs, TLS and SASL settings. Time the controller and broker restarts, confirm that consumer groups resume with their offsets intact, and note any configuration drift. Use the findings to settle open questions before the production change window.

Cutover runbook and rollback criteria

Write a step-by-step runbook with named owners, checkpoints and a rollback trigger agreed before the window opens. Typical triggers include under-replicated partitions that do not clear, sustained client errors, or consumer lag that does not recover. Confirm with the vendor upgrade documentation which downgrade steps are supported for your version pair.

What drives cost and duration

This table lists the factors that most change migration effort for Kafka, how each one raises the work, and what keeps it under control.

Driver	Why it raises effort	What keeps it down
Starting version and ZooKeeper mode	ZooKeeper-based clusters must move to KRaft before any 4.x upgrade, and clusters older than 3.3.x are advised to reach 3.9.x first, so each extra hop adds a rolling upgrade cycle.	Confirm the exact current version and metadata mode early, then plan the shortest documented path to a supported 4.x release.
Number of brokers and controllers	More brokers mean longer rolling restarts, more controller-quorum planning and more verification after each step.	Automate broker restarts with health gates and rehearse the sequence on a non-production cluster of the same shape.
Topics and partitions	High partition counts lengthen leader elections and metadata handling during restarts, and each topic needs an owner to confirm it survives the move.	Produce a topic inventory with owners and retention settings, and retire unused topics before cutover.
Client applications and Java versions	Brokers require Java 17 and clients need at least Java 11, so applications on older runtimes need rebuilds or runtime upgrades, and each team tests separately.	Build a client compatibility matrix with each application, library version and runtime, and agree owners and test dates before the work starts.
Kafka Connect connectors and custom plugins	Connectors and custom plugins must be checked against Kafka 4 APIs and Java 17, and vendor connectors may need new releases.	List every connector and plugin with its version and source, then confirm compatibility or plan rebuilds for custom code.
Security configuration (TLS, SASL, ACLs)	Changing listeners, authentication or ACLs during the move risks lockouts, and StandardAuthorizer ACLs stored in cluster metadata must be migrated and checked.	Document current listeners and ACLs, change one security layer at a time and test each client against the new configuration.
Number of environments	Each environment repeats the upgrade, test and cutover steps, and differences between them are a common source of rework.	Keep environments built from the same definitions and promote the same runbook through development, test and production.
Compliance evidence needs	Assessors may ask for change records, test results, access reviews and proof of configuration before and after the move, which adds documentation work.	Agree the evidence list with your auditor or compliance lead before the first change, and capture it as each step completes.
Migration from another messaging platform	A different data model, delivery semantics or client API means topics, consumers and offsets cannot simply be copied, and application logic may change.	Map each queue or stream concept to a Kafka equivalent in a design review, and build a pilot flow before committing the full estate.

Migration phases and outputs

The sequence moves a cluster from ZooKeeper, an older 3.x release or another messaging platform to KRaft-mode Kafka 4 with each phase closed by a written output.

Inventory clusters, clients and runtimes
DeployOpen's engineers catalog every broker, controller, topic, client application, Kafka Connect worker and Streams app, recording its Kafka version and Java runtime. The output is a cluster and client inventory with a version and Java runtime map, which you review and sign off before any design work begins.
Design the upgrade path and target architecture
Using the inventory, the team selects the upgrade route, including any required stop at an intermediate 3.x release, and defines the target KRaft controller and broker layout. The output is a documented upgrade route and target architecture, agreed in writing as part of the fixed scope.
Remediate Java 17 and client compatibility
Brokers, Connect and tools are moved to Java 17, while clients and Streams applications are checked against the target release and remediated where needed. Each runtime and client is tested in a non-production environment. The output is a compatibility report listing tested runtimes, client versions and any open issues.
Rehearse the migration on a production-like copy
The full upgrade is run end to end on a copy of production that matches its topology, data volume and security configuration as closely as the inventory allows. Each step is timed, and rollback is exercised. The output is a timed runbook and recorded rollback test results.
Execute the cutover under a change record
The production cutover follows the rehearsed runbook under an approved change record, with checkpoints for controller quorum, replication health and client connectivity. Verification checks run after each stage. The output is an executed change record and the verification results for every checkpoint.
Support hypercare and hand over operations
After cutover, the team watches lag, errors and client behavior through an agreed hypercare window, then transfers ownership to your operators. The output is operations runbooks and a named ownership matrix, accepted by your platform team.

What stays in-house and what goes to a partner

The table separates decisions and application-side testing that most teams keep internally from migration engineering work that teams commonly outsource, with the skills each part needs.

Part of the migration	Usually kept in-house	Commonly outsourced
Cutover window and risk acceptance	Business owners choose the window, accept downtime risk and sign off the go or no-go decision	Partner prepares the options, the rollback criteria and a written cutover checklist for those owners to approve
Version path design	Platform owners confirm which running release and KRaft state the cluster is in today	Partner plans the stops, such as 3.9.x before 4.x for older KRaft clusters, and the broker, controller and client order
Client compatibility testing	Application owners test their own producers, consumers and Kafka Streams apps against the target release and Java 11 or later	Partner builds a shared test matrix, runs client smoke tests and tracks failures back to owners
ZooKeeper to KRaft and controller operations	Internal platform team keeps the controller quorum running and owns the operational knowledge afterward	Partner performs the metadata migration rehearsal and documents controller health checks, quorum recovery and rollback steps
Kafka authorizer and SASL configuration	Security team approves principals, mechanisms (GSSAPI, SCRAM, OAUTHBEARER) and ACL ownership	Partner checks ACL migration into the KRaft metadata log and confirms no resource falls back to super-user-only access
JVM and Java 17 runtime management	Platform team owns the JDK rollout across brokers, Connect workers and admin hosts	Partner validates JVM flags, heap settings and Java 17 startup on a staging copy before production
Data retention and topic configuration	Data owners decide retention periods and any legal hold rules for topics	Partner checks that retention, compaction and replication settings survive the move unchanged
Runbook authoring and cutover support	Operations staff run the cutover and sign the final state	Partner writes the runbook, staffs the cutover and supports the first days of production traffic

What goes wrong and how to prevent it

Most failed Kafka migrations trace back to a few known gaps in version order, client runtimes, connector code and access control. Each pattern below has a check that should happen before cutover or during the first days after it.

Skipping the 3.9.x step

KRaft clusters older than 3.3.x are advised by the upgrade notes to move to 3.9.x before moving to 4.x. Teams that jump straight to 4.x from an early KRaft release can end up with a cluster they cannot upgrade cleanly. Prevent this by writing the full version path into the plan, rehearsing each hop on a copy of production metadata, and confirming the path against the current upgrade notes for your exact starting version.

ZooKeeper metadata left unmigrated

Kafka 4.0 supports only KRaft mode and removes ZooKeeper mode, so a ZooKeeper-based cluster must complete its migration to KRaft before any 4.x upgrade. Teams sometimes upgrade brokers and assume metadata will follow. Prevent this by confirming in a rehearsal that the controller quorum is active, metadata is in the KRaft log, and no broker still depends on ZooKeeper before the 4.x step is scheduled.

Clients below the Java minimum

Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications require at least Java 11. Applications outside your platform team often run older runtimes that nobody tracked. Prevent this by inventory-scanning every service's runtime and build configuration, and setting a deadline for each team to move before the broker cutover.

Connect plugins built on older APIs

Kafka Connect connectors and custom plugins compiled against older APIs may fail to load or behave differently after the upgrade. Vendor connectors can also lag behind the release you move to. Prevent this by scanning plugin jars and their dependency trees, confirming a compatible build exists for each connector, and running each connector in a staging Connect cluster before production.

ACL gaps after cutover

In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources with no ACL are limited to super users unless allow.everyone.if.no.acl.found is set to true. Topics or groups missing ACLs can stop working for normal applications right after cutover. Prevent this by exporting and diffing ACLs before and after the move, and testing each application identity against its topics and groups, rather than relying on the setting to hide gaps.

Unverified offsets and topic configuration

Consumer group offsets and topic settings such as partition counts, retention and min.insync.replicas can drift during a move or be left at defaults. Consumers may reprocess or skip data without any error being raised. Prevent this by capturing topic configurations and committed offsets before cutover, comparing them afterward, and checking consumer lag and throughput for each critical group during the first production window.

How the path differs by starting point

Clusters that still store metadata in ZooKeeper have the longest path. Kafka 4.0 supports only KRaft mode, so the full metadata migration to KRaft must finish before any upgrade to 4.x.

Plan the metadata migration: Follow the upgrade notes for the 3.x release that supports the move to KRaft, and treat that step as a prerequisite.
Confirm the intermediate release: Check the exact version and rolling steps the notes require before you touch broker binaries.
Size the controller quorum: Decide how many KRaft controllers you need and whether they run apart from brokers.
Verify before 4.x: Confirm the cluster runs in KRaft mode and is healthy before any broker moves to 4.x.
Check Java and clients: Confirm Java 17 for brokers and Connect, and Java 11 or later for clients and Streams applications.

What you need to provide

The migration can only be planned as well as the inputs you give us. Before discovery starts, you need to supply a current inventory of brokers, topics, consumer groups and client applications, exported from your cluster and client tooling, along with the exact broker version and Java runtime version on every broker, Kafka Connect worker and tool host. Those two facts decide the upgrade path: a ZooKeeper-based cluster must reach KRaft mode before any move to Kafka 4.x, and Kafka 4.x brokers require Java 17.

We also need read access to broker and controller configuration files, server logs and the relevant metadata during discovery, so we can confirm listener, authentication and authorizer settings directly rather than relying on summaries. Please name a change window for each environment and a non-production environment able to hold a copy of the data, so that the upgrade and cutover can be rehearsed against realistic topic and partition layouts before production is touched.

Provide access to the certificate authority, keystores and identity systems used by your SASL or TLS listeners, including Kerberos key distribution, SCRAM credential storage or the OIDC provider behind OAUTHBEARER. Migration changes how principals are presented to the cluster, so these owners must be reachable during testing. Name one owner for each client application, since those owners must test their own clients against the new version. We cannot certify application behavior on their behalf.

Questions buyers ask

Can our own team migrate Kafka to KRaft and Kafka 4 without a partner?

Yes, if your team already runs production Kafka and has a rehearsal environment that is close enough to production to test the upgrade path, client behavior and cutover. Migrating a ZooKeeper-based cluster requires moving to KRaft before any upgrade to 4.x, and every step should be rehearsed before anyone runs it in production. Teams without recent production Kafka experience often benefit from a partner or an independent review of their plan before the first change. The key test is whether someone on your team can explain the rollback path for each step.

What drives the cost and effort of a Kafka KRaft migration?

Most of the cost is people time, plus the infrastructure for a rehearsal environment and any support agreement you choose. Apache Kafka itself has no license fee. The main drivers are the number of clusters, the number of topics and partitions to verify, how many client applications and Kafka Connect connectors need compatibility checks, the security model in place, and whether you start from ZooKeeper mode or an older 3.x cluster. The drivers table on this page lists these in more detail. Ask any provider to price each driver separately in the written scope.

How long does a Kafka migration to KRaft take?

The duration depends on the written scope and your environment, so no single timeframe applies to every cluster. The main variables are your starting version, how many clusters and clients are in scope, how much compatibility testing your applications need, and how many change windows your operations calendar allows. A single cluster with a documented rehearsal and few clients differs greatly from a multi-site estate with many owners. Ask for a plan that lists each phase, its entry and exit criteria, and the change windows it depends on.

Do we need Kafka 3.9.x before upgrading to Kafka 4?

Not always. The Kafka upgrade notes recommend moving to 3.9.x first only for KRaft clusters older than 3.3.x. Clusters already on 3.3.x or later can follow the general upgrade notes for their version. ZooKeeper-based clusters are different: they must migrate to KRaft before upgrading to any 4.x release, because Kafka 4.0 supports only KRaft mode. Confirm your exact starting version and the current upgrade notes before planning, since the list of supported releases changes over time.

Is there a paid edition of Apache Kafka to buy?

No. Apache Kafka is released under the Apache License 2.0, and the Apache Software Foundation project offers no paid edition of its own. Your cost is therefore people, infrastructure for brokers, controllers, storage and network, and any support agreement you choose to buy from a vendor or partner. Support agreements are optional and priced by their providers, so compare what each one covers, including upgrade help, bugfix timing and incident response. Commercial distributions and managed services are separate products and should be compared on their own terms.

What does a fixed-scope Kafka migration quote cover?

A fixed-scope quote follows from a written scope that names the clusters, clients, connectors and environments in the migration, the deliverables for each phase, and the acceptance checks. DeployOpen agrees scope, deliverables and timelines in writing before work starts. From your side, the work needs an inventory of brokers, topics, clients and their owners, agreed change windows, and test access to a rehearsal environment and to the clients that must be verified. Ask for the stated assumptions, the items excluded from scope, and the process for changes that surface during the work.

How DeployOpen delivers this migration

DeployOpen plans, implements, tests, documents and hands over Apache Kafka migrations to KRaft and Kafka 4 inside your own infrastructure, from the upgrade path through cutover. Engagements are fixed-scope, and the scope is agreed in writing before work starts.

How we scope

Inventory current brokers, clients, Connect workers and ZooKeeper dependencies before agreeing any migration path.
Confirm the starting version and whether the cluster is ZooKeeper-based, which sets the upgrade path to Kafka 4.
Agree the scope in writing as a fixed-scope engagement, with deliverables and timelines written down before work starts.

How we build

Stage the move through supported releases, with 3.9.x as the recommended step for KRaft clusters older than 3.3.x.
Verify Java 17 on brokers and Connect workers, and Java 11 or later for clients and Streams applications.
Test client compatibility and rolling upgrades in a staging cluster that mirrors production topics, ACLs and listeners.

How we hand over

Document the cutover runbook, rollback points, and checks for brokers, controllers, clients and Connect.
Record the KRaft controller layout, listener security and ACL state as built, so your team can reproduce it.
Walk your operators through the cutover results and remaining tasks, including ZooKeeper decommissioning.

How we support

Keep the cluster on a supported release after cutover, since the project lists 4.3.1, 4.2.2 and 4.1.2 as supported.
Plan later upgrades within the rolling window the release plan supports, from the last three releases.
Offer ongoing support and upgrades as a retainer, scoped separately from the migration engagement.

Apache Kafka migration to KRaft and Kafka 4