What upgrades and support involve for Kafka
A self-hosted Kafka cluster needs planned upgrades because the project supports only the releases listed on its downloads page. At the time of writing, those are 4.3.1, 4.2.2 and 4.1.2. The project issues bugfix releases for those versions only, and earlier versions are archived. Once a release leaves that list, your brokers keep running, but no upstream fixes will reach them, so every security or stability fix becomes your team's job to backport or your problem to accept.
The project aims for three releases a year, one every four months, so the supported set moves steadily. The Kafka time-based release plan says rolling upgrades are supported from each of the last three releases to the latest version. A cluster that has fallen further behind needs an intermediate step, and some paths have prerequisites. Before you plan a move, read the upgrade notes for your exact source and target versions and confirm the sequence in a non-production copy of your cluster.
Four triggers usually start an upgrade: a release leaves support, a security fix lands, a client or connector needs a newer Java or Kafka version, or a broker hardware refresh is due. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, and clients and Kafka Streams applications require at least Java 11. Keep a current inventory of broker, Connect, tool, client and Streams versions and their Java runtimes so you can see which trigger applies to each component.
Many teams keep routine patching in-house and bring in a partner for the first major upgrade, compatibility testing and an agreed support window. Ask any partner to confirm which versions the support covers, how client and connector compatibility is tested before cutover, who approves the rollback plan, and how quickly issues are answered. Those answers belong in the written scope before work starts.
What the work includes
The work covers six components, from the version inventory through to verified production clusters. The support agreement covers named clusters, response expectations agreed in writing, patch review and agreed upgrade windows, and every deliverable is written into the fixed-scope agreement before work starts.
Version inventory and per-cluster plan
Record each cluster's current Kafka version, its mode (KRaft or ZooKeeper) and every client, Streams application, Connect worker and connector that depends on it. Compare each version with the supported releases on the Kafka downloads page, which currently list 4.3.1, 4.2.2 and 4.1.2. Each cluster then gets its own plan with a target version and an upgrade path, since rolling upgrades are supported from the last three releases to the latest.
Compatibility matrix
The matrix maps brokers, KRaft controllers, Java runtimes, clients, Kafka Streams applications, Kafka Connect workers and connectors against the target release. Since Kafka 4.0, brokers, Connect and tools require Java 17, while clients and Streams applications need at least Java 11. Each connector plugin is confirmed with its maintainer before the upgrade window is set.
Security configuration review
Confirm TLS on every listener that needs it, the SASL mechanisms used by brokers, clients and tools, and the ACLs held in the StandardAuthorizer metadata log. Kafka security is optional, so mixed secured and unsecured listeners often surface during inventory. Check that allow.everyone.if.no.acl.found is not set to true and that super user membership is current before and after the upgrade.
Non-production rehearsal and runbook
The upgrade runs first in a non-production environment built from production configuration, topics, ACLs and representative client traffic. The rehearsal records timings, failures and the exact commands used, and issues such as client or connector incompatibilities are fixed before any production broker changes. The validated steps become the runbook the operations team follows during the production window.
Production rolling upgrade and rollback plan
Controllers and brokers are upgraded one at a time, with partition leadership and under-replicated partitions checked between each step. The rollback plan states, for each step, whether it can be reversed and what state must be restored, based on the Kafka upgrade notes for the target release. Upgrade windows are agreed in advance with application owners.
Post-upgrade verification
After each cluster reaches the target version, checks confirm that all brokers and controllers are in the expected state, that producers and consumers work across every secured listener and that Connect workers and connectors are running. Monitoring thresholds and alerts are reviewed against the new version, and results are recorded as evidence for the client and any later audit.
What drives cost and duration
This table lists the factors that change the size of a Kafka upgrade, why each one matters and what to do to keep its effect under control.
| Driver | Why it matters | How to keep it down |
|---|---|---|
| Clusters and environments | Each cluster and environment is a separate upgrade, test and change event with its own rollback plan. | Standardize versions and configuration across environments, and consolidate clusters where topics can share one safely. |
| Current version | Clusters older than 3.3.x need an intermediate step to 3.9.x before 4.x, per the upgrade notes. Rolling upgrades run from the last three releases. | Confirm the running version first, then plan each hop in order from the version you run today. |
| ZooKeeper-based clusters | Kafka 4.0 removed ZooKeeper mode, so these clusters must migrate to KRaft before any 4.x upgrade. | Inventory every cluster's metadata mode early. The sibling migration topic covers the cutover steps. |
| Clients, Streams apps and connectors | Each client, Kafka Streams application and connector must be checked against the target release and its Java level. | Keep an inventory with named owners and current versions, and test each one against the target brokers before cutover. |
| Custom authorizer or authentication plugins | Plugins built for an older release may need rework or retesting before each upgrade. | Record each plugin's build target and dependencies, and keep automated tests that run on every release candidate. |
| Security model in place | TLS on each listener, SASL mechanisms and StandardAuthorizer ACLs all add settings that must be retested after an upgrade. | Document the listener map and ACL model, and confirm the allow.everyone.if.no.acl.found value before upgrading. |
| Java 17 for brokers and Connect | Since Kafka 4.0, brokers, Connect and tools require Java 17. Clients and Streams need at least Java 11. | Check the JVM version on every broker, Connect worker and admin host, including container images, before scheduling work. |
| Availability requirement and change windows | Rolling upgrades keep clusters serving traffic, but replication settings and sequencing decide whether each step stays safe. | Confirm replication and min.insync.replicas settings with service owners, and agree change windows and rollback criteria in writing. |
| Audit evidence | Assessors expect records of versions, approvals, test results and configuration baselines for every change. | Keep versioned configuration and change tickets, and ask the provider which evidence the agreement covers. |
Project phases and outputs
The sequence moves a Kafka cluster from an inventory of what runs today to a verified, documented upgrade and a handover your team can operate.
Inventory the current cluster and its clients
We record the Kafka version, Java runtimes on brokers, Connect workers and client applications, client libraries, connectors, plugins, authorizers, listeners and configuration files. You receive a dependency inventory with each item's owner and version, and you confirm it against your own records before any compatibility work starts.
Assess compatibility with the target release
We check the inventory against the target Kafka release and the supported rolling upgrade path, including any ZooKeeper-to-KRaft migration or a stop at an intermediate version. You receive a compatibility matrix listing blockers, required changes and the upgrade path, which you review and approve before planning begins.
Design the upgrade plan and rollback
We write the broker-by-broker sequence, the client and connector upgrade order, the go and no-go criteria, and the rollback conditions with the steps to reach them. You receive a written upgrade plan and rollback design that your change board can sign off on.
Rehearse the upgrade in non-production
We reproduce your production workload in a non-production environment and run the full upgrade, including a controller handling check and a rollback drill. You receive a rehearsal report with observed issues, timings from the rehearsal, fixes applied and a confirmed plan that matches what happened.
Execute the production rolling upgrade
We upgrade brokers one at a time, watching under-replicated partitions, controller election and client lag between each step, and we pause at each checkpoint defined in the plan. You receive a change log for every broker and confirmation that the cluster reached the target version with all partitions in sync.
Verify the upgraded cluster
We confirm producers and consumers, Connect tasks, replication, authentication and ACLs, and monitoring alerts behave as expected on the new version. You receive a verification checklist with results for each item and a list of any open follow-up actions with owners.
Hand over the runbook and onboard support
We deliver the runbook covering the upgrade procedure, rollback, routine checks and escalation, then onboard your team to the support agreement if you choose one. You receive the signed runbook, a recorded walkthrough, and a documented support scope showing what the agreement covers and how requests are raised.
What stays in-house and what goes to a partner
This table shows which Kafka upgrade and support activities most teams keep internally, which they often hand to a partner, and the skills each one needs, so you can plan coverage before choosing a model.
| Activity | Usually kept in-house | Usually outsourced | Skills required |
|---|---|---|---|
| Client compatibility confirmation | Application owners confirm producers, consumers and Streams apps work with the target release. | A partner may run a compatibility test matrix across client libraries if it is in the written scope. | Client library knowledge, consumer group behavior, serializer handling and the application's own logic. |
| Routine patch upgrades | Platform engineers apply patch releases in rolling order during agreed windows. | A partner runs the upgrade runbook when the internal team lacks capacity or time. | Rolling restart procedure, KRaft controller health checks, TLS and SASL configuration review. |
| First major or cross-line upgrade | Internal owners set scope, risk tolerance and the go or no-go decision. | A partner plans the path, rehearses it in staging and executes the cutover under agreed controls. | Kafka internals, KRaft controller operations, JVM runtime management and Java 17 for brokers and Connect. |
| Connect and connector testing | Data owners validate business flows and data correctness after each change. | A partner tests connector versions, error handling and plugin compatibility before rollout. | Kafka Connect distributed mode, connector configuration, JVM tuning and source and sink semantics. |
| Authorizer and authentication plugin updates | The security team approves ACL, principal and credential changes. | A partner rebuilds and retests custom authorizer or SASL plugins against the new release. | Authorizer and SASL plugin interfaces, TLS and SASL configuration, ACL model in KRaft metadata. |
| Monitoring and alert tuning | The platform team owns dashboards, alert thresholds and on-call routing. | A partner sets up metrics pipelines and tunes alerts during an agreed engagement. | Broker and controller metrics, JMX exposure, interpreting lag and under-replicated partitions. |
| Incident response on the cluster | On-call engineers handle first response, containment and internal communication. | A partner provides retainer support for escalations and root cause analysis. | Controller election behavior, partition leadership, log segment recovery and Kafka internals. |
| Change approval | The internal change board approves timing, scope and rollback plans. | A partner prepares change requests, rollback steps and evidence, while the client grants approval. | Change management practice, rollback design and handling evidence for compliance assessments. |
What usually goes wrong
Most Kafka upgrade problems come from decisions made before the first broker restarts. Each card names the step that prevents the failure.
Skipping the required upgrade path
KRaft clusters older than 3.3.x should move to 3.9.x before any 4.x upgrade, and ZooKeeper-based clusters must migrate to KRaft before 4.x. Prevent this by recording the version and mode of every cluster, planning each hop, and confirming that rolling upgrades are supported from the starting release to the target.
Java runtime not confirmed everywhere
Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications need at least Java 11. Inventory the JVM on every broker, Connect worker and application host before the first restart, because a node left on an older runtime will fail after the upgrade.
Clients and connectors built on older APIs
Client libraries, Kafka Streams code, custom serializers and connector plugins compiled against an older API can break after a broker or Connect upgrade. Build a compatibility matrix for each application and plugin, rebuild plugins against the target release, and test them in a staging cluster that matches production versions.
Assuming a release is still supported
The downloads page lists only the supported releases, and earlier versions are archived. A cluster can keep running long after its release stops receiving bugfixes. Compare each cluster's version against the current list, set an upgrade date before the release leaves it, and confirm which release line any quote assumes.
Settings changed without a rollback decision
Changing the inter-broker protocol or metadata version can make a return to the earlier release difficult, so record a go or no-go decision for each step. Also verify that allow.everyone.if.no.acl.found stays false after every restart, because an upgrade can silently open resources that have no ACLs.
Support agreement that omits applications
A support agreement covers only what the partner can reach. If the partner cannot access your brokers, Connect workers, client code or monitoring for a given application, it cannot diagnose failures there. Agree in writing which systems, access paths and change windows are in scope before the upgrade starts.
How the work differs by starting point
A cluster already on a supported 4.x release has the shortest path. The work is planned rolling upgrades and patching within the supported line, with release notes for each target version reviewed before the change window.
- Target selection: confirm the current supported releases on the Kafka downloads page and choose the latest patch in your line.
- Rolling order: restart brokers one at a time, check for under-replicated partitions between restarts, then move Connect workers and tools.
- Client check: confirm client and Kafka Streams applications meet the Java 11 minimum.
- Client provides: an inventory of broker, Connect and client versions, and a test cluster that mirrors production topic configuration.
A cluster on a 3.x line needs intermediate steps. For KRaft clusters older than 3.3.x, the upgrade notes recommend moving to 3.9.x before any 4.x target, so the plan includes more upgrade hops and a longer compatibility review.
- Upgrade hops: map each intermediate release, its rolling upgrade support, and its required configuration changes.
- Compatibility review: check removed or deprecated settings, client libraries and Connect connectors at each hop.
- Java runtime: brokers, Connect and tools require Java 17 from 4.0, so confirm the JVM on every host before the first step.
- Client provides: the cluster's version history, a list of configuration overrides, and named owners for each client application.
A ZooKeeper-based cluster cannot move to Kafka 4.x as it stands. Kafka 4.0 supports only KRaft mode and ZooKeeper mode has been removed, so migration to KRaft comes first. The migration steps are covered in the sibling topic on moving to KRaft and Kafka 4.
- Sequence: complete the KRaft migration on the current release, then plan the 4.x upgrade as a separate change.
- Controllers: size the controller quorum and plan the metadata cutover before any broker change.
- Tooling: identify scripts, monitoring and automation that reference ZooKeeper and schedule their rewrite before cutover.
- Client provides: ZooKeeper ensemble details, owners of tooling that touches it, and a maintenance window that covers both changes.
In an air-gapped or restricted network, the work depends on artifacts carried across a controlled transfer and validated offline. Each release, dependency and container image must be mirrored, verified and tested without internet access.
- Mirroring: move the Kafka release archive, Java runtime, Connect plugins and any container images through your approved transfer process.
- Verification: check checksums and signatures against the values published by the project before import.
- Offline testing: run upgrade rehearsals in an isolated staging network, because build and package steps may depend on internal sources.
- Client provides: the transfer and approval process, an internal artifact repository, and a staging network that matches production.
Clusters with custom authorizer or authentication plugins, serializers, Streams applications or large Connect estates carry the most regression risk. Each custom component raises its own compatibility question at every version step.
- Plugin inventory: list each custom authorizer, serializer, transform and connector with its build source and owner.
- Regression suite: run produce, consume, ACL and connector tests against each target version before production.
- Connect compatibility: confirm each connector plugin supports the target Kafka version and runs on Java 17 workers.
- Client provides: source code or build artifacts for custom components, vendor support status for connectors, and test data that exercises error paths.
What the client needs to provide
The upgrade cannot be planned well until the client supplies a complete picture of its Kafka estate. Start with an inventory of every cluster, listing the current Kafka version, the deployment method (for example, KRaft or a legacy ZooKeeper-based cluster), broker and controller counts, and the Java runtime on each broker and Kafka Connect worker. This inventory determines whether a direct upgrade path exists or whether an intermediate step, such as moving to a 3.9.x release first, is required before a 4.x target.
Provide access to a non-production environment that mirrors production topology, including controller quorum size, listener configuration, rack layout and the same authentication and authorization settings. Also name the client and connector owners for each topic and connector, with contact details and an escalation path. Those owners must confirm that their applications and connectors are compatible with the target release and the Java versions it requires, since client compatibility is often the step that delays a rollout.
Share the security configuration as it runs today: which listeners use SSL or SASL, which SASL mechanisms are enabled, how ACLs are stored and whether allow.everyone.if.no.acl.found is set, and which principals hold super user rights. Also share the change calendar and the maintenance windows that operations has approved. Confirm the backup and recovery arrangements for the cluster, including when the last restore test was run and who holds the procedure. Do not assume a backup is usable until a restore has been exercised.
Name one decision owner with authority to approve go or no-go at each stage and to call a rollback. The client remains accountable for its data, for the business impact of any downtime, and for approving every production change. DeployOpen plans, executes and documents the upgrade, but the client's owner signs off the production steps. Confirm this decision owner, and their deputy, before the schedule is fixed in the written scope.
Frequently asked questions
How often does Apache Kafka need an upgrade, and why does the cadence matter?
The Kafka project aims for three releases a year, one every four months, so a cluster left on one version falls out of the supported set within a predictable window. Staying close to the supported line matters because the project makes bugfix releases only for supported releases, so an older version stops receiving fixes for defects. Agree an upgrade cycle with your team that follows this cadence, and confirm which release line you will standardize on. The longer you wait, the more releases you must pass through, because rolling upgrades are supported only from the last three releases to the latest version.
Which Apache Kafka releases still receive bugfixes from the project?
The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and all earlier versions are archived. The project makes bugfix releases as needed, and only for supported releases, so an archived version will not receive fixes from the project. If your cluster runs an archived version, moving to a supported line is the first step before other work. Check the downloads page on the day you plan the upgrade, because the supported list changes as new releases ship. There is no paid edition from the project itself, since Apache Kafka is released under the Apache License 2.0.
Can you upgrade Apache Kafka with a rolling upgrade and no downtime?
Yes, if your starting point and settings allow it. The project supports rolling upgrades from each of the last three releases to the latest version, so a cluster outside that window needs intermediate steps first. ZooKeeper-based clusters must migrate to KRaft before moving to 4.x, and KRaft clusters older than 3.3.x should move to 3.9.x before 4.x. Replication factor, min.insync.replicas and client retry behavior decide whether producers see errors while brokers restart. Rehearse the exact path in a non-production copy and confirm each step against the upgrade notes for your target version.
Do Kafka clients and connectors need upgrading along with the brokers?
Not always, but you must check. Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications require at least Java 11. A runtime change can therefore reach applications and connector workers that sit outside the platform team's control. Inventory every producer, consumer, Streams application and connector, and record its Java runtime and Kafka client library version before you move brokers. Custom connectors and authorizer or authentication plugins need their own test run, because you must rebuild and retest them against the target release.
What does an Apache Kafka support agreement cover, and what does it exclude?
A support agreement is defined by its written scope, so coverage depends on what you contract. Typical components include upgrade planning and execution for supported releases, incident support, configuration review and handover documentation, but each item must appear in the agreed scope. Common exclusions are work on your own applications, data pipelines and infrastructure outside the Kafka cluster. DeployOpen does not issue audit reports or certifications, so ask for that separately from independent assessors. Ongoing support is available as a retainer, while most engagements are fixed-scope and agreed in writing before work starts.
Can our own team run Kafka upgrades, and what drives the effort and cost?
A capable platform team can run upgrades itself if it has time, a test environment and clear runbooks. Effort rises with the number of clusters, the size of the client inventory, the security configuration, and the rollback plan you require. One-time costs include planning, the test rehearsal and client fixes. Recurring costs come from keeping pace with the cadence and running a retainer if you want a partner on call. Ask any partner for a quote that lists each phase, the environments it covers, the client and connector work included, and what is excluded.
How DeployOpen delivers Kafka upgrades and support
DeployOpen plans, upgrades and supports self-hosted Apache Kafka inside your own infrastructure, starting from a written scope and a rehearsed upgrade path. Engagements are fixed-scope, and ongoing support is available as a retainer.
How we scope
- The written scope names the current version, target release, clusters, listeners and clients in the upgrade.
- We check each cluster's starting point against the supported path, including ZooKeeper-based clusters that must move to KRaft first.
- Engagements are fixed-scope, and the scope, deliverables and timelines are agreed in writing before work starts.
How we build
- We build a staging copy of the production cluster so each upgrade step can be rehearsed before production.
- Rolling upgrades follow the documented path, with client, Connect and Streams compatibility checked against the target release.
- Java runtimes for brokers, Connect and tools are verified on every host before the first broker restarts.
How we hand over
- Handover includes an upgrade runbook, rollback steps and a record of each version, configuration change and validation result.
- After each upgrade we re-check TLS listeners, SASL mechanisms and StandardAuthorizer ACL behavior against the agreed baseline.
- Your team receives documentation of the cluster as it runs, so operators can maintain it after the engagement ends.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing.
- Plan each upgrade around the cadence: Kafka aims for three releases a year and bugfixes only for supported releases.
- Confirm in the quote which Kafka releases, client versions and connectors the support agreement covers.
