Why Kafka Connect and Streams upgrades need their own plan
A Kafka 4.x broker upgrade is rarely the only change in motion. The Connect workers, the connector plugins they load and the Kafka Streams applications that read and write the same topics all have their own runtime, dependency and compatibility requirements, and they need their own plan. The supported path for the brokers is rolling upgrades from each of the last three releases to the latest version, according to the Kafka time based release plan. That path tells you when the cluster can move, but it does not tell you when your Connect and Streams components can move, so treat them as a separate track with their own sign-off.
This page covers the components around the brokers: the Java runtime for Connect workers, connector plugin compatibility and plugin discovery, the Streams rebalance protocol, and rolling Connect clusters without losing tasks. It does not repeat the broker upgrade steps, the metadata version finalization or client inventory. Those are covered on the sibling pages for the 4.x rolling upgrade and for checking client compatibility. Before you change anything on the Connect side, confirm the broker version you are moving to, because the Connect and Streams requirements you must meet are tied to the Kafka release you run, not to your own deployment schedule. Read the Kafka 4.3 upgrade notes for the exact requirements of your target release.
Two items need attention early. First, the Java requirement for Connect workers differs from the one for client applications, so a Streams application may already meet the bar while a Connect worker image does not. Second, Connect and Streams changes can alter runtime behavior even when the brokers stay compatible, so each track needs its own test window, rollback point and owner. Ask your team to inventory every Connect cluster, connector plugin and Streams application with its current runtime version before the broker change is scheduled. That inventory determines how long the Connect and Streams work will take, and it is the first input any partner or internal team will need to scope the work.
What changes for Connect and Streams in the 4.x line
This table lists the four changes that most affect Kafka Connect workers and Kafka Streams applications when moving through the 4.x line. Read each row as a question to answer for your own fleet: which runtime, which plugins and which Streams protocol you run today. The main takeaway is that the Java runtime and plugin discovery changes need checking before any rolling restart.
| Change | Impact | Action needed |
|---|---|---|
| Java 17 for Connect workers and tools (since 4.0) | Connect workers and Kafka tools cannot start on an older JVM after the move to 4.x, so a worker image or host still on an earlier Java version will fail to run the new release. | Confirm the JVM on every Connect worker host or image and on any tool hosts, then upgrade the runtime before changing the Kafka version. Source: Kafka upgrade notes. |
| Java 11 minimum for clients and Streams applications (since 4.0) | Kafka Streams applications and other clients need at least Java 11. Applications built for an older runtime must be rebuilt or redeployed on a supported JVM before they can use 4.x client libraries. | Inventory Streams application runtimes and build targets, and schedule rebuilds for any running on an older JVM. Source: Kafka upgrade notes. |
| Connect plugin discovery defaults to hybrid_warn (from 3.6) | The default mode scans plugins and logs warnings for those it finds problematic, which gives you a signal about incompatible connectors without stopping workers. | Review worker logs for plugin discovery warnings and list each connector plugin that triggers one. Source: Kafka Connect user guide. |
| Connect service_load discovery mode trade-off | Service loading speeds worker startup but assumes every plugin on the path is compatible, so one incompatible plugin can prevent a worker from starting cleanly. | Before switching to service_load, verify every plugin, and use bin/connect-plugin-path.sh to migrate any incompatible plugins. Source: Kafka Connect user guide. |
| Streams rebalance protocol KIP-1071 (production ready in 4.2) | KIP-1071 moved from early access to production ready in Kafka 4.2, so applications can plan to use the new rebalance protocol rather than treating it as a preview. | Decide whether each Streams application will adopt the protocol, and test rebalance behavior in a staging cluster first. Source: Kafka upgrade notes. |
How plugin discovery modes affect connector upgrades
Plugin discovery mode decides whether a Connect worker can start cleanly after you upgrade, so it belongs in the upgrade plan before any worker is restarted. Kafka Connect's default discovery mode is hybrid_warn from 3.6 onward. In this mode the worker scans the plugin path with the older reflection-based approach and logs warnings for plugins that are not compatible with the faster method, so an incompatible connector jar is visible but does not stop the worker. The service_load mode uses Java's ServiceLoader metadata, which speeds up worker startup because the worker does not have to scan every class. The trade-off is strict: every plugin on the path must provide that metadata, and a single incompatible plugin is a problem you must resolve before switching modes.
Start with a plugin inventory. For each worker, list every directory and jar on plugins.path, the connector or converter class it provides, its version, and its source (vendor build, open-source release or internal build). Many upgrade failures come from forgotten plugins copied onto a worker months earlier, or from the same connector name existing in two versions on the path. Once you know which plugins are incompatible, use bin/connect-plugin-path.sh to list and migrate them so their metadata is generated. Run this against a staging copy of the plugin path first, then confirm that the worker starts in the target mode and that every connector class still loads. Confirm the exact flags and behavior for your Kafka release in the Kafka Connect user guide before running it in production.
Decide the mode per cluster, not per upgrade. If your inventory is clean, service_load gives faster restarts during rolling Connect upgrades, which matters because each restarted worker must rejoin and rebalance its tasks. If any plugin cannot be migrated or replaced, keep hybrid_warn and treat its warnings as a tracked item rather than noise. Record the chosen mode and the plugin versions in your change record, and repeat the inventory after every plugin installation, since a new jar that lacks the metadata will only surface at the next worker start.
What to prepare before touching Connect or Streams
Connect workers and Streams applications fail in different ways during an upgrade, so the preparation differs for each. The cards below list the artifacts to produce before any node is restarted, from plugin and runtime inventories to backups and a test environment on the target broker version.
Connector plugin inventory
List every connector plugin on each Connect worker's plugin path, with its artifact name, exact version, vendor and source. Record which connectors are used by which connector instances. Note any plugin built against an older Connect API or without a published version for your target release, because these are the plugins most likely to fail discovery or class loading after the upgrade. Ask each plugin vendor to confirm compatibility with the Kafka version you plan to run.
JVM and runtime inventory
Record the Java runtime version on every Connect worker and every host that runs a Kafka Streams application, along with the JVM flags and container base images in use. Kafka 4.0 and later require Java 17 for Connect, while Streams applications and clients need at least Java 11. Identify any host still on an older runtime and plan its upgrade as a separate change so a failure can be traced to one cause.
Connect configuration and offset backup
Export worker configuration files, connector configurations and the internal offset, config and status topics for each Connect cluster. Store the copies outside the cluster so they survive a failed rollout. Confirm the names of the internal topics from the worker configuration, since restoring a Connect group requires them. Keep a record of current connector states so you can confirm the same tasks are running after each worker restarts.
Target-version test environment
Build a staging Connect cluster and a staging Streams application on the target broker version, with the same plugins, runtime and configuration as production. Replay representative traffic and run each connector through a pause, restart and resume cycle. Compare task assignment, lag and error rates with the current production baseline before approving the production change window.
Streams application and rebalance settings list
List every Kafka Streams application with its owner, deployment host, state store locations and current group.protocol setting. Kafka 4.2 made the KIP-1071 rebalance protocol production ready, so record whether each application uses the classic protocol and whether the upgrade will change that. Note any custom StreamsConfig values that control rebalancing, so the team can decide which applications move first.
The order to upgrade Connect workers and Streams applications
This sequence moves Kafka Connect workers and Kafka Streams applications to their new Java and protocol baselines in an order that keeps connectors running and gives each change a checkpoint you can reverse. It assumes the Kafka brokers are already on a supported 4.x release and finalized.
Confirm the broker version and supported upgrade path
The platform engineer confirms which Kafka release the brokers run, checks that it is on the supported list, and verifies that the cluster is on KRaft. Connect and Streams upgrades should not start while brokers are mid-upgrade, because the broker metadata version must be finalized first. Finalizing the broker release is a dependency for the later steps, and your broker runbook covers it. The check is a written record of the broker version, the controller mode and a confirmed finalization status. Rollback point: none is needed yet, because no client or worker has changed.
Inventory Connect workers, connectors and plugins
The Connect owner lists every worker cluster, its Kafka Connect version, every connector and every plugin path in use, including the plugin discovery mode each worker sets. Connectors from third parties often pin library versions, so confirm with each plugin vendor that a build exists for the target Connect release and Java 17. The output is an inventory with one owner per connector and an open question list for vendors. Rollback point: the inventory itself is read-only, so nothing needs reversing.
Upgrade Java to 17 on a test Connect worker
Since Kafka 4.0, Connect requires Java 17, so the platform team installs Java 17 on a single test worker in a non-production cluster and starts it with the target Kafka Connect release. Confirm that startup logs show no class loading failures and that the worker reports its connectors as running. Keep the previous JVM installed and the previous worker image or package available so the change can be reversed. The check is a clean startup and healthy connector status on the test worker. Rollback point: restore the previous JVM and worker package on that node.
Validate plugins and resolve discovery warnings in test
The Connect owner starts the test cluster with the plugin discovery mode in use and reads the logs for scanning warnings from each plugin. The Connect user guide documents that the service_load mode starts faster but requires every plugin to be compatible, and that bin/connect-plugin-path.sh can migrate incompatible plugins. Replace or migrate plugins that fail, then rerun discovery until no plugin reports a warning. The check is a log review with zero unresolved plugin warnings and a passing connector smoke test for each connector type. Rollback point: keep the previous plugin directories so you can point the worker back to them.
Roll Connect workers one at a time
In production, the platform team stops one worker, applies the Java 17 and Connect package change, and starts it again before touching the next worker. After each restart, confirm that the rebalance has completed and that tasks are distributed across all workers, not concentrated on the ones still running the old build. Pause any connector whose tasks cannot move to a mixed-version group, and confirm the sink lag and source offsets recover. The check is task count per worker matching the expected distribution and no connector stuck in a failed state. Rollback point: stop the upgraded worker, restore the previous package, and let the remaining workers take its tasks.
Upgrade Streams applications to Java 11 or later
The application team rebuilds each Kafka Streams application against the target client library and runs it on Java 11 or later, which the Kafka upgrade notes list as the minimum for Streams applications. Build the application image or artifact with a pinned Java version and record it in the release notes. Deploy to staging first and confirm the application starts, processes records and commits offsets as before. The check is a staging run that matches the expected output and offset progress. Rollback point: redeploy the previous artifact, which still runs on the previous client library.
Evaluate the rebalance protocol change in staging
Kafka 4.2 moved the Streams rebalance protocol from KIP-1071 from early access to production ready, so the application team tests whether enabling it fits the topology and state stores. Run a staging rebalance by adding and removing an instance, and compare task assignment, standby handling and recovery time against the current protocol. Confirm the setting with the release documentation before enabling it, because it changes how the group coordinates. The check is a written decision per application: adopt the new protocol, or keep the current one until a later window. Rollback point: switch the configuration back to the current protocol and redeploy.
Cut over and monitor the upgraded components
The operations team schedules the production Streams deployment after the Connect workers are stable, then rolls application instances in small groups while watching consumer lag, rebalance frequency and error logs. Keep the previous artifacts available until the new build has run through its normal processing cycle without rebalance storms or failed tasks. Record the final Connect, Java and Streams versions in the cluster inventory. The check is stable lag and no unexplained rebalances after the last instance is replaced. Rollback point: redeploy the previous artifacts group by group until the cluster is stable again.
Rolling Connect clusters without losing tasks
The core rule for a rolling Kafka Connect upgrade is to restart one worker at a time and confirm that its tasks have moved to the remaining workers before touching the next one. Connect distributes connectors and tasks across workers in a group, so a worker that stops takes its share of tasks with it until the group rebalances. Stopping several workers together can leave connectors paused or unassigned for longer than you planned. Before the first restart, record the connector and task inventory from the Connect REST API, including each connector's state and the worker each task runs on, so you have a baseline to compare against after every step. Confirm with your own environment's configuration that every worker in the cluster runs the same plugin.path and the same connector plugin versions, because a worker started with a different plugin set can change where tasks land.
After each restart, verify three things before moving on. First, check that the restarted worker has rejoined the group and that its former tasks are running on other workers. Second, check that every connector and task reports RUNNING, not FAILED, PAUSED or UNASSIGNED. Third, check offsets: for source connectors, confirm that offsets are still advancing in the offset topic named by offset.storage.topic, and for sink connectors, confirm that the consumer group for the connector shows committed offsets moving forward without a lag spike. A task that restarts from an older offset can produce duplicate records, so compare offset positions against the baseline, not just the status field. Watch the worker logs for repeated task restarts, which often indicate a plugin class that is incompatible with the new worker runtime and needs attention before you continue.
Choosing when to adopt the Streams rebalance protocol
Keeping the classic protocol is the lowest-change option during a broker and Kafka Streams upgrade. Your applications keep the rebalance behavior they already run under, so any regression you see can be traced to the library or broker upgrade rather than to a protocol switch. This suits teams whose topologies are stable and whose lag and rebalance metrics are already healthy, and teams that have no pressing need for the new protocol in this release window.
- Fewer moving parts: Only the Java runtime, client libraries and brokers change, so failures are easier to attribute.
- Known behavior: Your existing rebalance timings and standby task settings stay as they are.
- Deferred benefit: You will still need to plan the switch later, so record it as a backlog item with an owner.
- Configuration unchanged: Confirm which group protocol your applications currently use before the upgrade begins.
Before you choose this path, check the Kafka upgrade notes for the exact status of the rebalance protocol in your target release and confirm that the classic protocol remains supported for your version.
Moving staging applications first lets you test the new rebalance protocol under realistic load without exposing customer-facing consumers. The test should mirror production topology: the same number of instances, partition counts, standby replicas and deployment restarts. A staging-only test with one instance will not show how the group behaves when instances join, leave or fail together.
- Representative topology: Use the same partition counts and instance counts as production where you can.
- Failure scenarios: Test rolling restarts, a crashed instance and a scale-out event.
- Metrics to compare: Record rebalance duration, processing lag, task assignment and error rates against the classic baseline.
- Application review: Check for custom partition assignment, state store dependencies or external side effects that behave differently under the new protocol.
- Written exit criteria: Agree on pass and fail thresholds before the test starts.
Check the results against the Kafka upgrade notes and the Streams documentation for the release you run, and confirm any configuration changes that the new protocol requires.
Moving production applications after a soak period means switching only once staging results are stable over a period long enough to cover your normal operating cycle. A soak period should include routine deployments, traffic peaks and at least one unplanned instance failure. The switch is made application by application or group by group, so a problem can be contained and reversed.
- Staged rollout: Switch lower-risk applications before stateful or high-throughput ones.
- Rollback plan: Document how to return a group to the classic protocol and what state, if any, must be preserved.
- Coordinated changes: Avoid combining the protocol switch with a Kafka broker upgrade or a Connect cluster roll in the same window.
- Monitoring during soak: Keep alerts on consumer lag, rebalance frequency and exceptions active throughout.
- Ownership: Name the application owner who signs off each switch.
Confirm the soak length and rollback steps with your own change process and with the upgrade notes for your release.
The right choice depends on topology more than on the release calendar. Applications with many partitions, large state stores, heavy standby replication or custom assignors carry more risk and need more testing. Simple stateless topologies with few instances can often move earlier. Map each application before you decide, so the rollout order follows real risk rather than team preference.
- State size: Large state stores increase the cost of any rebalance mistake.
- Instance count: Groups with many instances exercise more join and leave events.
- Custom logic: Custom assignors, interceptors or external lookups need explicit review.
- Dependencies: Applications that share input topics or downstream systems may need coordinated timing.
- Inventory: Record the Streams version, Java version and group protocol for every application.
Use the inventory to group applications into waves, and confirm the protocol requirements for each wave against the Kafka upgrade notes before scheduling changes.
What makes Connect and Streams upgrades take longer
This table lists the factors that stretch the work of upgrading Connect workers and Streams applications, and the direction each one pushes the effort. Read it as a checklist for sizing the inventory and test work before you agree a plan or quote.
| Driver | Effect on work |
|---|---|
| Number of connector plugins and custom plugins | More plugins mean more JARs to check for Java 17 and API compatibility, more plugin discovery testing, and more connectors to restart and verify per worker |
| Incompatible or unmaintained plugins | Plugins without a maintained release for the target Kafka version add replacement, forking or vendor-approval work, which can dominate the schedule |
| Number of Streams applications and their state stores | Each application needs its own rolling plan, and large state stores lengthen restore time after restarts and raise the test effort for rebalance behavior |
| Java runtime changes on hosts | Moving Connect workers to Java 17 touches OS images, JVM flags, agents and monitoring tools, so every host image must be rebuilt and validated |
| Deployment tooling and configuration management | Ansible, Helm, Strimzi or custom scripts that pin versions or plugin paths must be changed and reviewed before any worker is restarted |
| Size of connector offsets and state to back up | Larger offset topics, connector state and Streams changelogs increase backup time and the rehearsal needed to prove a restore before the change window |
Questions teams ask before upgrading Connect and Streams
What drives the cost and effort of upgrading Kafka Connect and Kafka Streams?
The main driver is how many workers, connectors, plugins and Streams applications you run, and how many of them you have to change. Cost has two parts: the people who inventory and test the estimated work, and the people who change, deploy and verify each component. Connector plugins that are not built for the target Java and Kafka versions take the most time, because each one needs a compatible build, a vendor answer, or a replacement. Streams applications with custom state stores, many repartition topics, or long test cycles also add effort. Ask any partner for a quote that lists each Connect cluster, each connector plugin with its version, each Streams application, the test environments, and the rollback steps as separate line items. Those line items show where the effort sits and which parts recur after the upgrade, such as plugin updates and retained test suites.
How long does a Kafka Connect and Streams upgrade take?
The duration depends mainly on the inventory, so no fixed figure applies across environments. Teams with a short list of Connect workers, well-maintained connector plugins and Streams applications that already run in tests usually finish sooner than teams with many custom plugins and little test coverage. Time is spent on four activities: confirming the Java 17 runtime for Connect workers, checking each plugin against the target release, deciding on the plugin discovery mode, and rolling each cluster in turn. Streams applications need their own sequence, because the rebalance protocol change affects how groups move during deployment. Ask for a schedule built from your inventory and a list of the assumptions behind it, such as how many staging rounds are included and who approves production changes. Use the Kafka upgrade notes to confirm which requirements apply to your target release before you commit to dates.
What does the upgrade need from our own engineering team?
The work needs named owners for each Connect cluster, each connector, each Streams application and the Kafka cluster they depend on. Your team must supply the current configuration files, plugin paths, connector configs and the list of topics each application reads and writes. Someone with authority over production change windows must agree on the rollout order and the maximum acceptable pause for each pipeline. Your team also needs access to staging environments that match production closely enough to reproduce the behavior of connectors and tasks under load. Finally, the people who own downstream consumers must confirm they can handle the restart. Missing any of these inputs usually adds rework, so ask the partner to list them in writing before the work starts.
Can an internal platform team upgrade Connect and Streams without a partner?
An internal team with Kafka operations experience and Java build skills can do the work alone, provided it has time, test coverage and clear ownership of each pipeline. The hardest parts are usually plugin compatibility and application testing rather than the broker commands. Teams that have run Connect in production, kept plugin versions current and have automated tests for their Streams applications typically need less outside help. Teams with many undocumented plugins, forked connectors or no staging environment often benefit from a fixed-scope engagement that inventories the estate, sets up the test path and runs the first rolling changes with their staff. Decide based on your inventory, your staffing for the change window, and whether the same people will also handle the broker upgrade. Confirm each requirement in the release notes with your own team before you decide.
How does rollback work for Kafka Connect and Streams?
Rollback for Connect and Streams is usually a redeployment of the previous build and configuration, not a single switch. For Connect, you keep the previous worker image, plugin directory and configuration so that the cluster can restart on the older Java runtime and plugin set. Before each rollout, record the connector configurations and offsets so you can restore them if a connector behaves differently after the change. For Streams, the rollback plan must cover the application binary, the internal topics it created, and any state stores that the new version changed. Rollback is only safe if the old version still runs against the Kafka cluster in its current metadata state, so check the broker upgrade notes before you rely on it. Ask the partner to write the rollback steps for each component, the data they touch, and the point after which rollback is no longer practical.
What happens to running tasks and connectors during a rolling restart?
In a rolling restart, each Connect worker stops, is replaced and rejoins the group, and the remaining workers take over its connector and task assignments. Tasks that move pause while the new assignment completes, so throughput drops and lag grows for the affected pipelines. Plugin discovery settings and incompatible plugins can delay a worker from rejoining, which is why the inventory should come first. Streams applications follow a similar pattern, but the rebalance protocol change affects how partitions move between instances during the restart. The result depends on partition counts, state store size and how quickly each instance restores state. Plan the restart order so that a single failed worker does not leave a connector unassigned for long. Ask for a restart plan that states the expected pause for each pipeline and the checks that confirm tasks have resumed and offsets are advancing.
Where rollback is possible and where it is not
Rollback for Connect and Streams is governed less by the version you run and more by what the newer code has written and loaded. A Connect worker that has been upgraded may have rewritten plugin state, changed how plugins are discovered, or adopted a newer Java runtime, and none of those changes are undone by simply starting the previous worker image. Before the window opens, list every change you plan to make to the worker: the Kafka version, the Java runtime, the plugin path contents, and the plugin discovery mode. Treat each one as a separate reversal question. Rolling back the Java runtime on a host is usually simpler than reversing a plugin migration that a tool such as bin/connect-plugin-path.sh has already applied, so confirm in a test cluster whether the previous plugin layout still starts cleanly.
Streams applications are harder to roll back because their state lives outside the code. Offsets, changelog topics and repartition topics written by a newer application can be read by an older one only if the state format and the consumer group protocol remain compatible. A change to the rebalance protocol, such as the move of the KIP-1071 protocol to production ready in Kafka 4.2, can make a group that has already joined under the new protocol difficult to return to the old one. Decide in advance whether a rollback means restarting the old binary against the same consumer group, or redeploying from a state snapshot, and test the path you choose. Record the group IDs and state store names so that the rollback runbook names them explicitly.
Broker-side rollback limits are not described in this section and must be confirmed separately. Read the upgrade notes for the exact Kafka versions in your path, paying attention to any statement about metadata version finalization and downgrade, because a Connect or Streams rollback is only as safe as the brokers it talks to. If the notes leave a question open for your combination of versions, ask the vendor or project before the window and document the answer in your change record. Plan the Connect and Streams rollback steps and the broker rollback steps as one sequence, and agree who decides to stop the rollout if a component fails halfway.
How DeployOpen scopes and delivers Connect and Streams upgrades
DeployOpen plans Connect and Streams upgrades as written, fixed-scope work inside your own infrastructure. We start from an inventory of your workers, plugins and applications, then sequence each change so that tasks and state move safely between versions.
How we scope
- Inventory every Connect cluster, its worker Java version, connector plugins and plugin discovery mode before proposing a plan.
- List each Streams application with its Kafka client version and whether it depends on the rebalance protocol change.
- Agree the written scope, deliverables and timeline with you before any change is made to a running cluster.
How we build
- Build a staging copy of the Connect cluster and test each connector plugin against the target Kafka and Java versions.
- Plan the discovery mode change and any plugin migration with bin/connect-plugin-path.sh in a rehearsal before production.
- Test Streams applications under the new rebalance protocol in a pre-production environment, including restart and failover cases.
How we hand over
- Document the rolling Connect procedure, showing how workers are restarted one at a time while tasks are reassigned.
- Provide rollback steps for each Connect and Streams change, stating which steps cannot be reversed once applied.
- Deliver run books, configuration records and test results to your platform and application teams.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the upgrade is complete.
- Review the next Kafka release notes for Connect and Streams changes that affect your deployed plugins and applications.
- Confirm with your team the support boundaries, response expectations and access model before the engagement starts.
