Why the Kafka processor upgrade is a flow-level change, not a version bump
If your NiFi flows still use the versioned ConsumeKafka_2_6 and PublishKafka_2_6 processors, you need to replace them in the flow itself before or during the move to NiFi 2. The 2.6 processors were deprecated in NiFi 1.29.0, and the older ConsumeKafka_2_0 and PublishKafka_2_0 variants were deprecated in 1.21.0. The replacements are the unversioned ConsumeKafka and PublishKafka processors, which take their broker connection from a Kafka3ConnectionService controller service instead of settings on each processor.
This means the change touches more than processor types. Each flow has to be reviewed for the connection service, the Kerberos user service that replaces standalone Kerberos properties (deprecated in 1.21.0), and the record reader and writer settings that the new processors expose. Existing processor properties, relationships and scheduling do not carry over one to one, so each replaced processor needs its configuration rebuilt and checked against the data it handles. Treat it as a flow change with its own test and rollback plan.
Start with an inventory. Confirm which Kafka version your brokers run and check it against the releases listed on the Apache Kafka downloads page, because an unsupported broker version can block the cutover regardless of how the NiFi flows are rebuilt. Then list every flow, process group and parameter context that references the 2.6 or 2.0 processors, since replacements in one place can leave duplicate consumers or producers running elsewhere. Confirm the exact NiFi 1.x minor version in use, because the deprecation dates above apply to specific releases.
What changes in the Kafka processors and what each change requires
This table compares each deprecation or platform change that affects Kafka flows in NiFi 1.x and NiFi 2, with its impact on existing flows and the action each one requires. Read it row by row against your flow inventory, starting with any flow that uses the 2.6 processors or standalone Kerberos properties, since those need changes before cutover.
| Change | Impact on existing flows | Action needed |
|---|---|---|
| ConsumeKafka_2_6 and PublishKafka_2_6 deprecated in 1.29.0 | Flows on these versioned processors keep running on 1.x but will need rework before they can move to NiFi 2 with the same processor types. | Replace each instance with ConsumeKafka or PublishKafka and configure a Kafka3ConnectionService for the connection settings. |
| ConsumeKafka_2_0 and PublishKafka_2_0 deprecated in 1.21.0 | Flows still using the 2.0 variants are two deprecation steps behind, so they carry the most migration work in the inventory. | Map each 2.0 instance to ConsumeKafka or PublishKafka, then retest offsets and record handling before cutover. |
| Standalone Kerberos properties on Kafka processors deprecated in 1.21.0 | Processors that hold Kerberos principal and keytab values directly are using a pattern the project has moved away from. | Create a Kerberos User Service and point the processors at it, then confirm the principal has the access the topic requires. |
| Kafka3ConnectionService as the shared connection point | Broker addresses, security protocol and client settings move out of each processor and into one shared controller service. | Define the connection service once per target cluster, and check which processors reference it after the change. |
| Kafka 4.x requires KRaft mode and Java 17 for brokers | Brokers on ZooKeeper must migrate to KRaft before upgrading to 4.x, and client applications need at least Java 11 for Kafka Streams and clients. | Confirm broker mode and Java version with the Kafka cluster owner before changing NiFi flows that depend on a 4.x target. |
What to inventory and back up before touching a Kafka processor
Before you swap a Kafka processor, you need a complete picture of what runs today, what it connects to and how it authenticates. The cards below cover the artifacts and access points that decide whether a cutover can be rolled back and tested safely.
Every Kafka processor instance
List each ConsumeKafka_2_6 and PublishKafka_2_6 instance across all process groups, including those inside nested groups and disabled or stopped components. For each one, record topic names or patterns, group IDs, offset reset behavior, acknowledgment settings, compression and the controller or standalone connection settings. Export these values from the running flow rather than from memory, because copied configurations often drift from what is deployed. Confirm the full count with the owners of each flow before any change is planned.
Flow definition and parameter contexts
Export the flow definition for every process group that contains a Kafka processor, along with the parameter contexts those groups reference. Kafka settings often sit in parameters such as broker lists or topic names, so an export that omits parameter contexts will not reproduce the running configuration. Store the exports in version control with a tagged baseline so you can compare the old and new flow and restore the previous state if the cutover fails.
Kerberos principals and keytabs
Identify every Kerberos principal and keytab used by Kafka processors, including those set directly on the processor and those held in a Kerberos User Service. Standalone Kerberos properties on Kafka processors were deprecated in 1.21.0 in favor of a Kerberos User Service, so you may find both patterns in one estate. Record which keytab files live on which nodes, their owners and file permissions, and their expiry or rotation schedule. Ask your Kafka and identity teams which principals the brokers expect for each client.
Record reader and writer settings
Capture the record reader and record writer services attached to each ConsumeKafka and PublishKafka instance, including schema access strategy, schema text or registry references, date formats and null handling. The record strategies in the new processors produce results that depend on these settings, so a change in schema behavior can alter output without an obvious error. Keep a sample of input and output records for each flow to compare after the cutover.
Kafka cluster version and TLS posture
Confirm the Kafka version and the security protocols on every listener the flows use, since SSL, SASL and unsecured listeners can coexist on one cluster. Confirm the supported Kafka releases and upgrade paths with the Kafka team before you rely on any client version behavior, and check the Kafka upgrade notes for the Java runtime that brokers and clients need. Record the bootstrap servers and the CA certificates NiFi trusts so the new Kafka3ConnectionService can be configured against the same endpoints.
ACLs, super users and test access
Document the ACLs granted to each NiFi principal on the topics and consumer groups it uses, and who is listed as a super user. In KRaft clusters, resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is enabled, so a new client identity or consumer group can fail on first connect for reasons unrelated to the processor. Request a non-production test environment with a test Kafka cluster that NiFi can reach, using the same authentication model, so you can run the new processors before production cutover.
How Kafka3ConnectionService and the Kerberos User Service replace standalone settings
In the 1.x flows, each ConsumeKafka_2_6 or PublishKafka_2_6 processor carries its own bootstrap servers, security protocol, SASL mechanism and Kerberos fields. NiFi 2 moves those settings into a shared controller service, Kafka3ConnectionService, which one or more ConsumeKafka and PublishKafka processors reference. Standalone Kerberos properties on Kafka processors were deprecated in 1.21.0 in favor of a Kerberos User Service property, so the Kerberos identity is also referenced rather than typed into each processor. Before you replace anything, inventory every Kafka processor in your flows and record the connection values each one uses. Processors that point at different clusters or different principals will need separate connection services, and it is easier to see that split during the inventory than after cutover.
Confirm four things for each cluster. First, the SASL mechanism in use: GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER are the mechanisms Kafka documents, and the value you choose must match the broker listener configuration exactly. Second, whether TLS is configured on the listeners you connect to, since Kafka treats security as optional and a cluster may expose both secured and unsecured listeners. Third, how service accounts are referenced: for GSSAPI, identify the Kerberos principal, the keytab location on each NiFi node and the krb5 configuration those nodes read. Fourth, who owns the keytab files and how they are rotated. A keytab copied to several nodes with different permissions is a common source of failures that look like broker problems.
Verify every property name against the NiFi component documentation for the NiFi 2 release you are targeting, because the names in the controller service and the Kerberos User Service differ from the standalone properties you are used to. Create the new services in a test environment first, point a test processor at a non-production topic, and confirm authentication succeeds on every node, not only the node you are logged into. Record which services and principals each process group uses so that the cutover checklist can name them. Any mismatch in mechanism, listener or principal should be resolved before you touch consumer group offsets.
Record strategies and consumer offsets during cutover
The versioned processors and the NiFi 2 processors both use record readers and writers, but the configuration that wires them together lives in different places. In the older flows, the reader and writer were often set directly on the processor. In the NiFi 2 pattern, the Kafka connection is handled by Kafka3ConnectionService, and the processors reference it for transport. Review each flow to record which reader, writer and schema access strategy it uses before you change anything, because those choices carry over as configuration rather than as code.
- Reader and writer mapping: List every record reader and writer on the ConsumeKafka_2_6 and PublishKafka_2_6 processors, including the schema access strategy and any schema text or registry reference.
- Message format: Confirm whether producers send one record per message or batches, and whether the publisher's record strategy still produces the same serialized output downstream expects.
- Schema changes: Check whether any schema property was renamed or moved in the replacement processors, and test with a sample of real messages.
- Downstream consumers: Identify systems that rely on field order, data types or null handling, since a changed writer setting can alter output without failing the flow.
- Test set: Use representative messages, including malformed and edge-case records, to compare output from old and new processors byte by byte or field by field.
Before cutover, confirm the replacement reader and writer produce output your downstream systems accept, and keep the test results with the change record.
Consumer offsets determine where a consumer resumes after a restart or a processor swap. Kafka stores committed offsets per consumer group and partition, so the most important question is whether the new processor uses the same group ID as the old one. If it does, it can resume from the committed position. If it uses a new group ID, it has no committed position and falls back to the configured offset reset behavior, which can reprocess or skip data depending on the setting. Decide the group ID and reset policy as part of the cutover plan, not during it.
- Group ID decision: Record the group ID used by the versioned processor, then choose whether the new processor reuses it (continuation) or starts a new group (restart from a chosen point).
- Committed offsets: Read the current committed offsets for the group with Kafka's consumer group tooling before stopping the old flow, and save them.
- Offset reset policy: Set the auto offset reset behavior explicitly so a new group does not silently start at the beginning or the end of retained data.
- Overlap window: If both processors must run side by side, make sure they use separate group IDs or only one consumes a given topic, to avoid split delivery.
- Stop order: Stop the old consumer, confirm its last committed offset, then start the new one, so the handover point is known.
During the test run, compare committed offsets before and after the swap, and confirm the group ID and reset policy match the plan you approved.
Offset handover is where duplicate and missed records appear. A duplicate happens when the new consumer restarts from an earlier position than the old one committed, or when a publisher retries a send that had already succeeded. A gap happens when the consumer starts past unread records, usually because of a reset policy or a manual offset change. Neither failure usually raises an error in NiFi, so detection has to be designed into the flow and the test plan.
- Sequence keys: Where producers include a sequence number, event ID or business key, count and check for gaps and repeats in the destination after the test run.
- Partition-level checks: Compare the last offset processed per partition with the last committed offset, and confirm no partition was skipped.
- Idempotent writes: Make the downstream write idempotent with a unique key or upsert, so a replay does not create duplicates.
- Publisher delivery settings: Review the acknowledgment and retry settings on PublishKafka so retries do not create duplicate messages without a way to detect them.
- Provenance review: Use NiFi data provenance on a sample of flowfiles to confirm the path and timing of records across the handover.
Treat any duplicate or gap in the test run as a blocker. Confirm the detection method with your data owners before production cutover.
Cutover sequence with checks and rollback points
This sequence moves each Kafka flow from the versioned 2.6 processors to ConsumeKafka and PublishKafka with Kafka3ConnectionService, with a verified offset position and a working rollback path at every stage. Each flow is cut over alone, so a problem affects one data path rather than the whole estate.
Freeze and drain the existing consumers
Stop new input to each flow in scope, let the ConsumeKafka_2_6 processors finish the records already in their queues, and confirm the downstream connections are empty. Involve the owners of upstream producers so they pause or buffer publishing during the window. The step is done when every queue feeding the old consumers reads zero and the flow's last processed record is logged. Keep the old processors running but stopped, so they can be restarted if a later check fails.
Record committed offsets for each topic
Capture the committed offset for every topic and partition each consumer group reads, using the Kafka consumer group tooling your team already operates. Store the output alongside the flow version and the NiFi process group identifier, since this is the reference point for resuming without loss or duplication. Confirm the consumer group ID you plan to use in the new processor matches the one in the old flow, or record the change. The step closes when the offset snapshot is saved and a second person has reviewed it.
Deploy Kafka3ConnectionService and Kerberos User Service
Create the Kafka3ConnectionService with the bootstrap servers, TLS truststore and SASL settings that match the target cluster's listeners, and create a Kerberos User Service for the service principal instead of entering Kerberos properties on each processor. Confirm the principal, keytab location and krb5 configuration on every node in the cluster. Enable the services in a test process group first and check that the connection reports success from each node. The step is done when the services are valid and no existing production processor references them yet.
Swap processors in a test flow and validate records
Copy one flow into a test process group, replace the 2.6 consumer and publisher with ConsumeKafka and PublishKafka, and point them at a non-production topic with a known set of test messages. Set the record reader, record writer and the message demarcation or record strategy so they match the old output, then compare the resulting content and attributes with the baseline. Check the handling of malformed records against the error relationship you already use. The step closes when the output matches the baseline for the agreed sample and the error path behaves as expected.
Validate offsets and resume in the test flow
Start the new consumer in the test flow from the recorded offset position, then stop it and restart it to confirm it resumes where it left off without reprocessing or skipping messages. Verify that offset commits appear in the consumer group on the broker, and that the group is not left with a stale or orphaned state. If you rely on the auto offset reset setting, confirm which value applies to a group with no committed offset. Only proceed when a restart produces the expected next record in the test topic.
Cut over one production flow at a time
For the first production flow, apply the validated processor configuration, keep the old consumer and publisher disabled but present on the canvas, and start the new processors with the offset position recorded earlier. Keep the upstream pause in place until the first batch has flowed end to end and the downstream systems have acknowledged it. Repeat this stage for each remaining flow only after the previous one has passed. The rollback point is re-enabling the old processors and stopping the new ones, which works because the old components were never deleted.
Monitor lag, errors and publish failures
After each cutover, watch consumer group lag, NiFi bulletins for the new processors, provenance for routing to failure or retry relationships, and the broker's authentication and authorization logs for denied requests. Compare record counts and error rates with the period before the change. Confirm that the service principal has the topic and group permissions the new client needs under your authorization model. Define with the owning team what threshold triggers a rollback before the window opens, and treat it as a go or no-go decision.
Retire the 2.6 processors after acceptance
Once every flow has run on the new processors through an agreed observation period, remove the disabled ConsumeKafka_2_6 and PublishKafka_2_6 components and any standalone Kerberos properties they carried. Keep the offset snapshots, the flow versions and the test evidence with the change record. Update the flow documentation and any runbooks that name the old processor types. The step is complete when a search of the flow definitions finds no references to the deprecated processors and the change record is signed off.
What makes Kafka processor swaps take longer
This table lists the factors that expand or shrink a ConsumeKafka and PublishKafka swap, with the question to answer for each one. Use it to scope your own estimate before asking for a fixed-scope quote, since the number of flows, the authentication method and the offset handling usually matter more than the processor count alone. Rows with the most unknowns are the ones to settle first.
| Driver | Why it matters | What to confirm |
|---|---|---|
| Flows and process groups using Kafka | Each flow that references a 2_6 processor needs its properties mapped, retested and cut over, and shared process groups multiply the work. | How many processors of each type exist, which process groups hold them, and whether any are copied across environments. |
| Kerberos versus other SASL mechanisms | Standalone Kerberos properties were deprecated in 1.21.0 in favor of a Kerberos User Service, so Kerberos flows need controller service rework and keytab handling. | Which SASL mechanism each processor uses today (GSSAPI, PLAIN, SCRAM or OAUTHBEARER) and where keytabs and principals are stored. |
| Record schema complexity | Record strategies, schema access and reader or writer settings must match the message format exactly, and schema drift shows up only under real traffic. | Whether messages are raw, record-based or Avro, which schema registry or schema text is used, and how many distinct schemas flow through each topic. |
| Consumer group count and offset sensitivity | Changing processors can change group identity or starting position, so a careless cutover can replay or skip messages for downstream consumers. | How many consumer groups read each topic, their committed offsets, and whether downstream systems can tolerate duplicates or gaps. |
| Custom NARs or scripts wrapping Kafka calls | Custom processors or ExecuteScript code that call Kafka client APIs directly must be found, rebuilt and tested against the NiFi 2 bundle set. | Whether any custom NAR, scripted processor or external client code uses Kafka libraries, and which Kafka client version each one bundles. |
| Access to a test Kafka cluster | Without a non-production cluster that matches broker version, security settings and ACLs, offset and authentication behavior can only be checked in production. | Whether a test cluster exists, its Kafka version and security configuration, and whether producers and consumers can be replayed against it. |
| Kafka broker version and ACLs | Kafka 4.x requires KRaft and newer Java runtimes on the broker side, and ACLs without matching grants can block the new clients after cutover. | The broker version in use, its authorization settings, and whether the NiFi principal already has read, write and group permissions. |
Questions teams ask before upgrading NiFi Kafka processors
What does a Kafka processor upgrade cost and what drives the fee?
The fee follows from the written scope, not from a list price, because DeployOpen does not publish one. The main cost drivers are the number of flows and process groups that use the deprecated processors, how many distinct Kafka clusters and topics are involved, whether Kerberos principals and keytabs must be reworked, and how much offset handling and record schema work each flow needs. One-time work covers inventory, rebuilding flows, testing and cutover. Ongoing costs are limited to any retainer you choose for later support. Ask for a quote that lists each flow, each cluster, the Kerberos change and the cutover plan separately so you can compare options with your own team's effort.
How long does a Kafka processor upgrade take?
Duration depends on how many flows you have, how many of them consume or publish with the deprecated processors, and how many teams must agree to a cutover window. Kerberos changes and offset verification usually take longer than swapping processor types, because they need access to the KDC, the Kafka brokers and a test window that matches real traffic. Ask for a timeline in the written scope that separates inventory, build, test and cutover, and that names the approvals each stage needs. No fixed duration is stated here because none has been supplied for this work.
What must our team provide before the work starts?
Plan to provide the exported flow definitions for every process group that uses ConsumeKafka_2_6 or PublishKafka_2_6, a list of the Kafka clusters, topics and consumer group IDs, and a named contact for each flow owner. The team also needs access to the Kafka brokers in a test environment, and read access to the KDC or a documented keytab process if Kerberos is in use. Confirm a test window with enough time to compare message counts and offsets before production cutover. Missing exports or unclear flow ownership are the most common causes of delay.
Can an in-house team upgrade these processors without a partner?
Yes, an in-house team can do this if it has NiFi operators who can read flow definitions, Kafka engineers who understand consumer groups and offsets, and someone who controls the Kerberos configuration. The deprecation path is documented: the NiFi project lists ConsumeKafka_2_6 and PublishKafka_2_6 as deprecated in NiFi 1.29.0, with ConsumeKafka and PublishKafka plus Kafka3ConnectionService as the replacements. The harder part is coordinating test traffic, cutover and rollback across teams. A partner is most useful when the number of flows is large, the security model is complex, or the team has no spare capacity for a cutover window.
How do we verify consumer offsets during cutover without losing data?
Verify offsets by recording the committed offset and lag for each consumer group before the change, stopping the old consumer at a known point, and confirming that the new ConsumeKafka processor, configured with the same group ID, resumes from the committed position. Compare record counts and sample payloads between the old and new paths in the test environment, and keep the old flow available until the new one has run through an agreed period. Check how the new processor handles record strategies, since a mismatch in parsing can look like data loss. Document each step so you can roll back to the previous offset if counts diverge.
Do we have to upgrade the Kafka cluster before changing the NiFi processors?
Not necessarily, but you must confirm the version compatibility first. The NiFi change replaces processor types and the connection service, and the Kafka cluster version is separate. The Kafka project lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and rolling upgrades are supported from each of the last three releases to the latest version. If your brokers run an older release, plan that upgrade on its own schedule and confirm with the Kafka version matrix that your chosen NiFi client setup works with it. Also check Kafka's security settings, since SASL mechanisms and authorizer rules affect the new connection service.
Kafka cluster versions to confirm before the processor swap
Before you change any NiFi processor, confirm which Apache Kafka version your brokers run and whether they run in KRaft mode. The new ConsumeKafka and PublishKafka processors work with Kafka3ConnectionService, and the broker version decides which client behavior and security options you can rely on during cutover. Check the version on every broker, not only the one your team manages, because a mixed-version cluster changes what you can safely test. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as the supported releases, and all earlier versions are archived, so a cluster on an archived release should be scheduled for upgrade as its own project.
Kafka's time-based release plan supports rolling upgrades from each of the last three releases to the latest version. If your brokers sit outside that window, move them forward in stages before the NiFi change, not during it. The Kafka 4.x upgrade notes add two hard requirements. Apache Kafka 4.0 supports only KRaft mode and ZooKeeper mode has been removed, so a ZooKeeper-based cluster must migrate to KRaft before it can run 4.x. The notes also recommend that KRaft clusters older than 3.3.x move to 3.9.x before they go to 4.x. Brokers on 4.x require Java 17, and clients need at least Java 11, so check the broker JDK separately from the NiFi JDK.
Treat any broker upgrade as a separate change with its own owner, rollback plan and change window. Do not combine it with the NiFi processor swap, because a failure then has two possible causes and no clean rollback point. Ask your Kafka operator to confirm the current version, the controller mode, and the next supported target in writing, and agree which release the NiFi cutover will be tested against. Only start the processor swap once the brokers have been stable on that release for a period your change process defines.
How DeployOpen scopes and delivers a NiFi Kafka processor upgrade
DeployOpen approaches the move from ConsumeKafka_2_6 and PublishKafka_2_6 to ConsumeKafka and PublishKafka as a scoped engagement tied to your existing flows, cluster and Kafka environment. We inventory the affected processors, rebuild the connection layer around Kafka3ConnectionService, and test cutover before any production traffic moves. Work is delivered inside infrastructure you control.
How we scope
- We inventory every flow and process group that uses the deprecated Kafka processors and record each one's Kerberos and record settings.
- The written scope names the flows, environments, Kafka clusters and offset handling covered, so both sides agree on what is in and out.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts, and a quote follows from that scope.
How we build
- We build replacement flows in a non-production NiFi, moving Kerberos settings into a Kerberos User Service and connections into Kafka3ConnectionService.
- We test record reader and writer strategies, message keys and consumer group behavior against representative topics and payloads.
- Cutover steps for consumer offsets, queue draining and rollback are rehearsed and written down before the production change window.
How we hand over
- You receive the updated flow definitions, controller service configuration and a mapping from each old processor to its replacement.
- The runbook covers the cutover sequence, offset checks, verification queries and the rollback path for each flow.
- Documentation describes the configuration in your own environment so your team can operate and change it after handover.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers for NiFi and Kafka changes after cutover.
- Support can cover follow-up tuning of consumer settings, Kafka client upgrades and review of later NiFi 2.x releases affecting these processors.
- Ask for the support scope, response expectations and change process to be written into the agreement alongside the build scope.
