Apache NiFi and Apache Kafka integration: what the flow does and how it connects
If your team needs Apache NiFi to read from or write to Kafka topics on infrastructure you control, the NiFi 2 pattern is ConsumeKafka for reading and PublishKafka for writing, with both processors pointing to a Kafka connection controller service rather than holding broker settings themselves. The connection service is where bootstrap servers, security settings and client configuration live, so several processors and flows can share one definition. In NiFi 2, Kafka3ConnectionService manages connections to standard Kafka brokers, and AmazonMSKConnectionService manages connections to brokers in AWS MSK. Choose the service that matches where your brokers run, then confirm the broker address format, authentication method and TLS trust settings with your Kafka administrator before building flows against it.
This page covers the NiFi side of the integration: how the two processors and the connection service fit together, how record readers and writers shape message content, how output strategies control keys and headers, how parse failures are routed, how to connect to Amazon MSK, and how to move off the older versioned Kafka processors. It assumes a working NiFi 2 installation and a Kafka cluster that you already operate or are building separately. Broker deployment, topic design, partition counts and retention belong to the Kafka implementation work, so this page only touches them where they change NiFi settings. Confirm the Kafka release your brokers run against the supported versions on the Apache Kafka downloads page before you pin client expectations.
What you need before building Kafka flows in NiFi 2
Most Kafka flow delays come from decisions made before the first processor is placed. The cards below cover the runtime, the Kafka release line, broker security, network paths, and the record format agreed with topic owners, in the order they tend to block a build.
NiFi 2 on Java 21
NiFi 2 needs Java 21 as its minimum runtime, so confirm the JVM on every node before you install anything else. Check that the NiFi cluster version matches the one your team tested against, and that no node is still running a 1.x flow. Kafka processors in NiFi 2 depend on the connection controller service model, which 1.x flows do not use, so existing Kafka flows need to be rebuilt rather than copied across.
Kafka release line and support status
Confirm which Kafka release line your brokers run and whether it is still supported before you pick client settings. Check the downloads page for the current supported list, and plan an upgrade if your brokers are on an archived release. Upgrading brokers to 4.x requires KRaft mode, so a ZooKeeper-based cluster must migrate first. Record the broker version in the design document, since it decides which client features and security options are available.
Broker listeners and TLS details
Ask the Kafka owner for the bootstrap server list, the listener name and port for NiFi, and the certificate chain the brokers present. Confirm whether the listener uses SSL/TLS, SASL, or both, because Kafka allows unsecured listeners alongside secured ones. Get the truststore and any client certificate and key material through a controlled channel, and confirm the hostnames in the certificates match the advertised broker addresses.
SASL mechanism choice
Choose one SASL mechanism with the Kafka owner before configuring the connection service. The documented options are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, and each brings different credential storage and rotation work. Confirm which principal NiFi will authenticate as, and whether that principal has the topic read or write grants the flows need, since Kafka authorization denies resources without ACLs unless the cluster allows otherwise.
Network paths and firewall rules
Map every network path from each NiFi node to the brokers, including bootstrap addresses and any advertised listener addresses that resolve differently inside a private network. Confirm that each node, not only the primary node, can reach every broker on the agreed port. Clusters in separate subnets or behind a proxy often fail only on some nodes, so test connectivity from each node before scheduling the processors.
Record format agreed with topic owners
Agree the message format and schema with the topic owners before building the record reader. Decide whether messages are JSON, Avro, or delimited text, where the schema lives, and how schema changes are announced. Also agree what happens to malformed messages, since ConsumeKafka routes unparseable messages to a parse.failure relationship. A written record contract prevents the reader and writer from silently drifting away from the producer.
Amazon MSK items to confirm
If the brokers run on Amazon MSK, confirm the details with the AWS account owner before building: the cluster ARN, the IAM authentication setup, the IAM role NiFi nodes will assume, and the VPC subnets and security groups that allow access. NiFi 2 uses AmazonMSKConnectionService for MSK brokers, so the account owner also needs to confirm which MSK authentication mode is enabled. Ask who owns the IAM policies and who approves changes to them.
How to configure ConsumeKafka and PublishKafka step by step
This sequence takes a NiFi 2 cluster from no Kafka connection to a verified round trip, where ConsumeKafka reads one test topic and PublishKafka writes it back. Each step closes with a check so problems surface while the flow is still small.
Create the Kafka connection service
In the NiFi canvas controller services, add a Kafka3ConnectionService and set the bootstrap servers to the broker addresses your Kafka team provides, one list shared by every Kafka processor that uses it. Leave the security settings for the next step so the first test isolates networking and naming problems. Enable the service and confirm it reports an enabled state with no validation errors. If it stays invalid, check the bootstrap address format and that the NiFi nodes can reach the brokers on the listener port. For Amazon MSK, use AmazonMSKConnectionService instead and confirm the IAM or authentication method with your AWS team.
Configure brokers and client security
Match the connection service security settings to the listener your Kafka administrators exposed, which may be SSL or one of the documented SASL mechanisms such as SCRAM-SHA-512 or OAUTHBEARER. Ask for the truststore, keystore or credentials as sensitive parameters or controller service properties rather than typing them into processor fields. Confirm the client identity has been granted read on the test topic and, for the consumer group, the group access needed to join it, because Kafka authorization denies resources without ACLs unless the cluster is configured otherwise. Check the result by re-enabling the service and reviewing the NiFi bulletin board for authentication or authorization errors.
Add record readers and writers
Decide whether the messages are plain bytes, delimited text or structured records. For JSON or Avro payloads, add a record reader that matches the format, plus a record writer for the output, and attach an explicit schema or schema registry lookup that your data owners approve. Record-based processing is what allows ConsumeKafka to group records into FlowFiles and to send unparseable messages to parse.failure. Enable both services and confirm they validate against a sample message taken from the topic, which you can obtain from your Kafka team or a console consumer.
Configure ConsumeKafka with an output strategy
Add ConsumeKafka, point it at the connection service, set the topic name and consumer group, and choose a record reader. Select the output strategy deliberately: Use Content as Value is the default and keeps only the message value, while Use Wrapper also carries the key, headers and metadata that downstream routing often needs. Set header selection and grouping by topic, partition and selected headers only if your downstream logic depends on them. Connect the success relationship to a downstream processor and the parse.failure relationship to a separate queue. Start the processor and confirm that FlowFiles appear on success with the expected attributes and content.
Inspect the parse.failure path
Before trusting the consumer, deliberately send one malformed message to the test topic, such as a payload that does not match the chosen reader schema. Confirm the malformed message lands on parse.failure and not on success, and that the FlowFile keeps enough context to trace it back to its topic and partition. Decide what happens to these FlowFiles, such as a retained queue with an alert or a dead-letter topic, and record that decision in the flow documentation. Do not move to publishing until both the valid and invalid paths behave as designed.
Configure PublishKafka for the output topic
Add PublishKafka, reuse the same connection service, and set the target topic. Choose how FlowFile content is sent: as single messages, as delimited messages, or as records through a record reader. If you publish wrapped records, confirm how null values are handled, because a null record value produces a tombstone message that removes the key from a compacted topic. Set the key and header behavior from the flow design and connect success and failure relationships to queues you monitor. Run the processor once with a single FlowFile and check the broker side with your Kafka team that one message arrived on the expected partition with the expected key.
Test the round trip on a single topic
Using one dedicated test topic, route the PublishKafka output back into a ConsumeKafka processor, then send a small known set of messages that includes one valid record, one malformed record and, if your design uses it, one keyed or null-valued record. Compare the counts and contents at each stage with the input, confirming nothing is duplicated or silently dropped and that parse failures are isolated. Record the consumer group offsets and the NiFi provenance for the run. Only after this round trip matches expectations should the flow be promoted to the production topic and cluster, with the same steps repeated in that environment.
Choosing between self-managed brokers and Amazon MSK connections
The first decision is where the brokers live. Kafka3ConnectionService manages connections to brokers that your team deploys and operates, whether on virtual machines, bare metal or Kubernetes in infrastructure you control. AmazonMSKConnectionService manages connections to brokers in an Amazon MSK cluster, where AWS operates the broker fleet and you own the topics, clients and access configuration. The service you pick sets which broker endpoints, bootstrap settings and security options appear in the controller service, so it should be decided before any processor is configured.
- Operational ownership: With self-managed brokers, your team owns broker patching, storage, listener configuration and version upgrades.
- Managed control plane: With MSK, AWS handles broker hosts and the underlying infrastructure, while you still manage topics, ACLs and client credentials.
- Version alignment: Check which Kafka client version each service embeds in your NiFi release, since broker compatibility depends on it.
- Multi-cluster flows: A flow that reads from one cluster and writes to another may need two connection services, one per platform.
- Migration path: Moving a flow from self-managed brokers to MSK usually changes the connection service, not the processor logic.
Confirm the broker platform and the NiFi version together before you copy any configuration from an existing flow.
Authentication is where the two services are most likely to differ in practice. Kafka supports SSL and SASL for client authentication, with SASL mechanisms including GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Your self-managed cluster decides which of these listeners and mechanisms it exposes, and the connection service must match that choice. MSK offers its own set of authentication options, and the supported combinations are defined by AWS and by the NiFi component documentation for your version, so do not assume a mechanism carries over from one platform to the other.
- Mechanism match: The mechanism configured on the connection service must match a listener the broker actually exposes.
- Credential storage: Decide where SASL usernames, passwords or certificates are kept, and whether NiFi sensitive property handling covers them.
- Certificate trust: TLS trust stores and key stores are configured differently depending on whether your CA or AWS-managed certificates are involved.
- Authorization: Kafka ACLs on your cluster, or IAM-based permissions on MSK, determine which topics the NiFi principal can read or write.
- Principal per flow: Using one service principal for every flow makes audit trails harder to read, so plan principals per team or per data domain.
Confirm the MSK authentication options in the AmazonMSKConnectionService documentation for your NiFi version before you design the credential model.
Network reachability decides whether a connection works at all, and it often matters more than the choice of service. A self-managed cluster is reachable on the listener addresses your team publishes, which may be internal DNS names, load balancer endpoints or addresses advertised to clients. An MSK cluster exposes broker endpoints inside a virtual private cloud, and reaching them from NiFi depends on VPC placement, security groups, routing and, when NiFi runs outside AWS, private connectivity to the VPC. Advertised listener addresses are the most common source of failure, because a client can connect to the bootstrap address and then fail when the broker returns an address it cannot reach.
- Advertised listeners: Confirm that the addresses returned to NiFi resolve and route from every NiFi node, not only from the bootstrap host.
- Security groups and firewalls: Open the broker port set for the chosen authentication mode on every path between NiFi and the brokers.
- Cluster placement: Each NiFi node in a cluster opens its own connections, so reachability must hold from every node, including nodes added later.
- DNS resolution: MSK endpoints and private DNS names must resolve from the NiFi network, which often requires resolver configuration.
- Cross-region paths: Flows that cross regions or sites add latency and cost, and should be tested with realistic message volumes.
Test reachability from each NiFi node with the same endpoint and credentials the processors will use, not from a jump host.
Most of the work when switching services is in the connection service configuration, while the consuming and publishing processors stay largely the same. In NiFi 2, ConsumeKafka and PublishKafka reference a connection service rather than carrying broker settings themselves, so moving a flow between platforms means creating a new connection service and pointing the processors at it. Topic names, record readers and writers, and output strategies can usually be kept. The properties that change are the bootstrap endpoints, security protocol, SASL or AWS-specific settings, and the trust material used for TLS.
- Connection service swap: Create the new service, verify it, then change the processor reference during a planned stop.
- Parameterize endpoints: Use parameter contexts for bootstrap servers and credentials so the same flow can run in test and production.
- Keep processor settings: Consumer group names, offsets and record schemas should be reviewed, since a new cluster may not hold the same offsets.
- Sensitive values: Store credentials as sensitive parameters and confirm they are not written into exported flow definitions.
- Validation step: Run the connection service validation and a short test flow before enabling production consumers.
Review the connection service properties in the documentation for your NiFi version, since property names can change between releases.
How Kafka keys, headers and record fields map into NiFi
This table compares where each part of a Kafka message lands in NiFi under each ConsumeKafka output strategy, and how PublishKafka reads it back out. Read it before choosing an output strategy, because the choice decides whether keys, headers and offsets are available to downstream processors at all. The main takeaway is that the default strategy keeps only the value in content, so teams that need keys or offsets must select a wrapper or metadata strategy during design.
| Kafka element | Under Use Content as Value | Under Use Wrapper or Inject Metadata |
|---|---|---|
| Message value | Becomes the FlowFile content as raw bytes or as records parsed by the Record Reader | Inside the wrapper record's value field under Use Wrapper, or as content under Inject Metadata |
| Message key | Not carried into the FlowFile by default | Included in the wrapper record under Use Wrapper; confirm the exact field name in the processor documentation |
| Message headers | Not carried into the FlowFile by default; select header names to group records by them | Included in the wrapper record under Use Wrapper; Inject Metadata adds them as attributes, so confirm naming and limits |
| Topic, partition and offset | Not written as attributes under the default strategy; confirm which attributes the processor sets | Included as metadata under Use Wrapper; Inject Metadata writes them as FlowFile attributes |
| Unparseable messages | Routed to the parse.failure relationship when the Record Reader cannot read them | Same routing applies; keep parse.failure connected so bad payloads stay visible |
| Record Reader and Writer mapping | Reader defines the input schema; the Writer defines the output format in PublishKafka and downstream processors | Wrapper records add key, value and header fields, so the reader and writer schemas must include them |
| Null wrapped value on PublishKafka | Not applicable to plain content publishing | A null value field produces a tombstone message for that key, which compacted topics use to delete the key |
| Downstream processor reliance | Works with simple transforms and routing on content alone | Needed for keying by message key, routing on header values or preserving offsets for replay |
Securing Kafka connections from NiFi
NiFi's Kafka processors reach brokers through a connection controller service, so most of the security decisions sit in one place. The cards below cover transport encryption, the authentication mechanism, how credentials are stored, and the Kafka authorization model that decides what each NiFi identity may read or write.
Encrypt client connections with TLS
Decide whether the Kafka3ConnectionService connects to brokers over SSL/TLS and confirm that each broker listener NiFi uses is secured. Kafka security is optional and clusters can mix secured and unsecured listeners, so a plaintext listener left open for convenience can expose messages even when NiFi's own side is encrypted. Ask the Kafka team which listener endpoints the flow will use, and confirm the truststore that NiFi must trust for those brokers.
Choose a SASL mechanism
Kafka documents GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER for SASL authentication. Pick the one your identity platform can issue and rotate, and confirm that the Kafka3ConnectionService and AmazonMSKConnectionService settings you need exist for that mechanism in your NiFi version. PLAIN sends credentials that must be protected by TLS, so it should not run without encryption.
Store Kafka credentials as sensitive values
Put SASL passwords, keystore passwords and any OAuth or Kerberos secrets into NiFi sensitive property fields or parameter contexts marked sensitive, not into plain processor properties or flow definitions that teams export and commit to Git. Confirm how your NiFi deployment supplies the sensitive properties key and who can read parameter contexts, because those controls determine whether a flow export leaks broker credentials.
Scope topic access with Kafka ACLs
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is set to true. Keep that setting false and grant the NiFi principal read access only to the topics and consumer groups its flows use, and write access only to the topics it publishes to. Review the grants whenever a flow is added or retired.
Separate NiFi access policies from Kafka ACLs
NiFi's policies decide which users can view, edit or operate a processor, controller service or parameter context. Kafka ACLs decide which principal may read or write a topic at the broker. A user who can edit the ConsumeKafka processor in NiFi can change which topic it reads, so a NiFi edit right can widen data access even when broker ACLs are tight. Document both control sets and assign them to the same owners.
Common Kafka and NiFi integration failures and their fixes
This table pairs each common failure in a NiFi and Kafka flow with its likely cause and the first check to run. Read it top to bottom when a flow stops, routes records to failure, or delivers data that looks wrong. The main takeaway is that most failures sit in the connection service, the security settings or the record schema, so check those before changing processor logic.
| Failure | Likely cause | First check |
|---|---|---|
| Connection refused | Broker address, port or listener does not match the Kafka3ConnectionService settings, or the broker is not reachable from the NiFi node | Confirm the bootstrap server list and that each NiFi node, not only the primary node, can open a connection to the broker port |
| TLS handshake error | Truststore in the connection service does not trust the broker certificate chain, or the client is sending plaintext to a TLS listener | Compare the protocol the broker listener advertises with the security protocol set in the connection service, then check the truststore contents |
| SASL authentication failure | Wrong mechanism (for example PLAIN versus SCRAM-SHA-512), mistyped credentials, or a Kerberos principal that the broker does not know | Confirm the mechanism enabled on the broker listener matches the one configured in NiFi and that the credentials work with a standard Kafka client |
| ACL denial (authorization) | The client principal has no ACL for the topic or consumer group, and the cluster does not allow access when no ACL exists | Check the principal name NiFi presents against the ACLs on the topic and group, and review allow.everyone.if.no.acl.found on the cluster |
| Records routed to parse.failure | Record reader schema does not match the message payload, such as a missing field, wrong type or non-JSON content | Inspect a failed FlowFile's content and the reader's schema, and check whether producers changed the message format recently |
| Duplicate messages after restart | Offsets were not committed before the restart, so ConsumeKafka re-reads messages from the last committed position | Check consumer group offset state and make downstream writes idempotent using the Kafka key or a unique message identifier |
| Out-of-order messages | Records from different partitions are processed in parallel or merged across FlowFiles, so partition order is not preserved downstream | Confirm which partition each FlowFile came from, and check whether the flow splits, merges or runs concurrent tasks between consume and write |
| MSK connection errors | AmazonMSKConnectionService lacks the IAM or SASL settings, region, or broker endpoints that MSK requires, or the NiFi host role lacks permissions | Verify the broker endpoints and region in the service, then confirm the IAM principal can connect and access the topics |
Moving off the old versioned Kafka processors in NiFi 2
Flows that use the older versioned Kafka processors need a review before any NiFi 2 migration, because the way those flows connect to brokers changes in NiFi 2. The Kafka processors in NiFi 2 take their broker connection from a connection controller service, so broker address, security settings and client properties move out of each processor and into one shared service. Start by inventorying every processor that talks to Kafka, the version it uses, the properties it sets, and the relationships downstream of it. Record which flows depend on specific header names, key handling or record schemas, because those are the settings most likely to change behavior after the move.
The connection controller service changes how you configure and govern Kafka access. Kafka3ConnectionService manages broker connections for the generic Kafka processors, and AmazonMSKConnectionService covers AWS MSK brokers, so the choice of service follows the target cluster rather than the flow. Plan one service per cluster and credential set, confirm which security mechanisms your brokers accept (SSL or SASL, as configured on the broker listeners), and check that the consumer group and offset behavior you expect is preserved under the new settings. Confirm the exact processor replacement mapping, and the property names that carry over, in the NiFi component documentation for each processor, and follow the path in the NiFi migration guidance, which requires moving through NiFi 1.27.0 before 2.0.0.
A parallel run is the practical way to show that the migrated flow produces the same results. Run the NiFi 2 flow against the same topic with a different consumer group, or point the publishing flow at a test topic, while the production flow keeps running. Compare record counts, parse.failure volumes, key and header values, and the content of sample FlowFiles, and agree on the tolerance for differences before the run starts. Keep the parallel run long enough to cover peak traffic and at least one reprocessing or restart scenario, and only cut over once the comparison is signed off by the team that owns the downstream consumers.
Questions buyers ask before building NiFi Kafka flows
What drives the effort of a NiFi and Kafka integration?
The effort depends mainly on four factors: how many topics and consumer groups the flows touch, how well the record schemas are defined, which security model the brokers enforce, and whether the cluster is self-managed or Amazon MSK. Each topic often needs its own consume or publish path, error handling and tests, so a flow count of ten is not ten copies of one flow. Record schemas decide how much work goes into record readers and writers, because undefined or drifting fields force mapping and validation logic into the flow. The security model adds connection configuration through Kafka3ConnectionService, including SSL or SASL settings and credentials stored in NiFi. Amazon MSK changes the connection path, because AmazonMSKConnectionService handles AWS MSK brokers rather than a generic broker list. Ask for a scope that lists each topic, its direction, its schema source and its security requirements before you compare quotes.
Can our own team build NiFi Kafka flows without a partner?
Yes, a team with NiFi and Kafka experience can build these flows internally, and the decision usually depends on capacity and existing skills rather than on the tools themselves. The processors are documented in detail, and the work follows known patterns: choose a record reader and writer, set the output strategy, route parse failures and test the flow. The harder parts are usually outside the flow canvas: agreeing schema contracts with topic owners, getting broker access and certificates, deciding how consumer groups and offsets are managed, and testing failure behavior under load. If your team has not operated a secured Kafka cluster or a clustered NiFi before, plan time to learn both. An outside partner can take on the build and the handover, while your team keeps ownership of the topics and the running system.
What does DeployOpen need from our team during the work?
The engagement needs named people who can answer for four things: topic owners who can confirm what each topic contains and how it changes, broker and network access for the NiFi cluster, credentials or identity settings for the chosen authentication method, and a schema contract for every record-based flow. Without the topic owners, parsing and routing decisions rest on guesses about the data. Without broker access, connection testing is delayed, and that delay is often the longest item in the plan. Ask your team to provide a test topic or non-production cluster before the build starts, and to agree how schema changes are announced. Confirm these inputs in writing as part of the scope, because they determine which dates in the plan are realistic for your environment.
How does Kafka's release support affect the integration plan?
Kafka's release support determines which broker version you target, which in turn affects the client and connection settings that the flows depend on. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and earlier versions are archived. The project aims for three releases a year and supports rolling upgrades from each release in the past year to the latest version. Apache Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate first, and KRaft clusters older than 3.3.x should move to 3.9.x before 4.x. Confirm your broker version and its support status before scoping, and plan a test of the NiFi flows against any broker upgrade. Also check Java requirements, since brokers need Java 17 from Kafka 4.0 onward.
Is NiFi the right consumer for our Kafka topics?
NiFi fits well when messages need parsing, routing, enrichment or delivery to several systems that are not Kafka, and a plain Kafka consumer may fit better when one service needs low-latency processing in code. ConsumeKafka can interpret messages as NiFi records, group them into FlowFiles by topic, partition and selected headers, and route unparseable messages to a parse.failure relationship. That makes NiFi useful for fan-out to databases, search indexes and object storage. Consider NiFi less suitable when the topic is the only destination and the logic is complex code that your application teams already maintain, because the flow then adds an operational component without adding much value. Test the throughput needed for your largest topic before you commit, and compare the operating cost of the NiFi cluster with that of the code it would replace.
How does a fixed-scope NiFi Kafka engagement work?
Most DeployOpen engagements are fixed-scope, which means the scope, deliverables and timeline are agreed in writing before work starts. For a Kafka integration, the written scope usually lists the topics, the direction of each flow, the record schemas, the output strategies, the authentication method, the target NiFi and Kafka versions, and the test and handover deliverables. Changes to topic count or schema contracts after signing are handled as scope changes, so the quote should show what is included and what is excluded. Ongoing support, upgrades and testing are available as separate retainers. DeployOpen does not publish a price list, so ask for a quote based on your written scope, and ask the proposal to separate one-time build work from recurring operation.
What should a quote for NiFi Kafka work include?
A useful quote separates the one-time build from the recurring costs and states who is responsible for each. The build usually covers flow design, connection setup, record schema mapping, error routing, testing and handover documentation. Recurring costs include the NiFi and Kafka infrastructure you run, the time your team spends on topic changes and any retainer for upgrades. Ask the quote to name each topic and flow, the assumed broker and NiFi versions, the security model, the test environment and the acceptance criteria. Ask what happens if a topic schema changes during the build and how a Kafka upgrade would be handled afterward. A quote that cannot answer these questions in writing is not yet a basis for comparison.
Where to verify Kafka and NiFi component details
Start with the component pages in the NiFi components documentation before you configure anything. The ConsumeKafka and PublishKafka pages list each property, its allowable values and the relationships each processor routes to, and the Kafka3ConnectionService page defines the broker, security and connection properties that both processors depend on. If you connect to AWS MSK, check the AmazonMSKConnectionService page separately, because its authentication and bootstrap settings differ from the generic connection service. Treat the property names on these pages as the reference for your build, not a list copied from an older tutorial or a forum post.
Confirm every setting against the exact NiFi and Kafka versions in use. Property names, default output strategies, record reader requirements and the handling of parse failures can change between NiFi 2 releases, and the older versioned Kafka processors are no longer the target for new flows. On the Kafka side, check the Kafka 4.3 security overview for the SSL and SASL mechanisms your brokers actually enable, and the Kafka 4.3 authorization documentation for the ACLs NiFi's principal needs on each topic and consumer group. Note that the upgrade notes differ by release line, so the Kafka 4.3 upgrade notes are the right read only if your brokers run that version. Record the exact versions, the chosen SASL mechanism and the authorization model in your build runbook so a later upgrade can be tested against them.
How DeployOpen scopes and delivers NiFi Kafka integrations
DeployOpen plans, builds, documents and supports Apache NiFi and Apache Kafka integrations inside your own infrastructure, including your cloud accounts, data centers or air-gapped networks. Work starts from a written scope that names the topics, flows, connection services and handover items before any build begins.
How we scope
- Inventory the Kafka clusters, topics, schemas and authentication methods each flow must use
- Agree the record readers, writers and output strategy for each consumed and published topic
- Define how parse failures, duplicates and Amazon MSK connections are handled and who owns them
How we build
- Configure Kafka3ConnectionService or AmazonMSKConnectionService with credentials held in your secrets store
- Build ConsumeKafka and PublishKafka flows with tested routing for parse.failure and downstream errors
- Test with representative messages, including malformed records and tombstone publishing, before cutover
How we hand over
- Deliver flow documentation, parameter and controller service inventories, and the test evidence collected
- Walk your engineers through runbooks for topic changes, offset handling and connection rotation
- Record the upgrade path for NiFi 2 and Kafka releases so your team can plan later changes
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
- Ongoing support, upgrades and continuous testing for these flows are available as a retainer
- Confirm support hours, escalation paths and change control with us in the written scope
