How NiFi moves data in and out of Kafka with ConsumeKafka and PublishKafka
A team running NiFi flows that must exchange data with a self-hosted Kafka cluster can do it with the Kafka processors that ship with NiFi. ConsumeKafka reads records from one or more topics into a flow, and PublishKafka writes flow content to a topic. Both processors depend on a Kafka connection controller service, which holds the bootstrap servers, security protocol, SASL mechanism and TLS material once so that every processor using that cluster inherits the same settings. Both projects are *Apache Software Foundation* releases under the Apache License 2.0 (SPDX: Apache-2.0), and neither the Kafka project nor the NiFi project offers a paid edition, so the cost of this integration is the engineering and operations work around it rather than a license. Before writing any flow, confirm which NiFi and Kafka versions the target environment will run. The NiFi downloads page lists NiFi 2.12.0 as the current release at the time of writing, and the Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported. Check the Kafka client version bundled with your NiFi release against the brokers you operate, because the processor's client library and the broker version do not have to match exactly, and that compatibility should be verified in a test environment rather than assumed.
The first design decision is which processor family the flow needs, and that choice follows from how the data is shaped. A flow that moves one message per FlowFile is simpler to reason about, but a flow that reads many small Kafka messages can spend more time on FlowFile overhead than on the data itself. The record-oriented processors pair the Kafka connection with a record reader and writer, so a batch of messages can be parsed, validated and converted in one pass. Confirm the message format with the producing team (JSON, Avro or another format) and confirm which schema source the flow will use, because a mismatch between the reader configuration and the bytes on the topic is the most common cause of failed records in early testing. Treat the connection controller service as a shared configuration item: if security settings change on the broker side, the change should be made once in that service and tested through a single flow before it is rolled to others.
What the Kafka and NiFi sides need before the first flow runs
Before a NiFi flow touches a topic, both platforms need a supported version, a working security model and a network path that the flow's identity can use. The cards below cover those prerequisites in the order teams usually hit them, from release choice to the 1.x migration question.
Choose a supported Kafka release on KRaft
Plan the cluster on KRaft mode with a release the project still supports: 4.3.1, 4.2.2 or 4.1.2 according to the downloads page. Kafka 4.0 removed ZooKeeper mode, so any older ZooKeeper-based cluster must migrate to KRaft first. Confirm the Java 17 requirement for brokers and Connect hosts before you pick node images, since clients only need Java 11 or later.
Run NiFi 2.x on Java 21
New flows should target NiFi 2.x, which is the line the project is moving forward on. The administration guide sets Java 21 as the minimum runtime, so check the JVM on every node and any container base image. If you plan to use the optional Python-based processors, confirm a supported Python version (3.10, 3.11 or 3.12) and treat them as a beta feature.
Create a principal and ACLs for the flow
Give each flow group its own Kafka identity, using SASL/SCRAM or mTLS, rather than sharing an admin account. In KRaft clusters, ACLs live in the metadata log, and resources without ACLs are limited to super users unless allow.everyone.if.no.acl.found is enabled. Decide up front which topics, consumer groups and transactional IDs the principal needs, so a missing ACL shows up in testing rather than in production.
Open routes and trust stores between nodes
Every NiFi node that runs a Kafka processor must reach every broker advertised in the cluster, not only the bootstrap address. Check DNS names in the advertised listeners, firewall rules for each listener port and the trust store on NiFi that holds the broker certificate chain. Test from each NiFi node individually, because a single node with a stale trust store can fail only part of the cluster.
Confirm the Kafka TLS listener
Kafka can run unsecured or mix secured and unsecured listeners, so confirm that the listener the flow uses is the TLS or SASL listener you intend. Record which mechanism applies (SSL, SASL_SSL or another documented combination) and which trust store and key store the Kafka controller service needs. Ask the platform team to confirm that the certificate SANs match the broker hostnames NiFi will resolve.
Assess existing NiFi 1.x flows
NiFi 1.28 is the last minor release of the 1.x series, and the project states that 1.x cannot be maintained further because it depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8. Migration guidance requires moving to NiFi 1.27.0 before 2.0.0 and changing the flow file from flow.xml.gz to flow.json.gz. Inventory the variable registry expressions and distributed map cache components, since both changed in 2.0, before you plan the cutover.
Configuring the Kafka connection controller service and processors
This sequence builds the NiFi side of the Kafka connection, from a tested controller service through consuming and publishing processors, and proves each stage with a visible check before cluster-wide scheduling is enabled. The result is a flow that reads from and writes to your Kafka cluster with known group membership, record handling and failure routing.
Confirm Kafka connectivity and access before building
Before touching NiFi, the Kafka team or your platform team confirms the bootstrap server addresses, the listener that NiFi will use, and the security mechanism in use (TLS, SASL or both). Confirm that the NiFi service account has ACLs for the test topic and the consumer group you plan to use, because a missing ACL will show up later as a failed connection or an empty group. Ask for these values in writing so the NiFi configuration matches the cluster's listeners. The check is a short test from a host in the NiFi network that can list topics using the same credentials.
Create the Kafka connection controller service
In the NiFi canvas, open the controller services settings for the process group that will hold the Kafka flow and add the Kafka connection controller service. Set the bootstrap servers list, then choose the security protocol and SASL mechanism that match the listener you confirmed, and enter the TLS truststore or keystore paths and passwords if the listener requires them. Store passwords in NiFi's sensitive property fields or parameter contexts rather than in plain flow text. Enable the service only after the next step passes.
Test the controller service connection
Use the verify or test action on the Kafka connection controller service so NiFi attempts a connection to the bootstrap servers with the settings you entered. Read the validation output for authentication errors, certificate trust failures and unresolved host names, and correct them one at a time. The check is a successful connection result in the test output, and a matching entry in the broker logs or the cluster client connection list if your team has access to them. Do not continue until this passes.
Add ConsumeKafka with a consumer group and topic
Add a ConsumeKafka processor and assign it the Kafka connection controller service. Set the topic name or pattern, the consumer group ID that your team has agreed to, and the offset reset behavior, which decides where a new group starts reading. Choose the record reader and record writer here if the flow works in record mode, or leave them to per-message handling if it does not. Keep the group ID unique to this flow so that other consumers do not share its offsets. Start the processor on a single node for the test.
Verify group membership and arriving records
Produce a few known test messages to the source topic with the Kafka console tools or your producer, then start ConsumeKafka and watch its output queue. Use the Kafka consumer group tooling to confirm that the group ID appears with an active member and that its committed offsets advance after NiFi processes the messages. The check is matching counts between the messages you produced and the FlowFiles that reached the success relationship, with no messages reprocessed on restart.
Add PublishKafka with record reader and writer
Add a PublishKafka processor on the same controller service, set the target topic, and choose a record reader and record writer that match the schema of the data arriving from ConsumeKafka. Set the delivery and acknowledgment options that your Kafka team has approved, and decide how message keys are derived so that related events land on the same partition. Connect the success relationship to the next processor and route failure to a dedicated handling path, such as a retry loop or a queue with an alert.
Check publish results and failure routing
Run a single-node test that publishes known records to the target topic, then read them back with a console consumer to confirm content, key and partition distribution. Next, force a controlled failure, for example by pointing PublishKafka at a topic your service account cannot write to, and confirm that the FlowFile lands on the failure relationship with an error attribute rather than disappearing. The check is that both the successful and failed paths behave as designed in the test.
Enable cluster-wide scheduling after the single-node test
Once single-node tests pass, decide how the processors will be scheduled across the NiFi cluster and enable them for all nodes, confirming that your NiFi version supports the chosen scheduling strategy. Watch consumer group membership during the change, because additional consumers in the same group trigger partition reassignment, and record the expected behavior with the Kafka team beforehand. The check is that the group shows the expected number of members, lag stays within what your team has agreed, and no FlowFiles accumulate in failure queues.
Record-oriented processing or one FlowFile per message: choosing the model
The first decision in a NiFi and Kafka flow is whether each Kafka message becomes its own FlowFile or whether a batch of messages is handled as records. With record-oriented processing, the consuming and publishing processors use a record reader and a record writer, so a single FlowFile can carry many messages that share a schema. That usually means fewer FlowFiles, less repository churn and more efficient transformations with processors such as QueryRecord, UpdateRecord and ConvertRecord. The cost is that every record must be parseable against a schema, so the reader needs a reliable schema source, such as a schema registry or a schema text property, and a clear policy for what happens when that source changes. With per-message processing, each Kafka record becomes one FlowFile with its own attributes. Routing on attributes such as the topic, partition or key is simpler to reason about and to debug, but every message then carries its own FlowFile overhead, and a flow handling high volumes must be sized and monitored with that in mind.
The model you choose shapes everything downstream. Record-based flows keep transformation logic in record paths and schema-aware expressions, so a field rename or type change touches the schema and the writer configuration rather than many processors. Per-message flows push that logic into attribute handling and content parsing, which is easier to understand for small volumes but harder to keep consistent at scale. Malformed input needs explicit handling in either model. In a record flow, check how the consuming processor and record reader report parse failures and whether a bad record can be routed to a failure relationship without discarding the rest of the batch. In a per-message flow, a single bad payload typically maps to a single failed FlowFile that is simpler to replay. Confirm the exact property names, relationships and reader and writer options for your release against the NiFi component documentation for the release you run, since these have changed between the 1.x and 2.x lines.
How offset commits and delivery guarantees change the design
When a ConsumeKafka flow commits its Kafka offsets only after the NiFi session that holds the consumed FlowFiles has committed, a message is marked as consumed only once NiFi has durably handed it to the next step. If NiFi or the downstream system fails before the session commits, the offsets are not advanced and the same records are delivered again after recovery. This is at-least-once delivery. The cost is duplicates, which the design has to absorb rather than prevent.
- Duplicates on restart: Records from an uncommitted session can be redelivered after a node restart, a rebalance or a failed write.
- Ordering within a partition: Redelivery replays records in offset order, but a retry path can still reorder work across FlowFiles, so confirm the processor ordering behavior.
- Downstream effect: Each consumer of the output must tolerate a repeated record, usually through a stable message key or event ID.
- Recovery window: The size of the replay depends on how much work was in flight when the failure occurred, so keep session sizes and back pressure settings deliberate.
- Commit cadence: Check the processor documentation for how often offsets are committed in your NiFi version.
Before you rely on this pattern, test a forced failure between the Kafka read and the downstream write and confirm the replay count matches what you expect.
If the offsets are committed before the FlowFile content is processed or written to its destination, Kafka treats those records as consumed even when NiFi later fails to deliver them. A crash, a full queue or a failed downstream call after the commit means the records are not redelivered by Kafka. This is at-most-once behavior for the affected records, and it can silently drop data. It is rarely the right choice for business events, though it can suit low-value telemetry where loss is acceptable and documented.
- Silent loss: Nothing in Kafka signals that the records never reached their destination.
- Dependence on NiFi provenance: Recovery would require reconstructing the missing records from NiFi provenance or source systems, which is slow and not always complete.
- Acceptable cases: Loss may be tolerable for sampled metrics or debug logs that are recreated by the next run, but this must be a documented decision.
- Audit impact: Regulated flows need evidence that every event was delivered, so this pattern usually fails that expectation.
- Configuration check: Confirm how the processor you use commits offsets and whether any setting changes that order.
Do not choose this pattern for data that auditors or downstream ledgers depend on without a reconciliation process that detects gaps.
Exactly-once delivery across a NiFi flow and a Kafka cluster is not established by the facts this page relies on, so it should not be assumed from the names of processors or from general Kafka features. Kafka supports transactions and idempotent producers in its own documentation, but a NiFi flow that reads from one place and writes to another also depends on how its processors coordinate offsets, writes and session commits. Treat exactly-once as a design requirement to verify, not a default.
- Kafka transaction docs: Read the Kafka transaction documentation to understand the producer and consumer settings that the broker side provides.
- Processor documentation: Confirm in the ConsumeKafka and PublishKafka documentation for your NiFi version whether any transactional behavior applies and under what settings.
- Transactional IDs: If a transactional producer is used, plan stable transactional IDs and their lifecycle as part of the build.
- Non-Kafka sinks: Any database or API written from NiFi falls outside Kafka transactions unless that system also participates in a transaction.
- Test evidence: Demonstrate end-to-end behavior with injected failures before you describe the flow as exactly-once.
Unless the processor documentation and your tests confirm the behavior, plan for at-least-once delivery.
The practical way to get a correct outcome under at-least-once delivery is to make the downstream write idempotent, so that applying the same record twice leaves the same state. This moves the guarantee from the transport to the destination, where it can be tested and audited. The design starts with a stable identifier carried in each record, such as an event ID from the source system, and a write rule that rejects or merges repeats.
- Stable keys: Use an identifier assigned at the source, not one generated by NiFi during processing, because generated IDs change on replay.
- Upsert or conditional write: Write with a primary key or a conditional statement so repeats update the same row or are ignored.
- Deduplication window: If the destination cannot key every record, keep a bounded window of seen IDs and define its length with the data owner.
- Schema changes: Ensure the identifier survives any NiFi record transformation, and test it on the record reader and writer path.
- Reconciliation: Compare counts or checksums between the topic and the destination on a schedule to catch gaps and duplicates.
Check that every consumer of the topic applies the same rule, because a single non-idempotent consumer reintroduces the problem.
Mapping Kafka topics, keys, headers and record fields to NiFi attributes
This table maps each Kafka record element to the FlowFile attribute, record field or content location that ConsumeKafka exposes, and to the matching setting on PublishKafka. Read each row as a round trip: what arrives on consume and what must be set to send it back. Attribute and property names change between NiFi releases, so confirm them in the release you run before building routing rules on them.
| Kafka element | Consumed into NiFi | Published from NiFi |
|---|---|---|
| Record key | Available as the kafka.key attribute when the record has a key | Set with the processor's key property or a key field in record mode |
| Record value | FlowFile content in per-message mode, parsed by the Record Reader in record mode | Written from FlowFile content through the Record Writer or the message body |
| Headers | Copied to attributes when a header name pattern is configured on the processor | Attributes matching the configured pattern are sent as Kafka headers |
| Partition | Exposed as the kafka.partition attribute | Chosen by the configured partitioner, so confirm before setting one manually |
| Offset | Exposed as the kafka.offset attribute, useful for lineage and troubleshooting | Assigned by the broker on write, so there is nothing to set |
| Timestamp | Exposed as the kafka.timestamp attribute from the record | Set by the producer, so confirm how the release handles timestamps on publish |
| Topic | Exposed as the kafka.topic attribute | Set by the topic name property, which accepts Expression Language |
| Schema source | Named by the Record Reader's schema access strategy, such as a schema registry or an embedded schema | Named by the Record Writer, which must match the downstream data contract |
| Idempotency field | A business identifier, such as an event ID, is read from the record and carried as an attribute | The same identifier is kept in the record so consumers can deduplicate after retries |
Securing the connection between NiFi and Kafka
A NiFi flow that publishes to or consumes from Kafka crosses two trust boundaries: the Kafka cluster's client authentication and authorization, and the NiFi instance's own user access. The cards below cover the controls on each side and the checks that tie them together.
Choose the Kafka client authentication mechanism
Kafka authenticates clients over SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. For a NiFi service account, SCRAM or OAUTHBEARER usually fits best, because the credential can be scoped to one service and rotated without changing the broker. Confirm which mechanisms your broker listeners enable, since security is optional in Kafka and a listener may accept unsecured clients.
Encrypt traffic with TLS on each listener
Kafka encrypts traffic between clients, brokers and tools with SSL/TLS. Configure the NiFi Kafka connection controller service to trust the broker certificate chain, and confirm that the NiFi flow connects to a secured listener, not a plaintext one that happens to be open on the same network. Mixed secured and unsecured listeners are possible, so check the listener the client actually uses.
Scope Kafka permissions with ACLs
In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Grant the NiFi principal only the topics and consumer groups its flows use, with read and write rights matched to each processor role. Keep that default setting at false in production so a new topic does not open to every principal.
Keep NiFi on the loopback interface by default
When no security configuration is provided, NiFi binds to 127.0.0.1, so the UI is reachable only through the loopback interface. Any move to a network-reachable address should happen only after authentication is configured. Check the bind address and the HTTPS settings in the NiFi properties file before routing traffic from other hosts, because a flow that writes to Kafka can be altered through an open UI.
Select a NiFi authentication provider
NiFi supports single-user, LDAP, OpenID Connect, SAML 2.0 and X.509 client certificate authentication. Use the directory or identity provider your organization already operates, and map its groups to NiFi access policies so that only the team that owns a Kafka flow can change its processors. The Kerberos provider in NiFi is deprecated, so do not design new deployments around it, and plan a different provider if your environment currently depends on it.
Store Kafka credentials in parameter contexts
Keep SASL usernames, passwords, keystore paths and passwords out of processor properties that are exported with the flow. Put them in sensitive parameters within a parameter context, and bind the Kafka connection controller service to those parameters. Review who can read parameter contexts, because access to a context is access to the credentials it holds. Rotate Kafka credentials on a schedule that your team can run without editing every flow.
Testing the flow before production traffic
Before any production traffic reaches a NiFi flow that consumes from or publishes to Kafka, run a scripted test plan that targets the failure paths you have designed, not only the happy path. Create a dedicated test topic with the same partition count and retention settings you plan to use in production, and give the flow's ConsumeKafka processors their own consumer group so that test offsets never mix with a real group. Name the group clearly, for example nifi-test-orders-v1, and record the group's committed offsets before and after each run so you can show exactly what the flow consumed.
Exercise the routing logic with deliberately malformed records: invalid JSON or Avro payloads, records missing required fields, values that fail type conversion, and oversized messages. Confirm that the record reader and writer failure relationships send each case to the path you designed, such as a dead-letter topic or a failure queue, and that no record disappears silently. Then test two disruptions while the flow is processing. Restart one Kafka broker and observe whether PublishKafka retries and whether any messages are published twice. Restart one NiFi node in a cluster and observe whether ConsumeKafka resumes from the last committed offset, which is where duplicates usually appear. Also run a schema change test by publishing records with an added optional field and a removed field, and confirm the reader strategy and schema registry lookups behave as your compatibility rules expect.
Run the final round in cluster mode with at least two NiFi nodes and a topic with several partitions. Start the flow, add or stop a node, and confirm in the Kafka tools that the consumer group rebalances and that partitions are not left unassigned. Illustrative example: if a six-partition topic is consumed by three nodes and one node is stopped, you should see the partitions reassigned across the two remaining nodes within the group's session timeout. Set your own acceptance criteria before the run, including the maximum tolerated duplicate count, the allowed lag after recovery and the expected dead-letter volume for each malformed-record case, and have the platform owner sign off on them.
Common NiFi to Kafka failures and their causes
This table lists the failures teams most often hit when NiFi processors talk to Kafka, with the likely cause and the first check to run for each. Read it by matching the symptom in your logs or bulletins to a row, then working the first fix before changing anything else. Most of these failures trace back to ACLs, trust stores, partition counts or how the flow handles offsets and schemas, so the first fix is usually a configuration check rather than a redesign.
| Failure | Likely cause | First fix to try |
|---|---|---|
| TopicAuthorizationException in ConsumeKafka or PublishKafka | The NiFi client principal has no Read, Write or Describe ACL on the topic or consumer group, and the cluster restricts resources without ACLs to super users | Confirm the principal in the Kafka client properties, then grant Describe and Read (or Write) on the topic and Read on the group ID with the Kafka ACL tooling |
| TLS handshake failure or PKIX path errors | The truststore in the SSL Context Service does not contain the broker's CA, the certificate chain is incomplete, or the wrong listener port is configured | Test the broker endpoint with the same truststore outside NiFi, then add the full CA chain and confirm the listener uses SSL or SASL_SSL as configured |
| NiFi nodes cannot connect to brokers | Advertised listeners on brokers resolve to addresses NiFi nodes cannot reach, or a firewall blocks the port from some nodes only | Compare advertised.listeners with what each NiFi node resolves, then test connectivity from every node in the cluster, not only the primary |
| Consumer lag keeps growing | Too few partitions for the number of concurrent tasks, or a downstream processor or sink slower than ConsumeKafka | Check how many concurrent tasks are assigned and how many partitions the topic has, then find the slowest downstream processor with queue back-pressure |
| Duplicate records after a NiFi restart or rebalance | Offsets were committed after the flow's session was already acknowledged downstream, so records replayed after failure are processed again | Make the downstream write idempotent with a stable record key, and review when the consumer commits relative to the FlowFile being persisted |
| Records fail or route to failure in record readers | The Schema Registry or Avro schema does not match the message, the writer schema differs from the reader, or the Kafka message is not in the expected format | Compare the schema name and version the reader resolves against a sample message, then correct the schema access strategy or the producer's schema |
| Per-message processing is slow at high volume | Each Kafka record becomes a separate FlowFile, so overhead grows with message count | Switch to record-oriented processing with a record reader and writer, and batch the output before publishing |
Questions teams ask before connecting NiFi to Kafka
What does the NiFi to Kafka integration effort depend on?
The effort depends mainly on four things: the number of flows that touch Kafka, the record schemas those flows read and write, the security model on both sides, and the NiFi version you run. Each flow that publishes or consumes adds its own mapping, error handling and testing. Record schemas that change often add compatibility work. A secured Kafka cluster with SASL and TLS, plus NiFi users and certificates, adds configuration and verification steps. Ask any partner to break its estimate down by these four drivers and to state which assumptions it depends on.
Can our in-house team build this without a partner?
Often yes, if the team already runs Kafka and NiFi in production and has someone comfortable with both record-based processing and Kafka consumer group behavior. The harder parts are usually offset commit behavior, exactly-once patterns, and how ACLs map to the principals NiFi uses. If one of those areas is new to your team, a short review of the design before build can reduce rework. Confirm internally whether you have the skills for each area, not only whether you have the tools.
Who do we need to involve from our side?
Plan for at least four roles. A Kafka administrator owns the brokers, listeners and cluster configuration. A NiFi administrator owns the flows, the Kafka connection controller service and the cluster. Topic and ACL owners decide which principals can read or write each topic and which consumer groups exist. Schema owners approve the record formats that flows parse and produce. In smaller teams one person may hold two roles, but each role still needs a named decision maker before the build starts.
How long will the integration take?
Duration depends on the agreed scope and the written plan, not on a general rule. The plan should list the flows in scope, the topics, the security setup, the test cases and the cutover approach. Delays usually come from late schema decisions, waiting for ACL approvals and environment access, and changes to flow logic found during testing. Ask for a timeline tied to named milestones and named dependencies on your side, and ask which items would move it.
Do our NiFi 1.x flows have to migrate before we connect them to Kafka?
Not always, but it is the first decision to make. The NiFi project states that NiFi 1.28 is the last minor release of the version 1 series, that 1.28.1 has an end-of-support date of December 8, 2024, and that users should upgrade to NiFi 2. The migration guidance requires moving to NiFi 1.27.0 before 2.0.0 and changing the flow file from flow.xml.gz to flow.json.gz. Building new Kafka connections on a 1.x line adds work you may need to repeat, so confirm your upgrade plan before scoping the connection work. Current NiFi 2 release details are on the NiFi download page.
What should we ask for in a written quote?
Ask the provider to list each flow in scope, each Kafka topic and consumer group, the record schemas and their owners, the security configuration on Kafka and NiFi, the testing approach, and the handover material. Ask which items are one-time work and which recur, such as schema changes, NiFi upgrades and Kafka upgrades. Ask what your team must provide and by when, and what is excluded. A fixed-scope written plan makes these points comparable across bids.
Does the integration need exactly-once delivery?
It depends on what the downstream system can tolerate. Exactly-once behavior across NiFi and Kafka needs an explicit design, including how offsets are committed and how writes are made idempotent or transactional. Many flows instead accept at-least-once delivery and make consumers idempotent. Decide this per flow with the data owners before building, because the answer changes the processor settings, the test cases and the operational checks. Confirm the exact guarantees against the Kafka and NiFi documentation for your versions.
How DeployOpen scopes and delivers a NiFi to Kafka integration
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, including the NiFi and Kafka pieces that connect flows to topics. For Apache Kafka and Apache NiFi integration, the work starts with a written scope that names the flows, topics, authentication method and processing model before any build begins.
How we scope
- Inventory the NiFi flows, Kafka topics, consumer groups and downstream systems that the integration must serve.
- Decide record-oriented or per-message processing for each flow, with the reasons recorded in the written scope.
- Agree the offset commit, duplicate handling and exactly-once expectations for each path before engineering starts.
How we build
- Configure the Kafka connection controller service with the authentication mechanism and TLS settings your Kafka cluster enforces.
- Build ConsumeKafka and PublishKafka flows with schema handling, error routes and back-pressure tuned to the agreed topics.
- Test failure cases such as broker restarts, rebalances and NiFi node loss, and record the results against the scope.
How we hand over
- Deliver versioned flow definitions, controller service settings and runbooks covering offsets, retries and replay.
- Walk your platform and integration teams through the design so they can change flows without relying on the original build.
- Document any item the scope left to your environment, such as network paths, secrets storage or NiFi cluster coordination.
How we support
- Engage for ongoing support, upgrades and testing under a retainer when your team wants continued coverage after handover.
- Plan NiFi and Kafka version changes against the release notes for each product before they reach production.
- Confirm with your team which releases, Java runtimes and NiFi versions are in use before any upgrade work is quoted.
