Apache NiFi and Azure Event Hubs integration

How Apache NiFi and Azure Event Hubs integration works

An Azure event stream has to reach NiFi flows, or NiFi output has to reach an Event Hub. For receiving, the component is ConsumeAzureEventHub, and for sending it is PutAzureEventHub. Both processors sit on the same Azure integration path, so the decisions you make about namespace, hub, credentials and storage apply to both directions. Before building anything, confirm which direction carries the data, which Event Hub and consumer group the flow will read from, and whether the NiFi cluster or an external system owns the producer side.

Consumption depends on checkpointing, and checkpoints must live in external storage that every node can reach. The processor records how far it has read in each partition, so a restarted or replaced node resumes from the last recorded position rather than from an arbitrary point in the stream. Plan the storage account and container for these checkpoints as a dependency of the flow, with access granted to the NiFi runtime identity and a retention and recovery owner named. Confirm the current checkpoint behavior and property names in the ConsumeAzureEventHub component documentation before you write the runbook.

In a cluster, the processor instances form a *consumer group*, so the Event Hub partitions are shared across nodes and each message is processed on one node only. This gives you parallel consumption without duplicate processing under normal operation, but it also means a node failure shifts partition ownership to surviving nodes. Test that handover with a realistic message rate, and check for duplicates or gaps at the boundary, since delivery guarantees depend on how downstream processors acknowledge work. Decide early whether the downstream system can tolerate at-least-once behavior.

What you need in place before configuring Event Hubs in NiFi

The Event Hubs processors depend on more than a working NiFi cluster. Confirm the runtime, the Azure resources, the storage that holds consumer checkpoints, the network paths and the identity method before you add a processor to a flow. Each item below is a decision or a check that is hard to change once the flow is running.

NiFi 2 on Java 21

Every node must run a NiFi 2 release on Java 21 as the minimum runtime, as stated in the NiFi System Administrator's Guide. Check the JDK on each node, not only on the build host, because a mixed-version cluster will show inconsistent behavior that is hard to trace. If you plan to use the optional Python-based processors, note that the administration guide lists Python 3.10, 3.11 or 3.12 and describes that feature as beta. Those processors are not needed for the Event Hubs components.

Event Hubs namespace and event hub

Create or identify the Azure Event Hubs namespace and the event hub that the flow will read from or write to. Decide the partition count with your producers and consumer throughput in mind, since NiFi nodes share partitions as one consumer group. Record the namespace name, the event hub name and the region, and confirm them with the team that owns the producing applications. Changing these later affects every consumer.

External checkpoint storage

ConsumeAzureEventHub checkpoints its position in external storage, so you need an Azure storage account and a container dedicated to this consumer group. Do not reuse a container that other applications write to. Ask your Azure team about access logging, soft delete and retention on that container, and plan who can read or delete checkpoint blobs. Losing checkpoints can cause reprocessing, so treat this storage as part of the flow's recovery design.

Network paths from every node

Each NiFi node opens its own connections to the Event Hubs endpoints and to the checkpoint storage, so the paths must exist from every node, not only from the primary. Confirm outbound access through your firewalls, proxies and DNS resolution in the network segment where NiFi runs. In restricted or air-gapped networks, check with your Azure and network teams how private endpoints are reached before you assume a path works. Test from each node with the same tools the processor will use.

Identity for the chosen method

Decide how NiFi authenticates to Azure before you build the flow. Options include a connection string with a shared access key, or an Azure identity with role-based access, depending on what the processor supports in your NiFi version. Create a dedicated identity, grant it the minimum Event Hubs and storage roles it needs, and store the secret in a controller service or parameter marked sensitive. Confirm the exact Azure role names and processor property names against the component documentation for your NiFi version before you grant access.

Certificates and trust for TLS

Event Hubs and storage endpoints are reached over TLS, so each node's JVM must trust the certificate chain presented by Azure. If your organization inspects outbound traffic, the inspection CA must be added to the truststore that NiFi uses, and that change must be applied on every node. Also confirm how your NiFi cluster's own TLS certificates are issued and renewed, since a clustered deployment depends on them. Record the renewal owner and the expiry dates in your change calendar.

Configuring consumption and sending in NiFi

This sequence takes an Event Hubs namespace from empty Azure resources to a NiFi cluster that consumes each message once across nodes and sends FlowFile contents back to a hub. Each step ends with a check you can run before moving on.

Create the Azure Event Hubs resources
Create an Event Hubs namespace, the event hub that NiFi will read from and write to, and a separate consumer group for NiFi so it does not share a cursor with other readers. Also create a storage account and blob container that will hold consumer checkpoints, since ConsumeAzureEventHub requires external storage for checkpointing. Record the namespace connection details, the hub name, the consumer group name and the storage container name in your deployment notes. The step is done when a test message sent from the Azure portal or another client appears in the hub's metrics and the checkpoint container exists and is empty.
Grant NiFi access with a chosen authentication method
Decide how NiFi will authenticate before configuring processors. Shared access keys are the simplest option but give broad access and need rotation. Microsoft Entra ID service principals or managed identities limit access by role and avoid stored secrets, but they need role assignments on both the namespace and the checkpoint storage account. Confirm the exact credential options exposed by the Azure Event Hubs controller service or processor properties in your NiFi version, because they differ by release. Store the secret values as sensitive parameters, not plain properties. The step is done when a test connection or a simple send from a single node succeeds using the chosen identity.
Configure ConsumeAzureEventHub with checkpoint storage
Add a ConsumeAzureEventHub processor to a canvas on a cluster-capable process group. Set the namespace, event hub name, consumer group and checkpoint storage properties to the resources from the first two steps, and keep the consumer group name identical on every node. Choose a run schedule and a batch size that suit your message rate, and route the success relationship to a placeholder sink such as a funnel or a log-only processor for now. Start the processor on a single node first. The step is done when messages sent to the hub produce FlowFiles on the success relationship and a checkpoint blob appears in the storage container after they are processed.
Confirm the consumer group works across nodes
Bring the cluster to at least two nodes, which is the minimum you need to test sharing behavior, and start ConsumeAzureEventHub on the cluster. Because the processor instances form a consumer group, each message should be processed by one node only. Send a numbered test set of messages, for example a batch of distinct identifiers, and compare the output from every node. Duplicates or gaps indicate a configuration problem, such as differing consumer group names or checkpoint containers across nodes. Stop one node during the test and confirm that the remaining nodes resume from the stored checkpoint. The step is done when the identifier set appears once across all nodes with no missing entries.
Validate checkpoint recovery and processing semantics
Restart the NiFi cluster and the processor while test messages are arriving, then check which messages are reprocessed. Checkpoint behavior determines whether a restart replays recent messages, so record the observed result and design downstream sinks to tolerate duplicates where needed. Review the processor documentation for the exact checkpoint and delivery semantics in your version before depending on them for regulated data. Confirm that the checkpoint container holds one blob per partition and that the blob updates as messages are processed. The step is done when you have a written record of what a restart replays and a sink that handles that outcome.
Add PutAzureEventHub for the send path
Add a PutAzureEventHub processor that reads a FlowFile from an upstream source such as a test generator or a GenerateFlowFile processor. Point it at the same namespace and hub, or at a separate send-only hub if you want to keep test traffic apart from production consumers. Use the same authentication choice as the consume path so that role assignments and secrets remain consistent. Route success and failure relationships to a visible sink so that rejected sends are not lost. The step is done when a FlowFile produced in NiFi arrives at the hub, which you can confirm by consuming it with a second consumer group or the Azure portal metrics.
Test failure paths before production traffic
Introduce controlled faults to confirm how the flow behaves: an invalid credential, a missing hub name and a revoked role assignment. Observe where the failure appears in the processor bulletins and which relationship the FlowFile takes. Confirm that the failure relationship reaches a queue or alert that an operator watches, and that retries do not create duplicate sends unless you intend them. Record each fault and its observed result as part of the handover notes. The step is done when every injected fault produces a visible, routed outcome and no FlowFile is silently dropped.
Promote the flow with parameters and secrets
Move the namespace, hub, consumer group and storage settings into NiFi parameters so the same flow can be promoted between test and production clusters without editing processors. Mark credential parameters as sensitive and supply their values from your secrets process rather than from the flow definition. Version the process group in your chosen flow registry before promotion so changes are traceable. Confirm the exact parameter and secret behavior in the NiFi administration guide for your release. The step is done when the flow starts on a second cluster using only the parameter values for that environment and the consume and send checks pass again.

Choosing an authentication method for Event Hubs access

Managed identity lets the Azure resource hosting a NiFi node obtain tokens from Microsoft Entra ID without a secret stored in NiFi. It fits nodes running on Azure virtual machines or Azure Kubernetes Service where the identity can be assigned to the host. Access is granted by assigning the identity a data role on the Event Hubs namespace or hub, so permissions stay in Azure role-based access control.

Secret rotation: No client secret or key sits in the flow or a controller service, so there is no NiFi-side credential to rotate on a schedule.
Audit trail: Sign-ins and Event Hubs operations are attributed to the managed identity in Azure logs, which shows which host accessed the hub.
Ownership: The platform team owns the identity and role assignments, so application owners cannot widen access from inside the flow.
Scaling cost: Every new node needs its identity assigned, so cluster growth adds a step to the provisioning runbook.
Environment limit: The option applies only to NiFi nodes on Azure hosts that support managed identities, so it does not cover on-premises or other-cloud nodes.

Confirm that the Azure credentials options on your NiFi version include managed identity, and that your hosting pattern can assign the identity to every node.

How Event Hubs fields map to NiFi FlowFile attributes and content

This table compares each Azure Event Hubs element with where it lands in a NiFi FlowFile on the consume path and where it goes on the send path. Read it as a checklist for designing downstream routing, because attributes drive routing and content carries the payload. Confirm exact attribute names in the ConsumeAzureEventHub documentation for your NiFi version before you write expressions that depend on them.

Event Hubs element	Consume path (ConsumeAzureEventHub)	Send path (PutAzureEventHub)
Message body	Becomes FlowFile content as received, so parsing and record handling happen downstream	FlowFile content is written back as the message body
Partition identifier	Recorded as a FlowFile attribute, useful for tracing which partition a record came from	Not a content field; confirm in the processor documentation how partition targeting is configured
Offset	Recorded as an attribute that identifies the position of the message within its partition	Not set by the sender; the service assigns position on write
Sequence number	Recorded as an attribute, useful for ordering checks and duplicate investigation	Not set by the sender; confirm whether it is exposed on write
Enqueue time	Recorded as an attribute, useful for latency measurement and time-based routing	Not set by the sender; the service records it on arrival
Application properties	Mapped to FlowFile attributes, so validate names and avoid collisions with NiFi attributes	Typically taken from FlowFile attributes; confirm which attributes are sent as properties
Checkpoint state	Kept in external checkpoint storage, not in the FlowFile, so restarts resume correctly	Not applicable to sending

Securing the Event Hubs connection in a NiFi cluster

The Event Hubs connection carries credentials, data and operator access, and each needs its own control. The cards below cover Azure permissions, how secrets reach processors, who can change the flow, transport encryption and how NiFi authentication limits operators. Work through them together before the first consumer starts.

Scope Azure roles to each identity

Grant the consumer identity only the data-plane rights it needs to receive from the Event Hub and write checkpoints, and grant the sender identity only send rights on the hub it publishes to. Scope these roles to the specific Event Hubs namespace or hub rather than the subscription. Confirm the exact role names and the checkpoint storage permissions with your Azure administrator, because they depend on your authentication method and tenant policy.

Keep connection secrets out of flow definitions

Store connection strings or client secrets in NiFi sensitive properties, or retrieve them through a parameter provider backed by your secret manager, so they are not visible in exported flow definitions. Check that the sensitive-value key is protected and backed up, because losing it makes stored values unreadable. Confirm which property types your processor version treats as sensitive before you rely on them.

Restrict who can view or change processors

Use NiFi policies to separate users who can view the flow from those who can modify processors, controller services and parameter contexts. Processors that hold Event Hubs credentials should sit in a process group only the platform team can edit. Review policies after each change, because a user with write access to a controller service can often redirect the connection.

Encrypt the UI and cluster traffic

Enable TLS on the NiFi UI and API, and on node-to-node traffic, so credentials and flow content are not exposed on the network between nodes or to browsers. Without a security configuration NiFi binds to 127.0.0.1, so remote access requires you to set this up deliberately. Confirm your certificate authority chain and renewal process before the cluster goes live.

Choose an operator authentication provider

NiFi supports single-user, LDAP, OpenID Connect, SAML 2.0 and X.509 client certificate authentication, so pick the one that matches how your organization manages identities. Confirm that the chosen provider removes access when an employee leaves. The Kerberos provider is deprecated, so avoid it for new clusters and plan a move off it for existing ones.

Separate consumer and sender identities

Use different Azure identities, and different NiFi credential entries, for the flow that consumes from an Event Hub and the flow that sends to it. A shared identity makes it impossible to tell in Azure logs which side made a call and gives the consumer send rights it does not need. Record which identity each controller service uses in your change documentation so auditors can trace access.

How partitions and checkpoints decide ordering and duplicates

Ordering in an Event Hubs flow is guaranteed only within a single partition, and the consumer group decides which NiFi node reads which partition. When the ConsumeAzureEventHub instances in a cluster share one consumer group, they divide the partitions among themselves, so each message is handled by one node. Two messages from different partitions have no defined order when they reach your downstream systems, even if they were sent seconds apart. Before go-live, confirm with your application team which keys producers use for partition assignment, because that key determines which events stay in order. Confirm too how many partitions the hub has and how the cluster will spread them when a node joins or leaves, and record the answer in your design notes.

Checkpoints, not NiFi's own queues, decide what is replayed after a restart or a node change. ConsumeAzureEventHub stores its progress in external checkpoint storage, so the position a new owner resumes from depends on the last checkpoint written, not on the last FlowFile that left the processor. Events read after the final checkpoint but before a crash or a rebalance can be delivered again. The checkpoint write frequency and the point at which the processor commits relative to your downstream write are settings to confirm against the processor documentation for your NiFi version, not assumptions to carry into production.

Design the downstream path to accept redelivery, and prove the replay behavior in a test before go-live. Use an idempotent key from the event, such as a producer-assigned identifier or a partition and sequence combination, so that a repeated write updates rather than duplicates the target record. In your test plan, stop a node mid-batch, force a rebalance by adding or removing a node, and count what arrives twice at the destination. Repeat the test under realistic load and keep the results with your change records. Treat any duplicate you cannot explain as a defect in the design, not as an acceptable cost of the platform.

Common Event Hubs integration failures and their causes

This table lists the failures teams most often hit when consuming from or sending to Azure Event Hubs with NiFi, with the likely cause and the first fix to check for each. Use it as a triage starting point, then confirm exact error messages against the processor's logs and the Azure portal in your own environment.

Failure	Likely cause	Fix to check first
Consumer restarts from the beginning of the stream	Checkpoint storage is missing, unreachable or not persisted, so no prior position is recorded	Configure external checkpoint storage for ConsumeAzureEventHub and confirm the processor can write to it from every node
Consumer does not start or cannot read checkpoints	Storage account credentials or role assignments do not allow the processor to read and write checkpoint blobs	Grant the identity or key used by the processor the data access it needs on the checkpoint container, then restart the processor
Authentication failures when connecting to the hub	Wrong namespace, hub name, shared access policy or Microsoft Entra ID principal, or an expired secret	Recheck the namespace and event hub names, the chosen auth method, and rotate or re-enter secrets through a parameter or controller service
Authorization errors on send or receive	The identity lacks the Azure Event Hubs send or receive role, or the shared access policy has the wrong rights	Assign the send or receive role at the right scope, or use a shared access policy with only the rights the flow needs
Connection timeouts or no messages arriving	Outbound network path to the Event Hubs endpoint is blocked by firewall, proxy or private DNS settings	Test egress from each node to the namespace endpoint, then review firewall rules, proxy settings and private endpoint DNS
Every node processes the same partition or duplicates appear	Nodes use different consumer group names, or checkpoint storage is shared incorrectly across separate flows	Use one consumer group per flow, keep the same processor configuration on all nodes, and give each flow its own checkpoint container
Replay of messages after a node or cluster restart	Checkpoints were not committed before the restart, or the offset reset setting replays earlier events	Confirm checkpoint writes succeed, design downstream steps to be idempotent, and document the expected replay window
PutAzureEventHub fails on large or invalid content	FlowFile content exceeds the size the hub accepts, or the payload is not in the format the consumer expects	Split or compress large FlowFiles upstream, validate the content format before sending, and route failures to a retry or dead-letter path
Send failures with no clear error	Batch settings combine several FlowFiles into one request that exceeds limits, or a single bad record fails the batch	Reduce batch size, isolate malformed records with validation, and check the processor's failure relationship and bulletins

Questions buyers ask about Event Hubs integration effort and ownership

What drives the cost of an Event Hubs integration with NiFi?

The main cost drivers are the number of event hubs and consumer flows, the partition count and message throughput, the authentication design, the checkpoint storage design, and the size of the NiFi cluster that will run the processors. Each additional hub or consumer group adds flows to design, test and operate. Higher throughput raises the sizing and tuning work on NiFi and on the storage account used for checkpoints. Ask any partner to price these items separately in the written scope, and to state which parts are one-time build work and which parts recur, such as monitoring, upgrades and retainer support.

How long does an Event Hubs integration take to deliver?

Duration depends on how many hubs and flows are in scope, how complex the transformations are, how many authentication paths must be validated, and how quickly your Azure and network approvals arrive. Teams often lose more time waiting for access and firewall changes than building flows. A fixed-scope plan should show the build sequence, the test windows and the cutover steps, with dates tied to the client inputs listed in the scope. Ask for the timeline as a dependency list so you can see which delays belong to you and which belong to the partner.

What does our team need to provide before the work starts?

You need to provide Azure access for the people doing the work, including the event hub namespaces, a storage account for checkpoints, and the identities or service principals that will be granted rights to them. You also need network paths between your NiFi nodes and Azure, which may include outbound rules, private endpoints or proxy settings. Finally, name the flow owners who will approve designs, own the data contracts and decide on replay and retention rules. Missing any of these usually stalls the build, so confirm each item in writing before the scope is signed.

Can an internal team build and run this integration alone?

An internal team with NiFi, Azure and networking experience can build a basic integration alone, especially for a single hub with simple routing. The harder parts are usually cluster behavior, checkpoint design, authentication choices and error handling, because mistakes there can cause duplicate or missed processing that only shows up under load or during node restarts. Decide whether your team has someone who has already run a clustered NiFi consumer group. If not, a partner can design and verify the flows while your staff learn to operate them.

What is included in a fixed-scope quote for this integration?

A fixed-scope quote should list the hubs, consumer and producer flows, authentication method, checkpoint storage configuration, test cases, documentation and handover sessions that are included. It should also state what is excluded, such as Azure resource provisioning, changes to upstream producers or downstream systems, and ongoing support. Ask for the acceptance criteria in writing, including how message loss, duplicates and failover will be tested. Scope, deliverables and timelines are agreed in writing before work starts, so anything not listed should be requested as a change.

Which authentication choice affects effort the most?

The choice affects effort mainly through how credentials are issued, rotated and stored across all cluster nodes. Connection strings with shared keys are quick to configure but need careful secret handling and rotation. Identity-based approaches fit better with central Azure governance but require role assignments and validation on every node. Confirm with your Azure administrators which method your security policy allows, and ask the partner to show how the chosen method is tested for each node and after a credential change.

What ongoing work follows the initial integration?

After go-live, the recurring work includes monitoring consumer lag and processing errors, rotating credentials, reviewing checkpoint storage, applying NiFi and processor updates, and adjusting flows when producers change schemas or partition counts. Ask whether these tasks are handed to your operations team with runbooks, or whether they are covered by a retainer for support and upgrades. Either way, the handover should name who is on call for the flows and how a failed node or stalled consumer group is recovered.

How DeployOpen scopes and delivers Event Hubs integration

DeployOpen plans, builds, tests and documents Apache NiFi flows that consume from and publish to Azure Event Hubs, inside infrastructure you control. Each engagement starts from a written scope that names the event hubs, consumer groups, checkpoint storage and authentication method before any flow is built.

How we scope

Inventory the Event Hubs namespaces, event hubs, partitions and consumer groups the flows will use.
Agree the authentication method for each NiFi node and each flow, including identity and secret ownership.
Define message format, record readers and writers, and how duplicate or out-of-order messages are handled downstream.
Confirm the checkpoint storage account, retention needs and expected cluster size with your Azure and NiFi teams.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.

How we build

Configure ConsumeAzureEventHub with checkpoint storage and verify that cluster nodes share partitions as one consumer group.
Build PutAzureEventHub flows with explicit error routing so failed sends are retried or parked, not lost.
Parameterize namespaces, hub names and credentials so the same flow promotes cleanly between environments.
Test failover by stopping a cluster node and confirming that partition ownership moves and checkpoints resume.
Version flows in a Git repository where your change process requires it, using NiFi's Git-based flow registry clients.

How we hand over

Deliver the flow definitions, parameter contexts, controller service inventory and checkpoint storage layout as documentation.
Document the authentication setup, role assignments and how secrets are rotated without restarting the cluster.
Provide a runbook covering consumer lag checks, failed send investigation and checkpoint reset procedures.
Walk your operations team through the test results, known limits and the decisions recorded during the build.
Confirm acceptance against the written scope before the engagement closes.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Plan NiFi version upgrades and re-run the Event Hubs flow tests before each change reaches production.
Review Azure-side changes such as partition count, retention or identity policy that affect the flows.
Confirm with your team which support hours, response paths and escalation contacts the retainer should include.