How to add and decommission Apache NiFi cluster nodes safely
This section explains how to change the node count of an Apache NiFi cluster without losing data or stalling flows. A team whose throughput has outgrown its cluster, or whose hardware is ageing out, needs two things: a join procedure that brings new nodes onto the current flow, and a removal procedure that moves queued FlowFiles off a node before it leaves. Adding a node is mostly a configuration and flow-synchronization task. Removing one is where data risk sits, because the node may hold FlowFiles in its queues and content in its content repository that must be rebalanced to the nodes that remain. Every node in a NiFi cluster carries the same flow and participates in coordination, whether through an embedded or external ZooKeeper or through Kubernetes state. When you add a node, it must start with a compatible configuration and join the cluster, after which it receives the current flow and begins sharing work. When you remove a node, the cluster must first stop that node's processors, stop remote process group transmission and redistribute its FlowFiles. The offload step handles this for a connected node, while a disconnected node that will not return needs to be removed in the Cluster dialog. Each path has different consequences for in-flight data, so the choice should be made deliberately. Teams usually need this work for four reasons: capacity growth, hardware retirement, failed hosts and Kubernetes rescheduling, where pods may move between nodes during upgrades or node maintenance. Before the first change, confirm with your own runbook which of these you are handling, how much queued data each node holds, and whether load-balanced connections will spread work to new members. The NiFi System Administrator's Guide documents the cluster settings, and the Apache NiFi User Guide describes the Cluster dialog used for disconnect, offload and delete.
How to add a node to an existing NiFi cluster
This sequence brings a new NiFi node into a running cluster with the same version and flow, so it joins as a connected member that receives the current flow. Each step ends with a check you can run before moving on.
Confirm version and flow match the cluster
Before touching the new host, confirm that it will run the same NiFi version as the existing nodes, using the same Java runtime and the same build. Compare the version shown in the UI or the
nifi-apiversion reported by the existing nodes with the package you have staged. Do not add a node on a different minor version, because the cluster will refuse or misbehave when versions differ. The check is a matching version string on the staged package and on a current node.Copy cluster configuration to the new node
Apply the cluster settings to the new node's
nifi.properties: the cluster flag, the node protocol address and port, and the coordination settings. Choose one coordination model that matches the existing cluster, either embedded or external ZooKeeper, or Kubernetes state through ConfigMaps or Leases. Use the same security keystores and truststores as the other nodes so the node can authenticate. The check is a diff of the node's cluster properties against an existing node, where only host-specific values differ.Start the new node
Start NiFi on the new node, or roll out the new pod on Kubernetes, and watch the application log for the cluster join. The node should report that it is connecting to the cluster and then that it has connected. If it does not, check for certificate errors, unreachable coordination endpoints or mismatched cluster identifiers before retrying. The check is a log line showing the node connected to the cluster, with no repeated join failures.
Confirm the node in the Cluster dialog
Open the Cluster dialog in the NiFi UI from a node that was already connected. The new node should appear with a connected status, and its address should match the host you configured. Count the connected nodes against the number you expect. If the node shows as disconnected or is missing, do not continue to the flow check until the status is resolved. The check is that every expected node is listed as connected.
Check that the node receives the current flow
Once the node is connected, confirm that it loaded the same flow as the rest of the cluster. Compare the process group structure, the processor states and the parameter contexts on the new node with those on an existing node. Confirm that the new node is not running a stale flow from an earlier copy of the flow configuration. The check is a matching flow structure and matching component versions across the connected nodes.
Verify the node takes its share of work
Watch the queues and the status history on the new node for a period of normal load, and confirm that it processes FlowFiles and does not sit idle. Also check that load-balanced connections include the new node. Note that existing nodes do not automatically move data to the new node until all nodes have connected as expected, so confirm the cluster state before reading the results. The check is that the node shows processing activity and is listed as a target in load-balanced connections.
Disconnect, offload or delete: which step fits the node
A node that is still connected and working can be retired in a controlled sequence. You disconnect it first so the cluster stops sending it new work, then offload it so its processors stop and its queued FlowFiles move to the nodes that remain connected. Offloading is the step that moves data, so it is the one to plan around. Do the work in a quiet window if the node has heavy queues, and watch the queue counts drop before you go further.
- Disconnect first: Disconnecting stops the node from taking part in cluster work but keeps it in the cluster membership list.
- Offload second: Offloading stops and terminates the node's processors, stops remote process group transmission and rebalances its FlowFiles to the connected nodes.
- Confirm the queues drain: Check that queued FlowFiles on the node fall toward zero before you remove it.
- Check primary node duties: If the node was primary, confirm the primary role has moved and that primary-only processors are running elsewhere.
- Remove only after offload completes: Delete the node from the cluster only when offloading has finished and the node holds no data you need.
Before you start, confirm that the remaining nodes have enough disk and heap headroom to absorb the moved FlowFiles.
A disconnected node may still hold FlowFiles in its repositories, and those FlowFiles are not processed by the rest of the cluster while the node is away. Offloading is the step designed for this case. It stops and terminates the processors on the disconnected node, stops remote process group transmission from it, and rebalances its FlowFiles to the connected nodes. The node's own disk and repository state determine how much data can be recovered, so check the node's condition before you act.
- Check the node's repositories: Confirm the content and FlowFile repositories are intact on the host, if the host is reachable.
- Understand what offloading stops: Processors on the node stop and terminate, and remote process group transmission from it stops.
- Expect rebalancing load: The connected nodes take on the moved FlowFiles, so check their heap, disk and back pressure first.
- Watch for partial recovery: If the node's repositories are damaged, the FlowFiles that remain there may not be recoverable through offloading.
- Record what was moved: Keep the provenance and log evidence of the offload for change records.
If the node can be brought back and reconnected, reconnecting it may be safer than offloading, so review the situation with your operations team first.
Some nodes will not come back, for example after a failed host, a destroyed virtual machine or a container that cannot be recovered. NiFi persists cluster node membership across restarts, so the remaining nodes do not redistribute data until every node in the persisted list has connected. A node that is gone therefore keeps the cluster waiting. In this case you remove it from the Cluster dialog so the cluster can operate without it. Deleting is a membership action, and it does not replace offloading when the node is still reachable and holds data.
- Confirm the node is unrecoverable: Verify with your operations team that the host will not return before you delete it.
- Understand the blocking behavior: Until the missing node is removed, data is not redistributed across the connected nodes.
- Delete from the Cluster dialog: Remove the node from the Cluster dialog in the UI so it leaves the persisted membership.
- Plan for lost FlowFiles: FlowFiles that existed only on the deleted node cannot be recovered through NiFi, so check your backups and upstream sources for replay.
- Do not delete before offloading when you can offload: If there is any chance the node is reachable, offload first.
The main risk is deleting a node that still holds unique data. Verify the node state and your recovery options before you confirm the delete.
What changes when NiFi scales on Kubernetes
On Kubernetes, the NiFi node is a pod, and pods are replaced, rescheduled and deleted by the platform. Scaling a cluster therefore depends on how state, storage and node identity are attached to each pod, and on whether the cluster's persisted membership matches what the platform is doing.
Coordination through ConfigMaps or Leases
NiFi on Kubernetes can keep cluster state and elect the cluster coordinator and primary node using ConfigMaps or Leases instead of an external ZooKeeper ensemble. Confirm which mechanism your deployment uses before scaling, because the state object is what new pods read when they join. Check that the service account has permission to read and update those objects, and that a failed update does not leave the cluster without a coordinator.
StatefulSet identity
A StatefulSet gives each NiFi node a stable name and ordinal, which is what the cluster uses to recognize a returning node. If a node comes back under a new hostname, NiFi sees a new member and the old entry stays in the persisted node list. Decide on naming and ordinal ranges before the first scale event, and document which ordinal is removed when the replica count drops.
Repository data stays with the pod
The FlowFile, content, provenance and state repositories must sit on persistent volumes that follow the pod identity. If a pod starts with an empty volume, its queued FlowFiles are gone from its point of view, and the node reconnects with no data while the rest of the cluster holds the flow state. Check the storage class's reclaim policy and volume binding mode, and test a pod delete and reschedule in a non-production cluster.
Scale-down ordering
Offloading a node moves its FlowFiles to the remaining nodes before the node is deleted. On Kubernetes, the platform may terminate a pod before NiFi has offloaded it, which is the main risk when reducing replicas. Confirm with your Kubernetes distribution whether pod termination can be sequenced or paused, how the termination grace period applies to NiFi, and whether scale-in can be run as a documented manual step.
Cluster dialog and rescheduled pods
NiFi persists cluster node membership across restarts, so a pod that is rescheduled and returns under the same identity rejoins without redistribution. A pod that is removed and never returns stays in the persisted list until an administrator removes it in the Cluster dialog. Build a runbook step that checks for disconnected entries after every scale event, so that data is not held back indefinitely.
Scaling options and when each fits
This table compares three ways to add and remove NiFi cluster nodes: hand-managed hosts, scripted or API-driven changes, and Kubernetes-managed scaling. Read each row as a question to answer for your team, since the same step can be a routine change in one model and a high-risk operation in another. The main takeaway is that the risk sits in scale-down and in node membership state, so the option you choose should match the skills you can keep on call.
| Factor | Manual host-based scaling | Scripted or API-driven changes | Kubernetes-managed scaling |
|---|---|---|---|
| Effort per change | Hands-on work on each host, including installing the same NiFi version, copying configuration and restarting the node | Effort is front-loaded in building and testing scripts, then repeated changes need less manual work | Effort goes into manifests, storage and state configuration, with the platform handling pod placement and restarts |
| Risk during scale-down | Highest chance of removing a node before offloading it, which leaves FlowFiles stranded on a host that is gone | Risk depends on the script, so disconnect, offload and delete order must be enforced and checked in code | Pod termination can outrun NiFi offload unless the lifecycle hooks and grace periods are configured and tested |
| Persisted node list | Operators must remove departed nodes in the Cluster dialog, or redistribution waits | Scripts must clear stale nodes through the API, or they block redistribution after a restart | Stale members can remain after pods are replaced, so the cleanup step must be part of the automation |
| Operational skills needed | Linux administration, NiFi cluster administration and change control discipline | Linux administration, NiFi REST API use and scripting with error handling and rollback | Kubernetes operations, persistent volume management, NiFi cluster administration and cluster-level troubleshooting |
| Best fit | Small, stable clusters with few changes and a team that already runs NiFi on hosts | Teams with repeatable change windows that want consistent, reviewed steps across environments | Teams already operating Kubernetes that expect frequent changes and have the skills to own the state model |
Security and operational considerations when changing cluster size
Every node you add or remove changes who can join the cluster, which copy of the flow and sensitive values it holds, and what happens to queued data while it moves. The cards below cover the security and operational checks that connect those changes, so each step can be reviewed before it runs in production.
Node certificates and cluster trust
A new node must present a certificate that the cluster's trust configuration accepts before it can join. Issue node certificates from the same certificate authority and with the same subject naming pattern as existing nodes, and confirm the truststore on the new host matches the one the cluster uses. Check that the node's hostname resolves consistently in every node's configuration, because a mismatch often looks like a join failure rather than a security problem.
Flow and sensitive value consistency
A joining node takes the cluster's current flow, but sensitive parameter values and the sensitive properties key must match across nodes or the node will not decrypt the flow correctly. Before adding a node, compare the nifi.sensitive.props.key value and confirm the parameter contexts resolve to the same values. Keep the flow definition under version control so any node can be checked against a known revision.
Access control for node actions
Disconnecting, offloading and deleting nodes change where data is processed, so these actions should be limited to a small set of administrators. Map them to explicit cluster-level policies in your authorization provider, and separate them from flow editing rights. Confirm that a user who can change flows cannot also remove nodes unless your change process requires it.
Audit trail for cluster changes
Record who disconnected, offloaded or deleted a node, when, and why, in the change ticket that authorized the action. NiFi's own logs and the cluster dialog show the state changes, but they do not replace a change record that your auditors can test. Retain application logs from every node so that a node removed during an incident can still be reviewed after it leaves the cluster.
Monitoring during offload and rebalance
Offloading moves FlowFiles to the remaining connected nodes, so watch queue depths, content repository usage and back pressure on those nodes while the rebalance runs. Confirm that remaining nodes have enough disk and heap headroom for the extra load before you start. Track the offloading node until its processors are stopped and its queues are empty, and pause the change if a remaining node approaches its back pressure thresholds.
Rollback and recovery plan
Write down what you will do if a node fails to join, stalls during offload or leaves the cluster with unknown data. Keep a recent backup of the flow definition and the repositories for the cluster, and know the procedure for removing a node that will not return. Practice the sequence in a non-production cluster first, so that the first time your team runs it is not during an incident.
Questions teams ask before scaling a NiFi cluster
What does it cost to add or remove a NiFi cluster node?
The cost is mostly the time of the people doing the work, plus whatever extra infrastructure the new node needs. The project itself is Apache-2.0 licensed with no paid edition, so there is no license fee to add a node. Costs come from the new host or virtual machine, storage for its repositories, the engineering hours to plan capacity and change control, and testing after the change. Recurring costs grow with each node because every node adds compute, disk and monitoring to run. Ask any partner for a quote that lists each of these components separately, states what is one-time (planning, the first join, runbook creation) and what recurs (infrastructure, monitoring, patching), and names the environments covered, such as staging and production.
How long does a NiFi node join or decommission take?
The duration depends on how much data sits on the node and how well the flow is designed, so no single figure applies. Joining a node is usually short when the host is ready and it can reach the cluster coordination service. Decommissioning takes longer because the node must stop its processors, and then its FlowFiles must be rebalanced to the connected nodes. Large queues, big content repositories and slow destinations all stretch that work. Plan a window for each step and ask for the timing assumptions behind any estimate. Do not commit to a production date until a rehearsal in a non-production cluster has shown how long offload takes on representative data.
What does my team need to provide for a cluster change?
Your team needs to supply the infrastructure, access and decisions that only you control. That means host or virtual machine capacity, network paths between nodes, the coordination method (embedded or external ZooKeeper, or Kubernetes ConfigMaps or Leases), identity and TLS material for the new node, and a named owner for flow changes. You also need a maintenance window, a rollback plan and someone who can approve removing a node that will not return. Current backups of the flow configuration and the repositories should exist before any change. Confirm these inputs before a quote is finalized, because missing access is one of the most common causes of delay.
Can our own platform team add and remove NiFi nodes?
Yes, a platform team with Linux, JVM and NiFi administration experience can perform these steps, and the NiFi Administrator's Guide documents the procedures. The work is simpler when the team has a tested runbook, a non-production cluster to rehearse on and clear rules for when a node is removed. Teams that run NiFi infrequently often find the risk sits in edge cases: a node that disconnects during offload, persisted membership that blocks redistribution, or a flow that depends on primary-node behavior. If your team has not handled these, rehearse them before a production change. Treat the decision as a capability question, not a yes or no question about NiFi itself.
What goes wrong when a NiFi node will not reconnect?
NiFi persists cluster node membership across restarts, so data is not redistributed until all nodes have connected. A node that is powered off, unreachable or misconfigured therefore holds back rebalancing for the whole cluster. The fix is to restore the node or, if it will not return, remove it in the Cluster dialog so the remaining nodes can take over its work. Removal is a judgment call because data on that node may be lost, so decide in advance who approves it and how the team checks what was left behind. Ask a partner how their runbook handles this case and whether they rehearse it.
Is it worth hiring a partner to scale a NiFi cluster?
A partner is worth bringing in when the cluster carries critical flows, the team has not run a node change before, or the change coincides with a version upgrade, a move to Kubernetes or a compliance review. DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, and most engagements are fixed-scope with scope, deliverables and timelines agreed in writing before work starts. A partner can also prepare the runbook, rehearsal and evidence, but it does not issue audit reports or certifications. Ask for a written scope that names the environments, the number of nodes and the rehearsal plan before you decide.
Does scaling NiFi on Kubernetes change the work?
Yes, the steps change because cluster state is held in Kubernetes ConfigMaps or Leases rather than ZooKeeper, and the pods are managed by the cluster scheduler. The NiFi Administrator's Guide describes Kubernetes coordination for state and leader election, so confirm your chosen approach against that documentation and your distribution. Scaling replicas up or down still needs the same offload and removal logic, and storage for each pod's repositories must survive rescheduling. Ask whether the plan covers persistent volume behavior, pod disruption settings and how a node removal interacts with autoscaling, because those are the points where Kubernetes changes the risk.
What should a scaling change plan include before it starts?
A useful plan names the trigger for the change, the target node count and the rollback point. It lists the backups taken, the steps for joining or offloading each node, the checks after each step and the person who can stop the change. Include the flow inventory, the processors that run on the primary node, the load-balanced connections that will be affected and the monitoring you will watch during offload. Ask for the plan in writing, rehearsed on a non-production copy, and reviewed by the team that owns the flows. A plan that cannot describe what happens to in-flight data is not ready for production.
Official NiFi documentation for cluster changes
Before you add or remove a node, read the cluster sections of the Apache NiFi System Administrator's Guide and the cluster workflow in the Apache NiFi User Guide. The Administrator's Guide sets out how a node joins with the cluster's current flow, how a node is disconnected and offloaded, and how it is deleted. The User Guide covers the Cluster dialog, where you see node status and where you remove a node that will not return. Treat these two documents as the source of truth for the exact order of steps, and check them against the version of NiFi you run, since procedures can change between releases.
Read the offload behavior closely before you start. Per the Administrator's Guide, offloading a disconnected node stops and terminates its processors, stops remote process group transmission and rebalances its FlowFiles to the connected nodes. Confirm with your own flow design which processors and remote transmissions will be affected, and plan a maintenance window for any flow that cannot tolerate stopped processors. Also confirm how your load-balanced connections and primary-node processors will behave while a node is absent, because those choices decide where the data goes during the change.
The User Guide explains one behavior that catches many operators: NiFi persists cluster node membership across restarts, so data is not redistributed until all nodes have connected. A node that is powered off or cannot reach the cluster therefore keeps the cluster in a partial state. Decide in advance how long you will wait for a node to return, who has authority to remove it from the Cluster dialog, and how you will record that decision. Your runbook should name these owners before the change starts, not during an incident.
