Why NiFi list processors run on the primary node only
List processors such as ListSmb run on the primary node only, and this keeps a single listing state from being written by several nodes at once. A team that builds a flow to list files on an SMB share, an SFTP server or a cloud drive often tests it on one node, sees files appear, and then finds that the same flow in a cluster either lists nothing new or produces duplicates. The cause is not the connection. Each list processor records how far it has read, such as the last modification time it has seen, and that record must have one owner. If every node listed the same location and kept its own record, each node would emit the same files, and the cluster would process them several times.
You need this knowledge whenever a cluster deployment pulls from SMB, SFTP or cloud drive sources. The primary-node behavior described here comes from the NiFi ListSmb documentation, which states that a new primary node picks up where the previous one left off without duplicating data. Confirm the equivalent behavior for each other list processor you plan to use, such as ListSFTP or ListGoogleDrive, in that component's own documentation, because listing state and execution settings can differ between components. In practice, the flow designer should schedule the list processor on the primary node, verify in the cluster view that it is not running on every node, and plan how a failover will affect the listing interval and the downstream work already in flight.
The downstream consequence is that the list processor's output is produced by one node but the files are fetched and processed by many. NiFi includes a flow analysis rule, RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor, that flags a connection leaving a primary-node source when it is not load balanced. Treat that rule as a design check, not a formality: without a load-balanced connection, all FlowFiles stay on the primary node and its disk, heap and network become the throughput limit. Ask your implementation partner or platform team to confirm the rule is enabled in your development environment and to review each list processor's outgoing connection before the flow reaches production.
How state carries over when the NiFi primary node changes
When the primary node is lost, the cluster elects a new one, and a list processor such as ListSmb continues from the state the previous primary left behind. According to the ListSmb documentation, the new primary picks up where the old one stopped, so the listing does not restart from the beginning and does not duplicate data already listed. That guarantee depends on the state being stored where every node can read it after an election, so the first task is to confirm where that state lives in your coordination setup. In a cluster using embedded or external ZooKeeper, confirm how the state is written and replicated across ZooKeeper members. On Kubernetes, confirm whether the cluster uses ConfigMaps or Leases for state and leader election, and who can read and modify them.
Two things are not settled by the documentation and must be confirmed in your environment. First, how long a failover takes: measure the interval between losing the primary and the new primary running the list processor in a test cluster that mirrors your network, coordination service and load, and do not assume a figure from another deployment. Second, what happens to a listing that is in progress when the primary fails. Ask your engineers to test a failover while a large remote directory is being listed, then compare the files emitted downstream with the files present at the source. Check for gaps and duplicates, and confirm that any partially listed batch is handled by your downstream flow.
Plan for the failure case before it happens. Record which coordination option you run, the names of the state and election resources, and the procedure for restoring them after a coordination outage. Add a provenance and log review to the failover test so you can show, after the event, which node listed which files. Keep these results with the flow's change records, because they are the evidence an operations team will need when a listing is questioned after a primary change.
Which list processors follow primary node scheduling
Listing processors record what they have already seen in cluster-wide state, so only one node may run them at a time. The cards below cover the families most teams build with and the checks each one needs before it goes into a clustered flow.
SMB share listings
ListSmb polls a Windows or Samba share and emits one FlowFile per file found, with attributes such as the file name, path and size. Because its state records which files it has already listed, two nodes listing the same share would emit duplicates. Confirm its scheduling behavior on the NiFi components index for the version you run, and check that the share path and credentials are set through parameters.
SFTP directory listings
ListSFTP lists files on a remote SFTP server and produces FlowFiles that carry the remote path and timestamps as attributes. The downstream fetch step reads the content only for files the listing has returned. Plan the listing state as one owner for the whole cluster, and decide how files that are still being written on the server are excluded before they are picked up.
Google Drive listings
ListGoogleDrive enumerates files in a Drive folder or drive through the Google API, using the modification time and file identifiers as its listing state. Its output is metadata only, so a fetch processor must follow to move content. Confirm the scheduling behavior and the quota handling for your account type on the component documentation before you size the poll interval.
Object storage and queue listings
Cloud storage listings and queue-style sources share the same ownership question: which node holds the cursor. Treat each one as a separate component with its own documented behavior, and do not assume that a list processor in one bundle schedules the same way as one in another. Check the NiFi 2 component documentation for each processor you add, and record the result in your flow design notes.
Custom and scripted listing logic
A custom processor or scripted component that polls an external system and tracks its own cursor will not get primary node behavior unless the code is written for it. Without that, every node runs the poll and each one records different state. Review how your component stores state, and test with two nodes in a sandbox cluster before you rely on it in production.
Components with a documented primary node setting
Some processors expose a scheduling strategy that you can set to run on the primary node, and some sources are designed to run that way by default. Read the scheduling section of each component's documentation and set the strategy in the flow, not on one node by hand. Document the decision so that a later reviewer knows why a source runs on one node only.
Scheduling choices for listing and fetching in a cluster
This table compares three ways to feed listed work into a NiFi cluster: a primary-node list followed by a load-balanced fetch, a primary-node list feeding one node, and an event-driven source that avoids listing. Read each column against your own volume, failover tolerance and source system, since the right choice depends on what the source supports and how much duplicate or delayed work you can absorb.
| Design | How work is spread | What happens on primary failover | Flow analysis rule outcome | When it fits |
|---|---|---|---|---|
| List on primary, load-balanced fetch | Listing runs on the primary node only; the listed FlowFiles are spread across nodes by a load-balanced connection before fetch processors run | The new primary node resumes listing from the stored state, so listing continues without duplicating entries already recorded | The RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor rule is satisfied because the connection after the list processor is load balanced | Large file sets where fetch and transform work must scale across all nodes and the source is a polled file or share |
| List on primary, single node downstream | Listing and all downstream work stay on the primary node, so one node carries the fetch and transform load | The new primary resumes listing from stored state, but the downstream work moves with the primary node, which can shift load abruptly | The rule expects a load-balanced connection after a primary-node source, so this design usually needs a documented reason to omit it | Low volume flows, or flows where downstream steps must run on one node and the team has confirmed that in its design |
| Event-driven intake without listing | Each node receives pushed events or messages from the source, so no primary-node listing step is needed | No primary-node listing state is involved, so failover does not reset a listing position; delivery guarantees depend on the source and receiving processor | The listing rule does not apply because there is no list processor on the primary node | Sources that push events, such as message queues or webhooks, where polling latency or listing cost is the main concern |
| Scheduling note | Confirm which processors your NiFi version schedules on the primary node in the component documentation before designing | Test a forced primary change in a staging cluster and check that no entries are duplicated or skipped | Run flow analysis on the draft flow and review every warning against the design decision | Record the chosen design and the reason for any suppressed rule so reviewers can trace it |
How to set up a list processor with a load-balanced connection
This sequence takes a clustered NiFi from a working coordination setup to a list processor that runs on the primary node, hands its output to the fetch step over a load-balanced connection, and keeps listing correct after a primary node change.
Confirm the cluster and coordination method
Before adding any list processor, confirm that the NiFi nodes form one cluster. Check that the cluster coordination method is working, whether that is embedded or external ZooKeeper, or Kubernetes ConfigMaps or Leases if you run NiFi on Kubernetes, as described in the NiFi administration guide. In the UI, open the cluster view from the global menu and confirm every intended node shows as connected and that one node is marked as primary. If a node is missing or disconnected, resolve that first, because listing state and primary election depend on the cluster working.
Add the list processor and set primary node scheduling
Add a list processor such as ListSFTP, ListSmb or ListGoogleDrive to the canvas and configure its source connection details. Open the processor's Scheduling settings and set the execution option to run on the primary node, so only one node lists the source at a time. Confirm the processor's configuration validates without errors. The step is complete when the processor shows the primary node execution setting and does not report invalid properties. Keep it stopped until the rest of the flow is wired.
Connect the fetch processor with a load-balanced connection
Add the matching fetch processor downstream, for example FetchSFTP or FetchSmb, and draw the connection from the list processor. In the connection settings, set the load balance strategy so FlowFiles from the primary node are distributed across the cluster. Choose the distribution strategy that fits the fetch work, such as round robin for even spread, or partition by attribute or single node where ordering or affinity matters. The check is that the connection shows a load-balanced indicator on the canvas and that the strategy you chose is visible in the connection configuration.
Run the flow analysis rule and resolve the warning
Open the flow analysis rules in the global settings and enable RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor, as listed in the NiFi components index. Run flow analysis on the process group. If the rule reports that the connection after the primary node source is not load balanced, return to the connection and set the load balance strategy, then rerun the analysis. The step is complete when the analysis shows no open warning for that connection. Record the analysis result with the flow version so a reviewer can see it was checked.
Start the flow in a non-production cluster
Deploy the flow to a non-production cluster that matches the production topology as closely as you can. Start the list processor and the fetch processor, then confirm in the processor status that the list processor runs on the primary node only. Check the connection queue and the cluster view to see FlowFiles spread across nodes. Use the data provenance view to confirm that each listed object was emitted once. Keep a list of the source file names or identifiers so you can compare them later.
Change the primary node and confirm listing resumes
In the non-production cluster, trigger a primary node change by stopping the current primary node or by restarting it, as your environment allows. Watch the cluster view until a different node is marked as primary, then check that the list processor now runs on that node. Compare the new listing output with your recorded list. The check is that listing continues from the point the previous primary left off and that no object appears twice in provenance. If duplicates or gaps appear, review the processor's state and the source listing settings before you move to production.
Document the result before promoting the flow
Write down the coordination method, the primary node scheduling setting, the load balance strategy, the flow analysis result and the failover test outcome. Store these notes with the flow version in your change record. If you use a Git-based flow registry, commit the flow version that passed the checks so production deploys the same definition. The step is complete when a colleague who did not run the test can reproduce each check from the notes and reach the same result.
What flow designers and platform operators each need to decide
Designers decide where listing happens and what consumes its output. Start by asking whether the source must be polled with a list processor at all. A list processor records what it has already seen and runs on the primary node only, which suits polling a remote directory or share. If the source can send notifications, or the team can use a different fetch pattern, compare that option with listing and confirm which event-driven alternatives exist in your NiFi version.
- Choose list or event-driven sources: Polling fits sources that can be listed reliably. Confirm whether the source offers notifications before accepting the polling interval and the load it adds to the source system.
- Place the load-balanced connection: Connect each list processor to a downstream connection with a load-balanced strategy so FlowFiles spread across nodes before fetch and transform steps. Without it, that downstream work stays on the primary node.
- Read flow analysis results before deployment: The RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor rule flags a primary-node source with no load-balanced connection after it. Treat each finding as a blocking review item, not a cosmetic warning.
- Confirm how state reacts to edits: Ask the component documentation whether changing a source path or credential parameter resets listing state, and test that before production.
- Test on more than one node: A single-node test shows neither distribution nor what happens when the primary changes.
Before deployment, check that every list processor has a load-balanced connection after it, that the analysis report is clean, and that path and credential parameters match the target environment.
Operators decide how the cluster elects and tracks the primary node, and how they will notice when it changes. The NiFi administration guide describes coordination through ZooKeeper, either embedded or external, or on Kubernetes through ConfigMaps or Leases for state and leader election. The choice determines what must be monitored, backed up and patched, so make it before the first production flow starts.
- Choose the coordination method: Compare embedded ZooKeeper, external ZooKeeper, and Kubernetes ConfigMaps or Leases. Confirm which options your NiFi version and deployment model support, and name the team that owns that service.
- Monitor the primary node: Alert on primary changes and on which node holds the role. Without this, list processors can sit idle for a long time without anyone noticing.
- Plan for downstream queues during a primary change: Listing resumes on the new primary, so downstream connections may see a pause followed by a burst. Confirm that queue thresholds and back pressure settings can absorb that burst on your hardware.
- Verify state carryover in a test: The ListSmb documentation says a new primary node picks up where the previous one left off without duplicating data. Confirm this against your own source system and processor type before relying on it.
- Record runbook steps: List every primary-only processor, the expected behavior after a failover, how to confirm listing has resumed, and who to call.
Before go-live, check that primary change alerts fire, that the runbook covers each primary-only processor, and that a failover test has been run on a cluster that matches production.
Some decisions belong to neither role alone. The designer knows the flow logic and the operator knows the failure behavior, so the change should carry both views into one review. Agreeing these points in writing before deployment prevents a situation where each side assumes the other has covered a risk.
- Assign owners for each primary-only processor: Name the person who approves changes to each list processor, its path, and its schedule. Each one should map to a runbook entry.
- Attach the flow analysis output to the change record: Store the analysis result with the version of the flow that produced it, so auditors and reviewers can see what was checked.
- Define acceptable failover results: Agree in writing what duplication, delay and queue growth the business accepts during a primary change. Without an agreed limit, a failover test has no pass or fail result.
- Test with the source system's real behavior: Run a failover with a representative source and confirm how the source reports new items, deletions and changed timestamps.
- Set escalation paths: Decide who responds when listing stops, who can pause downstream flows, and who approves a restart.
Before the change is approved, check that both roles have signed the same list of primary-only processors and the same failover criteria.
Security and operational risks of a moving primary node
A list processor runs on whichever node is primary at the time, so a failover changes where fetches originate, which credentials they use and which queues they feed. The cards below cover the risks that follow from that behavior and the checks to make in your own cluster before the flow goes live.
Duplicate or missed files after failover
A list processor stores its listing state, and the new primary resumes from that state so it should not duplicate data. The risk comes from state that was never persisted, state reset by a flow change, or a listing that was edited while a failover happened. Test a forced primary change with a known set of files and count what arrives downstream. Confirm how your processor stores state, and whether downstream steps can tolerate a repeat.
Source credentials on the primary node
Every fetch step after a list processor runs on the node that is currently primary, so the source credentials are used from that node. If the service account has broader rights than the listing needs, a compromise of any node that can become primary exposes the whole share or bucket. Scope the account to read-only access on the required paths, store the secret in a sensitive parameter or approved secret provider, and confirm which nodes can take the primary role.
Network paths from every node
Because the primary can move to any node, every node needs a route to the source system, not only the node you tested from. A firewall rule or DNS entry that exists on three nodes but not the fourth will work until a failover lands on the fourth. Check connectivity from each node with the same protocol and port, including TLS certificate trust, and repeat the check after node replacement or subnet changes.
Effect of a primary change on queued work
FlowFiles already queued on the old primary stay with that node and are not moved by the list processor's state transfer. Downstream work can pause or arrive out of order while the new primary starts listing. Check where queues sit during a failover, how back pressure behaves on the connections after the list processor, and whether a load-balanced connection is in place to spread the listed work across nodes.
Monitoring which node is primary
Operators need to know which node currently runs the listing, or a stalled source can look like a healthy flow. Use the cluster view and the processor's node indicator, and alert when a list processor has produced no new FlowFiles for longer than the expected polling window. Confirm that your monitoring stack records primary role changes, and that the on-call team knows where to look when the primary moves.
Change control for unchecked flows
The flow analysis rule that requires a load-balanced connection after a primary-node source only applies when it is enabled and the flow is analyzed. A change that adds a list processor, moves it into a new process group or bypasses the analysis step can reach production without the check. Make the rule part of the review checklist, record the analysis result with each flow version, and confirm that every production change passes through the same review path.
Questions teams ask before changing a list processor in a cluster
Can our own team set up list processors on the primary node?
Yes, if your engineers already run NiFi clusters and have access to the flow, the source system and a test environment. Setting a list processor to run on the primary node is a scheduling choice in the processor configuration, and the change itself is small. The harder part is confirming that the source, the state handling and the downstream connections behave correctly under a primary node change. Teams without recent cluster experience usually benefit from a written test plan before they touch production flows, and a fixed-scope engagement can cover that plan along with the build.
What drives the effort of changing list processors in a cluster?
Effort grows with the number of list processors, the number of distinct source systems and how many downstream flows depend on their output. Other drivers are the coordination method your cluster uses, which can be embedded or external ZooKeeper or Kubernetes-based state, and how much testing your change control process requires. Ask for a quote that lists each list processor, its source, the downstream connections that need a load-balanced design and the failover tests planned for each source.
Is a load-balanced connection always required after a list processor?
A load-balanced connection is not needed in every case, but NiFi flow analysis flags the case where it is missing. The built-in flow analysis rule named RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor checks connections that leave a primary-node source. Because a listing processor emits its FlowFiles only on the primary node, a connection without load balancing can keep all the work on one node. Confirm whether your downstream processors need the work spread across nodes, and document any exception you accept.
What does the work need from our team?
Plan for access to the NiFi cluster with rights to change processor scheduling and connection settings, plus the source credentials the list processor uses, such as a service account for an SMB share, SFTP host or cloud drive. You also need a non-production cluster or a copy of the flow where failover can be tested without affecting live data. Ask your source system owners to confirm the account permissions, the paths or folders to list and any network rules between the NiFi nodes and the source.
How do we confirm behavior for a specific processor version?
Start with the component documentation for the exact NiFi version you run. The ListSmb documentation states that it is designed to run on the primary node only and that a new primary node picks up where the previous one left off without duplicating data. Check the same statement for each other list processor you use, because behavior and state handling can differ by component. Then test it in your own cluster by forcing a primary node change and checking for duplicate or missing FlowFiles in the downstream results.
What does a fixed-scope engagement cover for list processors?
A fixed-scope engagement covers the agreed set of list processors, the connections after them, the flow analysis check, the failover test and the handover notes. Scope, deliverables and timelines are agreed in writing before work starts, and a quote follows from that written scope. The engagement does not change how your source systems are operated. Ask for the list of processors and environments included, the test evidence you will receive and what happens if a source behaves differently than expected.
Does DeployOpen guarantee that a primary node change will not lose data?
DeployOpen designs and tests the flow and documents what was checked, but no engagement can guarantee the behavior of every source system during every failure. The result depends on the processor, the source, the cluster coordination method and the test coverage you agree in scope. Ask for the failure scenarios tested, the evidence produced and the operational runbook so your team can repeat the checks after each upgrade.
Limits to confirm in your NiFi environment
Primary node scheduling depends on the NiFi version, the processors in your flow, the coordination method and the systems you list from. Each card below names one item to verify, and what to ask your vendor or check in the components documentation.
NiFi version and rule behavior
The RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor flow analysis rule is part of the NiFi components index, but the exact rule set and scheduling behavior can differ between releases. Confirm which version your cluster runs, then check the components index for that release before you rely on the rule. Record the version in your change records so the check can be repeated after upgrades.
Primary-node processors in your flow
Not every list processor runs on the primary node in the same way, so inventory each one in your flow instead of assuming. Processors such as ListSmb are documented as primary-node processors, but your flow may use other List processors for cloud storage, SaaS or file systems. Check each component's documentation page and mark which ones need a load-balanced connection downstream.
Coordination and primary failover
Primary node election depends on how the cluster coordinates, either through ZooKeeper (embedded or external) or on Kubernetes using ConfigMaps or Leases. Confirm which method you run and how quickly your team expects a new primary to be elected after a node loss. Test failover in a non-production cluster and verify that listing resumes on the new primary.
Source system listing limits
A listing processor on the primary node still calls your source system, so the source sets the practical limits on concurrent listings, connection counts and session lifetimes. Ask the owner of each file share, bucket or SaaS tenant about connection and API rate limits, and whether they allow multiple sessions. Set run schedules and connection pool sizes to match those limits, not the defaults.
Python and custom processor behavior
Custom processors and Python-based processors do not automatically follow the primary-node model. Check whether each one declares primary-node execution, and how it stores listing state. Python processors are documented as a beta feature, so test their failover behavior on the Python version you run, which the administration guide lists as 3.10, 3.11 or 3.12.
NiFi 1.x flows before migration
Flows running on NiFi 1.x may depend on primary-node behavior that changes or is configured differently in NiFi 2. Before migration, check whether your 1.x flows use the same listing processors and whether the flow analysis rule flags them. The migration guidance requires moving to NiFi 1.27.0 before 2.0.0, so plan the flow review as part of that path.
How DeployOpen scopes and delivers NiFi cluster flow work
DeployOpen plans, builds and documents NiFi flows in infrastructure you control, including the primary node placement of list processors and the load-balanced connections that follow them. Each engagement starts from a written scope that names the flows, cluster topology and state handling to be covered.
How we scope
- Inventory every ListSFTP, ListSmb and ListGoogleDrive processor and the downstream connections that consume their output.
- Record the cluster topology, state provider and node count that the primary node failover design depends on.
- Agree the written scope, deliverables and timeline before work starts; most engagements are fixed-scope.
How we build
- Place listing processors on the primary node and confirm each downstream connection uses a load-balanced strategy.
- Run the RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor flow analysis rule and resolve every finding before promotion.
- Test a primary node change in a non-production cluster and confirm listing resumes without duplicate records.
How we hand over
- Deliver the flow, parameter setup and a diagram showing which processors run on the primary node.
- Document the state carry-over behavior, the expected listing gap during failover and the checks operators should run.
- Walk your platform and operations teams through runbooks for primary node changes and load-balanced connection issues.
How we support
- Ongoing support, upgrades and continuous testing for these flows are available as a retainer.
- Review flow analysis findings and cluster behavior after NiFi upgrades, including any move from 1.x to 2.x.
- Quotes follow from the written scope, so ask for the support hours, response expectations and change handling in writing.
