Apache NiFi SFTP integration: when scripted transfers should become NiFi flows
Many organizations still move partner files with cron jobs and shell scripts that poll an SFTP server, download anything new, and push outbound files on a schedule. These scripts often work until a partner changes a directory name, a job overlaps with the previous run, or a failed transfer leaves no record of what was sent. In Apache NiFi, the same work is done with ListSFTP to list remote files, FetchSFTP to retrieve their content, and PutSFTP to deliver files to a partner, all from the built-in NiFi processor bundle. This page covers the pattern and the decisions that come with it, not the behavior of any particular partner server, which you should confirm with each partner before design starts.
The first decision is whether the transfer needs to be a flow at all. A script that moves one file per night for one partner may be acceptable if someone owns it and it fails loudly. A flow is justified when several partners or directories share the same logic, when a missed or duplicated file has a business or regulatory cost, or when you need a visible record of each transfer. The reader should list every existing job, its schedule, its failure handling, and who receives alerts when it breaks. That inventory tells you which jobs map cleanly to ListSFTP and FetchSFTP and which depend on conventions, such as a rename after pickup, that must be reproduced or replaced with an agreed partner process.
What you need before building SFTP flows in NiFi
Most SFTP problems in NiFi start before the first processor is placed. The runtime, the partner's account and host keys, the network path from every node, and the choice of listing nodes all shape how the flow behaves once it runs unattended. Settle these items in writing before the build so the flow design can rely on them.
NiFi 2 and Java 21 runtime
Plan the build on NiFi 2 rather than 1.x, since the project states that 1.x cannot be maintained further. The administration guide lists Java 21 as the minimum runtime, so confirm the JVM on every node matches before installing. Confirm the target version against the current release on the downloads page, because the release you pick sets which processor behavior you test against.
Partner host keys and fingerprints
Obtain the partner's SFTP host public key or fingerprint through a verified channel, not from the first connection prompt. NiFi must trust that key before any transfer runs, and a key change on the partner side should be treated as a planned change that you approve. Record who owns the key, how it is rotated, and where it is stored for each cluster node.
Partner account rights
Agree the exact permissions the NiFi service account needs: read and list on the outbound folder, delete or rename rights if files must be removed after pickup, and write rights on the inbound folder for delivery. Ask the partner whether they support atomic rename so that partially written files are not picked up. Document the folder structure as a contract.
Network paths from each node
Every cluster node that runs a processor opens its own SFTP connection, so the firewall rules must allow outbound TCP from each node's address to the partner endpoint, not only from a single gateway. Confirm NAT, allowlisting by source IP, and any proxy requirements with the partner before the build. Test each node separately.
Which nodes run listing
Decide in advance whether listing runs on the primary node only or is scheduled across the cluster. Listing processors use NiFi-managed state to track what has been seen, so the choice affects duplicate handling, failover behavior and how many connections the partner sees. Write the decision into the flow design so operators know where to look when a pickup stops.
Credentials and secrets handling
Keep SFTP passwords and private keys out of flow definitions and out of version control. Plan where the private key file lives on each node, who can read it, and how the key is rotated without an outage. Confirm that your NiFi security configuration is in place before storing credentials, because unsecured instances bind only to the loopback interface by default.
How to configure ListSFTP, FetchSFTP and PutSFTP step by step
This sequence replaces a scripted SFTP job with NiFi processors that pick up files once across a cluster, retrieve them safely, route them into your flow, and deliver results with controlled file completion.
Create the SSH credentials in an SSHService
Your platform engineer creates a controller service that holds the SFTP connection details, including the host, port, username and the private key file the service account uses. Store the key outside the flow definition, restrict who can read the controller service, and confirm that the host key verification setting matches the partner's published fingerprint. Enable the service and check that its state reads enabled without validation errors. Confirm the service account can log in from a NiFi node with a manual connection test on the server before you wire any processor to it.
Configure ListSFTP with the remote path and primary node schedule
Add ListSFTP to the canvas, point its SSH service property at the controller service, and set the remote path to the exact folder the partner drops files into. Set the scheduling strategy so the processor runs on the primary node only, because listing on every node would create duplicate FlowFiles. Leave the listing state in framework-managed state so NiFi remembers which files have already been listed across restarts and node changes. Verification: start the processor once against a test folder containing one known file and confirm exactly one FlowFile appears with the filename attribute set to that file's name.
Chain FetchSFTP on the filename attribute
Add FetchSFTP and connect the success relationship of ListSFTP to it. Reference the filename attribute and the remote folder attributes that ListSFTP writes, so FetchSFTP retrieves the same file that was listed, rather than a path you typed by hand. Run FetchSFTP on all nodes so the content transfers are spread across the cluster, since each FlowFile carries the instructions to fetch its own content. Verification: open the provenance for the test FlowFile and confirm that the content claim has a size that matches the file on the SFTP server, and check the checksum on both ends.
Route fetched files to downstream processing
Connect the FetchSFTP success relationship to the processors that validate, transform or deliver the content, and send the failure and communications failure relationships to a handling path that logs the file name and retries a limited number of times. Keep the original remote path as an attribute so the record of origin survives processing. Define a clear rule for what happens to files that fail validation, because a file that is fetched but not processed will not be picked up again by ListSFTP. Verification: send one good file and one deliberately malformed file, then confirm that the good file reaches its destination and the bad one lands in the failure path with its provenance intact.
Configure PutSFTP for delivery with temporary names
Add PutSFTP, set its SSH service to a controller service that points at the receiving server, and set the remote directory where the partner expects files. Enable the temporary file naming option so the file appears under a working name and is renamed only after the transfer completes, which prevents the partner from reading a partial file. Decide whether the processor should overwrite existing files or reject them, and confirm the choice with the partner before go-live. Verification: deliver one test file, then confirm on the receiving server that no file with the temporary name remains and that the final file matches the source checksum.
Confirm completion behavior and failure handling
Review the PutSFTP success, failure and reject relationships and connect each to a monitored path. Confirm with the partner which signal they use to start their own processing, such as the presence of the final file name or a companion control file, and make sure the rename step happens before that signal appears. Test a failed delivery by pointing a copy of the processor at an unreachable port in a non-production environment, and confirm that FlowFiles remain queued and are retried rather than dropped. Verification: check that the failure path produces a visible alert and that no partial files appear on the receiving side.
Run an end-to-end checksum comparison
Drop a small batch of representative files, including one with a name containing spaces or unusual characters, into the source folder and let the full chain run. Compare the checksum of each source file with the file received at the destination and confirm that the count of files matches. Check that the listing state does not reprocess files after you restart the processor, which confirms that duplicate pickup is prevented. Verification: a documented run log with file names, sizes, checksums and a clean restart test, kept with the change record for the integration.
How NiFi state prevents duplicate SFTP pickups across a cluster
ListSFTP does not move file content. It lists the files on the SFTP server and emits one FlowFile per file, with the filename attribute set, so that a downstream FetchSFTP can retrieve each file's content. The reason no duplicates appear is that ListSFTP records what it has already listed in NiFi's framework-managed state. That state works for a single node and across a cluster, and it is the same mechanism listing processors such as ListFile use. Because the record lives in NiFi rather than on the local disk of one node, the listing does not repeat when a different node is elected primary or when the cluster changes shape. Before you rely on this in production, confirm the state scope and the timestamp or listing strategy your NiFi version applies in the ListSFTP documentation, since those settings decide what counts as a new file.
On restart, the stored state is what tells ListSFTP where it stopped. A restarted processor, or a restarted node, should not re-emit files it already listed, but this only holds if the state was persisted and the cluster was healthy when the listing ran. Test the behavior deliberately: stop the processor, restart NiFi, and confirm that no previously listed file reappears in the queue. Files that are rewritten in place are a different case. A file that keeps its name but gets a newer modification time can look new to a timestamp-based listing, and a file overwritten by a partner during a transfer may be fetched while it is incomplete. Agree with the partner that files are written under a temporary name and renamed on completion, and confirm how your chosen listing strategy treats a changed timestamp on an existing name.
The design depends on FetchSFTP being downstream because the listing only proves that a file name existed at a point in time. It says nothing about whether the content was retrieved, validated or delivered. If a FetchSFTP run fails, the FlowFile should be routed to a retry or failure path, not dropped, and the file must not be treated as completed just because it was listed. Keep the listing and the fetch in one flow with explicit success and failure relationships, and add a checkpoint after delivery, such as moving or deleting the source file on the server only once the content has been written to its destination. Plan this step with the partner, because the server-side cleanup rights and the timing of the move decide whether a retry can ever pick up the same file twice.
Mapping SFTP paths, file attributes and partner identifiers into NiFi
This table maps each part of an SFTP transfer, from the remote directory and filename to partner routing and delivery paths, onto the FlowFile attributes or processor properties that carry it in NiFi. Read it as a checklist for the design review: each row names the value, where it lives, and what breaks if it is set inconsistently. The main takeaway is that partner identity should come from the path or a controlled lookup, not from the filename alone.
| Element | How NiFi carries it | Design decision and risk |
|---|---|---|
| Remote directory | ListSFTP sets the path attribute for each file it lists | Confirm the attribute values in a test run; a changed partner folder layout alters routing silently |
| Filename | ListSFTP sets the filename attribute, which FetchSFTP uses to build the retrieval path | Keep the filename unchanged between List and Fetch; renaming upstream breaks the fetch |
| Fetch path | FetchSFTP combines the path and filename attributes to retrieve content | Check the remote separator and absolute versus relative path handling on your server |
| Partner or folder identifier | Extract from the path with an attribute expression into a routing attribute | Map each folder to a named partner in a lookup table, and route unknown folders to a failure relationship |
| Delivery path on PutSFTP | Remote directory property built from partner attributes using expression language | Agree the partner naming convention in writing before build; changing it later affects every consumer |
| Checksums from the partner | Read the value from a manifest or sidecar file and store it as an attribute | Verify the content hash against the attribute before accepting the file; a mismatch should route to a quarantine path |
| Manifest files | Parse the manifest record and attach each entry to its matching FlowFile | Confirm with the partner how manifests are named, whether they arrive before or after data files, and what happens when one is missing |
| Original names and timestamps | Keep the source filename and modification time as attributes for audit | Retain these attributes through delivery so downstream teams can trace each file back to its source |
Key-based authentication and hardening for SFTP flows
An SFTP flow is only as safe as the credentials, host trust and account scope behind it. The four cards below cover the controls that keep a NiFi pickup or delivery from becoming an unmanaged path into partner systems. Confirm each processor property against the component documentation for your NiFi version before you rely on it.
Private keys through sensitive properties
Configure the SSH private key as a sensitive value rather than pasting key material into plain processor text. In the SFTP processor properties, check whether the key is supplied as a file path or as content, and confirm how the passphrase is handled for your version. Keep key files readable only by the NiFi service account on every cluster node, and plan how a rotated key reaches all nodes at the same time. Confirm the exact property names in the ListSFTP and PutSFTP documentation.
Host key verification over trust-on-first-use
Do not let a flow accept unknown host keys. Pin the partner server's expected host key and configure the processor to verify against it, so a changed key stops the transfer instead of silently succeeding. Ask each partner for its key fingerprint through a separate, trusted channel, and record who approved it. Plan a change process for legitimate host key rotation, because a failed verification in a scheduled flow will surface as repeated processor errors rather than a clear alert.
Least-privilege SFTP accounts
Create one SFTP account per flow or partner, restricted to the directories that flow reads or writes. ListSFTP needs read and listing rights on the inbound folder, and PutSFTP needs write rights on the delivery folder only. Deny shell access and remove rights to browse parent directories. Ask partners whether they can separate inbound and archive folders so that a flow never has permission to delete or overwrite files it did not create.
Who can edit credentialed processors
Processors that hold SFTP keys or passwords are a privilege boundary. In NiFi, use the authorization policies to restrict who can view and modify those components, and keep the flow editing role separate from the people who manage partner accounts. The NiFi authentication options, such as LDAP, OpenID Connect, SAML 2.0 and X.509 client certificates, govern who can reach the UI, not what an SFTP processor sends, so treat them as the first gate and policies as the second. Review these policies whenever staff change roles.
When a managed file transfer pattern with partners fits better than ad hoc SFTP
A managed file transfer pattern fits better than ad hoc SFTP once the flows you build in NiFi depend on another organization's behavior: the folders they write to, the accounts they use, the host keys they present and the way they handle failed deliveries. ListSFTP and FetchSFTP can pick up files reliably on your side, and PutSFTP can deliver them, but neither component decides who owns a file after it lands, how long it stays there, or what happens when a partner changes something without telling you. Those answers have to be agreed with each partner in writing before the build starts, because a flow cannot encode a rule nobody has stated. Typical questions include the agreed folder structure for inbound and outbound files, whether the partner or your team removes processed files and on what schedule, how many retries a failed transfer gets and whether a redelivery is a new file or a replacement, and who is notified when a transfer fails and through which channel.
Host key and account changes deserve the same care. A partner who rotates an SSH host key will cause your SFTP processors to fail their connection checks, and an unplanned change can look like an attack. Agree in advance how the partner announces key changes, how you verify the new fingerprint through a second channel, and who approves updating the trusted key in the controller service. The same applies to account rotation: agree on the notice period, which credential type the partner will use (for example, key-based authentication with a named public key), and who is responsible for updating the private key or password in your NiFi parameter or credential store. Ask each partner for a named technical contact and an escalation path, because a stalled transfer without an owner usually surfaces only when a downstream team asks where the file is.
DeployOpen scopes these partner agreements as part of the fixed-scope work, so the written scope lists the folder layout, retry and redelivery rules, cleanup ownership, notification routes and key-change procedure for each partner before any flow is built. Ask any prospective implementation partner, including DeployOpen, to show which of these decisions are captured in the statement of work and which remain open. The agreements themselves are yours to sign with each partner, and they should be reviewed by the people who will operate the flows. If a partner cannot commit to these points, treat the arrangement as ad hoc and plan for manual intervention, because no NiFi configuration can compensate for a transfer agreement that does not exist.
Common SFTP failures in NiFi and how to fix them
This table pairs each common SFTP failure in a NiFi pickup or delivery flow with its likely cause, the check to run and the fix. Read it as a triage list: start with the symptom you see, then confirm the cause before changing the flow. The main takeaway is that most duplicate, partial and premature-read problems come from state scope, write timing or host key handling rather than from SFTP itself.
| Failure | Likely cause and check | Fix |
|---|---|---|
| Duplicate files after a node change or state reset | ListSFTP state is stored per cluster in NiFi framework-managed state, but a cleared state, a new flow version or a changed listing path can cause files to be listed again. Check the processor state and the listing directory settings. | Confirm the state contents before and after the change. Keep the remote path and filters stable, and plan any state reset with a documented replay window. Downstream steps should tolerate duplicates or check a unique key. |
| Files fetched before writes complete | ListSFTP lists a file while the sender is still writing it, so FetchSFTP reads a partial file. Check whether the sender writes directly to the final name and whether file sizes change between listings. | Ask the sender to write to a temporary name and rename on completion. Filter out temporary patterns in ListSFTP, or add a minimum file age check before fetching. |
| Host key mismatch | The server host key changed, or the stored known hosts entry does not match the server. Check the processor error for host key verification and compare the fingerprint with the value the partner gives you through a separate channel. | Confirm the new fingerprint out of band before updating the known hosts setting. Do not disable host key checking to clear the error. |
| Permission denied on the remote directory | The SFTP account lacks read access for listing or fetching, or write access for PutSFTP on the target directory. Check the account on the server with a manual SFTP session using the same key and path. | Ask the partner or server administrator to grant the minimum read or write permissions on the exact directories used. Verify with a test file before enabling the flow. |
| PutSFTP leaves partial files | A transfer interrupted mid-stream can leave an incomplete file on the target, and a retry may create a second copy. Check the target directory for partial files and the processor failure relationship history. | Configure PutSFTP to write to a temporary name and rename after completion, if the partner's receiving process supports it. Route failures to a retry path and have the receiver ignore files without a completion marker. |
| Listings run on every node | The listing processor is not scheduled on the primary node only, so every node lists the same directory and creates duplicate FlowFiles. Check the scheduling strategy and the execution setting on ListSFTP. | Set ListSFTP to run on the primary node only. Pair it with FetchSFTP on all nodes so fetch work still spreads across the cluster. |
| Fetches succeed but downstream sees empty content | FetchSFTP ran against a name that changed or was removed after listing. Check the failure and not-found relationships and the timing between list and fetch. | Route not-found results to a logged path, not to the success path. Reduce the gap between list and fetch, and keep the source files in place until the flow confirms receipt. |
Questions buyers ask about NiFi SFTP integration
What drives the effort and cost of replacing our SFTP scripts with NiFi?
The main drivers are how many scripts and jobs exist, how many partners and remote hosts they touch, and how much logic lives inside each script. Each job needs its pickup and delivery paths mapped, its naming and filtering rules captured, its retry and failure behavior understood, and its outputs checked against what downstream systems expect. Scripts that rely on local folders, cron timing or undocumented file conventions take longer to map than ones with clear contracts. Cost is mostly the time of engineers who understand the business rules and the NiFi build. Ask a partner to break the quote into discovery, flow build, testing and handover so you can see where effort sits.
How long does an NiFi SFTP build take?
Duration depends mainly on the number of partner connections, the number of distinct file formats and naming rules, the volume and file sizes moved, and the number of environments that need testing. A single partner with one folder structure is a much smaller job than a dozen partners with different key types, directory layouts and delivery windows. Clustered operation, duplicate prevention and monitoring add design and test time. Fixed-scope engagements agree the timeline in writing before work starts, so ask for the schedule to be tied to the partner list and the environments in the agreed scope.
What do we need to provide before a build starts?
Plan to supply a short list of inputs that the build cannot invent:
- Partner contacts who can confirm folder layouts, file naming and delivery schedules
- SFTP host names, ports and the directories each flow reads from or writes to
- Authentication details, meaning the public keys each side will accept and a secure route for private keys and passphrases
- Test files that match real formats, including edge cases such as empty files, large files and unexpected names
- Access to a non-production NiFi and SFTP test endpoint, plus the change window for production cutover
The more complete these inputs are at kickoff, the fewer assumptions the build carries into testing.
Can our internal team build an NiFi SFTP integration alone?
Yes, if your team has NiFi administration skills, comfort with SFTP key management and time to learn the ListSFTP and FetchSFTP pattern, state handling and cluster behavior. The hardest parts are usually not the processors themselves but the operational details: making sure pickups are not duplicated across cluster nodes, handling partial files, deciding how failed deliveries are retried, and proving the flow works under failure. Teams without recent NiFi experience often find that a partner shortens the learning curve. Assess your team against those specific skills before deciding, and confirm who will operate the flows after go-live.
How does DeployOpen quote an SFTP integration project?
DeployOpen quotes from a written scope. Most engagements are fixed-scope, so the scope, deliverables and timeline are agreed in writing before work starts. For SFTP work, that scope normally lists the partners and jobs in scope, the environments to build and test, the authentication method for each connection, the testing and handover deliverables, and any ongoing support you want after go-live. DeployOpen does not publish a price list. Provide the partner inventory and current script list, and the quote will follow from what is written down.
Does the NiFi SFTP pattern prevent duplicate file pickups in a cluster?
It can, when the flow is designed for it. ListSFTP is built to list files and create one FlowFile per file, and NiFi framework-managed state tracks what has already been listed across a cluster, so listing is intended to run on the primary node and avoid repeats. The design still needs checks: confirm the listing schedule, decide how files that change after listing are handled, and test what happens when the primary node changes during a run. Confirm the exact behavior for your NiFi version against the ListSFTP documentation before relying on it.
Should we use managed file transfer software or NiFi for partner exchanges?
The choice depends on what you need to control. Managed file transfer products typically bundle partner onboarding, audit reporting and a dedicated transfer interface, while NiFi gives you flexible routing, transformation and enrichment inside infrastructure you control. Many organizations use NiFi for the data movement and transformation logic and keep a managed file transfer product for partner-facing endpoints. Confirm which audit, onboarding and contractual features your partners require before deciding, because those requirements often settle the question more than technical preference.
How DeployOpen scopes and delivers NiFi SFTP integrations
DeployOpen plans, builds, tests and hands over Apache NiFi SFTP file transfer integrations inside your own infrastructure, with ListSFTP, FetchSFTP and PutSFTP flows designed around your partners, key management and state handling. Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we scope
- Inventory every scripted SFTP job, its schedule, partner, file naming rules and downstream consumer
- Record which partners need pickup, delivery or both, and where each directory lives
- Agree the written scope for key-based authentication, host key verification and credential storage
- Confirm the NiFi cluster topology and whether the listing runs on the primary node only
How we build
- Build ListSFTP and FetchSFTP pairs with framework-managed state so each remote file is picked up once
- Configure PutSFTP delivery with temporary file naming and failure routing for partner retries
- Test duplicate handling by restarting nodes and changing the cluster leader during a controlled run
- Verify key rotation, permission errors and unreachable partner servers against the agreed test plan
How we hand over
- Deliver versioned flow definitions and a runbook describing each processor, controller service and parameter
- Document partner onboarding steps, host key checks and the owner of each credential
- Walk your operations and security teams through alerts, bulletin review and recovery steps
- Provide a written list of open items, decisions and any settings that still need your confirmation
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover
- Plan review of SFTP flows when NiFi moves between 1.x and 2.x or when partner servers change
- Agree in the written scope which partner changes, key rotations and incident responses are covered
- Re-run the agreed verification tests after each NiFi upgrade or significant flow change
