How Airflow moves files over SFTP and SSH
A data team often must pull drop files from partner SFTP servers or push extracts to legacy hosts on a fixed schedule. In Airflow, the SFTP and SSH providers handle these transfers and remote commands. They are separate packages, installed on top of Airflow core, so the first decision is which provider packages your image needs and how they will be pinned to your Airflow version. The provider packages reference lists apache-airflow-providers-sftp and apache-airflow-providers-ssh alongside the FTP and Samba providers, and each one is versioned on its own.
This page covers four building blocks. Connections hold the host, port, user and key material that every operator and sensor uses, so credentials stay out of DAG code. Sensors wait for a drop file to appear before downstream tasks run. Host key checking decides whether Airflow trusts the remote server's identity, which is where most transfer failures and most security findings originate. Remote commands run a script on a legacy host over SSH, which needs the same care as any other privileged access. Each of these is explained below in the order you will build them.
The providers do not cover everything around a transfer. They do not decide who owns the partner relationship, how keys are issued and rotated, which network paths are opened through your firewall, or how files are validated, quarantined and archived once they arrive. Those controls sit with your team or your implementation partner, and they should be designed before the first DAG is written. Confirm the exact operators, parameters and behavior for your provider version in the provider documentation, since these packages change independently of Airflow core.
What you need before installing the SFTP and SSH providers
The SFTP and SSH providers sit on top of a working Airflow 3 platform, so the platform prerequisites come first. The cards below cover the Airflow baseline, the provider packages, the network paths to partner hosts, host key trust and who owns the key pairs.
Airflow 3 and supported Python
Start from Airflow 3.3.2, the current patch release, and confirm your Python interpreter falls within the supported range of Python 3.10 to 3.14. If you are still on Airflow 2, the upgrade path must be finished first, because the provider install is pinned to the Airflow version. Confirm the target version with your platform team before you pin anything.
Metadata database and hosting
Use PostgreSQL or MySQL for the metadata database. The SQLite default is for testing only, and MariaDB is not supported as a metadata database. Choose Kubernetes or virtual machines based on your existing platform, and confirm that your chosen hosting is covered by the supported Kubernetes range before you size the cluster.
Provider packages
Install apache-airflow-providers-sftp and apache-airflow-providers-ssh as a separate step, pinned to your Airflow version. Install them with a constraints file so that transitive dependencies do not drift. Teams that need repeatable builds often host a frozen constraints file on an internal server.
Network paths and firewall rules
List every partner SFTP host, legacy server and object storage endpoint that workers must reach, with the port and direction for each. Open only those paths from the worker or executor network, and confirm which component actually opens the connection, since in some layouts that is a dedicated worker pool rather than the scheduler.
Host key fingerprints from out of band
Obtain each remote host key fingerprint through a channel separate from the connection itself, such as the partner's signed onboarding document or a call to a known contact. Record the fingerprint in your change record before enabling strict host key checking. Accepting an unverified key on first connection defeats the protection the check is meant to provide.
Key pair ownership
Assign each key pair to a named owner who approves its creation, rotation and removal. Generate keys on a controlled host, store private keys in your secrets backend rather than in DAG code, and give each partner a dedicated key so that revoking one does not affect others. Confirm who signs off on the partner's public key registration.
Configuring SFTP and SSH connections in order
This sequence takes a team from a clean Airflow environment to verified connections to partner SFTP servers and legacy hosts, with each connection proven before any DAG depends on it. Following the order keeps credential, host trust and network problems separate so they are easy to diagnose.
Install providers pinned to the Airflow version
The platform team installs apache-airflow-providers-sftp and apache-airflow-providers-ssh as a separate step from Airflow core, using a constraints file that matches the running Airflow version. Constraints apply only to the pip command they are used with, so the team should freeze and host its own constraints file on an internal server so every worker and the scheduler resolve identical versions. The step is done when a fresh image or virtual environment imports both provider packages and reports the same versions on every component that runs tasks.
Create a dedicated service account on each remote host
The remote host administrator creates one service account per Airflow environment, such as a non-interactive user that owns only the inbound drop directory or the outbound staging path it needs. Permissions are limited to those directories, password login and shell access are disabled where the partner allows, and the public key is installed with restrictive file permissions. Ask each partner or legacy host owner to confirm the account's allowed paths in writing. The step is done when the account can list its own directory and cannot read sibling paths.
Generate keys and store them outside DAG code
The Airflow platform owner generates a key pair for each service account on a controlled workstation or inside the secrets tooling the organization already runs, then stores the private key in the secrets backend or a protected key file on the workers. DAG files and repositories never contain key material or passphrases. Confirm with the security team where the private key will live, who can read it, and how rotation will be handled. The step is done when the private key exists only in the approved location and the public key is registered on the remote host.
Register connections with key-based auth
An Airflow administrator creates an SSH or SFTP connection for each remote host in the Airflow connections store, referencing the service account and the stored private key, with the host and port the partner has supplied. Passwords are left empty for key-based connections, and the connection is restricted to the teams that need it. The step is done when each connection appears in the connection list with the expected host, port and login, and no secret values are visible in the UI or logs.
Verify the host key before trusting the server
Before enabling host key checking, the platform owner obtains the server's host key through a channel the partner has confirmed, such as a fingerprint provided in writing or read from the server by its administrator, and records it in the connection's host key configuration. Strict checking stays on so an unknown or changed key fails the connection rather than being accepted silently. Ask partners to confirm when their keys rotate so the process is planned. The step is done when a deliberate connection test succeeds and a mismatched key, tested in a non-production connection, fails as intended.
Test each connection with a small DAG
The team deploys a small test DAG that opens each connection, lists the remote directory it is allowed to read, and closes the session without transferring any business data. The DAG runs manually in a non-production environment first, and its logs are checked for the expected authentication and host key messages. The step is done when every connection returns the expected directory listing and the test DAG passes on a worker in the same network zone that the production DAGs will use.
Add sensors and transfer tasks
Once connections are proven, the data engineering team adds sensors that wait for drop files and transfer tasks that move files to or from object storage or other destinations, using the timeouts and file patterns agreed with each partner. Sensors are tested against a sample drop file, and transfer tasks are run against a copy of the test data before a production schedule is enabled. The step is done when a test drop file triggers the sensor, the transfer lands in the expected destination, and the run appears correctly in the Airflow UI and logs.
Why provider installs should be pinned and frozen
Provider packages such as apache-airflow-providers-sftp and apache-airflow-providers-ssh install separately from Airflow core, and each one should be pinned to the Airflow version you run. The installation from PyPI guide describes this two-step pattern and warns that constraints apply only to the pip command they are used with. That scoping matters: a constraints file passed on one install does not protect a later pip install run on a worker image or an API server image. Make the constraints explicit in every build step that installs providers, and treat a missing constraints argument as a failed review item.
Teams can freeze the resolved set and host their own constraints file on an internal server. For air-gapped or restricted networks, this is the practical route: the build pipeline pulls packages once from an approved mirror, the constraints file is versioned alongside the DAG repository, and workers and the API server install from that controlled source without reaching public package indexes. Confirm with your security team which internal index, artifact store or mirror is approved, who signs off on changes to the frozen file, and how a security fix to a provider is promoted through the same path.
Repeatable rebuilds depend on the same discipline. When a worker node, an API server container or a scheduler image is recreated after an incident, it should resolve the same provider versions as the one it replaces, or the team should know exactly which versions changed and why. Record the Airflow version, the provider versions, and the constraints file reference in the image metadata or the deployment manifest. Then test a rebuild in a non-production environment before relying on it, and decide in advance who owns provider upgrades so that pinned versions do not quietly age past the support you need.
Choosing between SFTP operators, SSH operators and sensors
A partner or upstream system writes a file to an SFTP directory on its own schedule, and the pipeline must not start until that file is complete. The SFTP sensor checks the remote directory for a matching file name or pattern until the file appears or the sensor times out. It needs an SSH connection defined in Airflow, a read-only account scoped to the drop folder, and a verified host key.
- Completeness checks: Many partners write a file in place, so ask whether they upload to a temporary name and rename on completion. Otherwise the sensor may detect a partial file.
- Poll cost: In poke mode, the sensor holds a worker slot between checks. Confirm whether the pinned SFTP provider version offers a deferrable mode that hands the waiting to the triggerer.
- Timeouts: Set a timeout that matches the partner's delivery window and alert on it, so a missing file becomes a visible failure.
- Access scope: Grant list and read rights on the drop folder only, with no write access to the rest of the host.
Check the file naming contract with the partner and the sensor timeout before the first production run.
When a task must send a file up to a partner or pull one down into a landing zone or object storage, the work is a file transfer, not a wait. The SFTP operator covers the common get and put steps in a single task. The SFTP hook exposes the same connection to custom Python, which suits transfers that also list, verify, rename or post-process files. Transfers run on a worker, so the worker's memory, disk and network path set how long the task takes.
- Operator or hook: Use the operator for simple transfers. Use the hook when one task must list, verify and move files together.
- Worker sizing: Staging large files consumes worker disk and memory, so size workers for the largest expected file or route those tasks to a dedicated queue.
- Integrity: Compare checksums or byte counts after each transfer and fail the task on a mismatch.
- Object storage handoff: When the destination is S3 or another bucket, confirm which provider package handles that step and where its credentials are stored.
- Retry safety: Write to a temporary path and rename on success, so a retry does not leave a half-written file in the destination.
Confirm the destination, staging capacity and retry behavior with the partner before the first full-volume run.
Some legacy hosts offer no file interface for the step you need. A batch script, a database export or a reconciliation job runs only as a shell command on that machine. The SSH operator runs a command over an SSH connection and returns the output and exit status to Airflow. It can do the most on the host of the three options, so it needs the tightest controls: a dedicated account, a restricted shell or forced command where the host allows it, and no interactive login.
- Command surface: Limit the account to the commands the job needs. Ask the host owner whether a forced command or restricted shell is possible.
- Network path: Allow connections only from the worker network to that host and port, with no general outbound access.
- Secrets: Keep passwords and tokens out of the command string, because command text can appear in logs. Use Airflow connections or a secrets backend.
- Exit codes: Confirm how the script reports failure, since a script that exits zero on error will appear as a success in Airflow.
- Long jobs: The task holds a worker for the full run, so check whether a deferrable option exists in the pinned SSH provider version before scheduling long jobs.
Ask the host owner to document each command, its typical runtime and its failure codes before you wire it into a DAG.
Mapping partner accounts, paths and host keys to Airflow connections
This table maps each remote system to the Airflow connection that reaches it, with the account, authentication method, base path, host key, owner and permitted operations recorded for each one. Use it as the register that reviewers check before any DAG opens a transfer, and treat any row without a named owner or a pinned host key as not ready for production.
| Remote system | Connection and access | Scope and ownership | Permitted operations |
|---|---|---|---|
| Partner SFTP server | Connection ID in the SFTP provider, dedicated login account, key file referenced from a secrets backend | Remote base path limited to the partner's inbound and outbound folders, owned by the integration team with the partner contact recorded | Get and put on named folders, sensor on drop files, no shell access |
| Internal legacy host (file transfer) | Connection ID in the SSH provider, service account with a key held in the secrets backend, not in DAG code | Single application directory, owned by the team that runs the legacy job, with a documented change contact | File get and put only, sensor on output files, no interactive login |
| Legacy host (commands only) | Separate SSH connection ID with a restricted account and forced command or allowlisted wrapper scripts | Named scripts only, owned by the platform team that approves each script change | Run approved commands and read exit status, no file write outside the script directory |
| Object storage target | Cloud or storage connection ID in its own provider, with credentials from the secrets backend and a role scoped to one bucket or prefix | Bucket and prefix named in the register, owned by the data team that consumes the files | Upload and list on the approved prefix, no delete rights unless the owner signs off |
| Host key fingerprint | Pinned host key stored with the connection, verified against the value the partner or server owner supplies out of band | Fingerprint record includes date collected, collector and the change approver for rotation | Connection fails if the key differs, so rotation runs through the register first |
| Authentication method | Key file reference or secrets backend lookup, with passphrase handling documented | Key owner, rotation schedule set by the owning team, and the storage location recorded | Connection uses key-based login only, password login disabled on the remote account |
| Connection ownership and review | Connection ID naming convention, one connection per remote account and environment | Owning team, backup owner and last review date for each entry | Add, change or retire connections through the register, with review before each change is applied |
| Environment separation | Distinct connection IDs and host keys for development, test and production | Production entries limited to the production scheduler and named operators | Test DAGs point only at test remotes, so partner production folders are not used for trials |
Hardening SFTP and SSH access without opening broad network paths
An SFTP or SSH transfer DAG is only as safe as the account it logs in with, the host it trusts and the network path it can reach. The cards below cover the controls that connect those three layers, so the Airflow workers that move partner files do not become a broad route into your network.
Per-partner key pairs
Give each partner or legacy host its own key pair and its own Airflow connection, rather than sharing one key across several feeds. When a partner rotates keys or leaves, only one connection changes, and the audit trail shows which feed used which identity. Ask each partner to confirm how they will generate, store and rotate their side of the key, and agree on a rotation date before the first production run.
Pinned host key fingerprints
Configure each connection with the expected host key fingerprint instead of accepting whatever key the server presents on first contact. Auto-accepting host keys lets a spoofed server receive your credentials and drop files. Collect fingerprints through a channel the partner confirms out of band, store them with the connection definition, and treat any mismatch as a failed task that someone investigates.
Chrooted least-privilege remote accounts
Create a dedicated remote account for each feed with access limited to the directories it needs, ideally chrooted so it cannot list or read the rest of the host. Separate upload and download paths where possible, and deny shell access if the workflow only transfers files. Ask the host owner to confirm these settings on the server, since Airflow cannot enforce them from the client side.
Egress limited to named hosts and ports
Restrict outbound traffic from the Airflow workers that run SFTP and SSH tasks to the specific partner hosts and ports each feed requires. Enforce this with network policy, firewall rules or proxy controls outside Airflow, because a DAG can connect to any address its code names. Document the allowlist with the owner of each destination, and review it whenever a DAG is added.
Private keys kept out of DAG code and logs
Store private keys in a secrets backend or an encrypted connection field, never inline in DAG files, environment variables baked into images or task parameters. Check that key material does not reach task logs through verbose client output or exception messages. Anyone who can write DAG code can read what the worker can read, so review who holds deploy rights to the DAG repository.
Secret masking and connection permissions in the UI
Confirm that passwords, key passphrases and sensitive extras are masked in the Airflow UI and API responses, and test this with a throwaway connection before relying on it. Limit who can create or edit connections, because the Airflow security model for DAG authors also applies to anyone who can write connections. Ask your security team to review the role mapping for connection management as part of the authentication design.
Testing transfers before a partner goes live
Start the test sequence with connection checks that move a throwaway file, because a DAG that cannot authenticate or reach the partner host will never reach the harder cases. Create a dummy file in the partner's test folder, run a single SFTP or SSH task that uploads and downloads it using the connection exactly as production will define it, and confirm the host_key value (or the known hosts entry) matches the key the partner published through an out-of-band channel. Run the same check from the actual worker node or Kubernetes pod that will execute production tasks, since egress rules and DNS often differ from a developer laptop. Ask each partner which test endpoints they offer, whether that host uses the same key pair and host key as production, and whether they can place files in a separate test directory. Confirm these answers in writing before you schedule any staging run.
Next, test the sensor behavior in both failure modes. With no file present, the sensor should keep poking at the interval you configured and should time out without marking the run as successful, so confirm that the timeout and poke settings match the partner's delivery window. With a partial file present, the sensor may fire on a file that is still being written. Ask the partner whether they upload under a temporary name and rename on completion, and if they do not, agree on a size-stability check or a completion marker file before the DAG treats the file as ready. Verify both paths with real timed uploads in the test environment.
Finally, prove that retries cannot duplicate or lose data. Make the download or upload step idempotent by writing to a temporary local path, verifying the result, and only then moving the file into its final location or marking the batch as processed. Kill a task mid-transfer on purpose and confirm that the retry produces exactly one final file. Compare checksums (such as SHA-256) or, where the partner provides them, a manifest of sizes and hashes, before and after transfer, and fail the task on any mismatch. Record the test results as change evidence, and treat the staging run against the partner test server as the last gate before cutover, not the first.
Common SFTP and SSH failures and their fixes
This table pairs each common failure in Airflow SFTP and SSH transfers with its usual cause and the fix that addresses it. Read it as a checklist when a transfer task fails, hangs or leaves files in the wrong state. Most of these issues come from key handling, timing and connection lifecycle rather than from Airflow itself.
| Failure | Likely cause | Fix |
|---|---|---|
| Host key mismatch | The partner rotated its server host key, or the connection points at a different host behind the same name. | Store the new host key in the SSH connection's host key field after verifying it out of band with the partner. Keep host key checking enabled and confirm the change through your change process. |
| Permission denied on connect | The private key is in an unsupported format, the public key is not on the remote account, or the account lacks rights to the target directory. | Confirm the key pair matches the account, export the private key in a format the provider accepts, and test the login with the same key outside Airflow. Ask the partner which directories the account can read and write. |
| Permission denied on write | The account can log in but cannot write to the drop folder, or the target is a symbolic link to a restricted path. | Check write permissions on the exact directory and its parents. Use a dedicated landing folder that the account owns, and verify with a test upload before scheduling. |
| Sensor timeouts | The SFTP sensor polls for a file that arrives late, the wait exceeds the sensor timeout, or the filename pattern does not match the partner's naming. | Set the sensor timeout to match the partner's delivery window, confirm the pattern against real filenames, and alert on timeout rather than retrying silently. |
| Sensors holding worker slots | Poke-mode sensors occupy a worker slot for the whole wait, starving other tasks on busy deployments. | Use deferrable or reschedule mode where the provider supports it, so waiting frees the slot. Confirm the mode your provider version offers before the build starts. |
| Partial files picked up | The sensor or downstream task detects a file while the partner is still writing it, so the transfer reads an incomplete file. | Ask the partner to write to a temporary name and rename on completion, then match only the final name. Where that is not possible, compare file size across two checks before the transfer starts. |
| Firewall or egress blocked | Outbound port 22 or the partner's SFTP port is closed from the Airflow worker network, or a proxy or NAT rule is missing. | Open egress from the worker subnet only to the partner's host and port, and test from the same worker that runs the task. Document the rule with the partner so the path survives network changes. |
| Connection leaks across tasks | Each task opens a new SSH session and never closes it, especially when a task fails before its cleanup runs. | Use context managers or explicit close calls in every task, keep one hook per task rather than shared global clients, and watch the remote server's session limit during load tests. |
| Remote command fails on spaces or special characters | A remote command string is built by joining paths without quoting, so spaces or shell characters split the command. | Quote each path argument with proper shell quoting or pass a list of arguments where the operator supports it. Avoid building commands from untrusted filenames, and test with a file name that contains spaces. |
Questions buyers ask about SFTP and SSH work in Airflow
What drives the cost of SFTP and SSH transfer work in Airflow?
The biggest drivers are the number of partner and legacy hosts, how many distinct key and host key arrangements you must manage, the network approvals each connection needs, and how much testing is possible against each partner's environment. Each host adds its own connection definition, its own host key decision and its own failure modes, so cost grows with variety more than with raw file volume. Ask any provider to price the work per host and per environment, and to separate one-time build effort from the ongoing effort of rotating keys, updating partner contacts and handling new onboarding requests.
How long does it take to get a first working file transfer?
The first transfer depends less on the Airflow install than on the external dependencies: partner key exchange, firewall change approvals, and access to a non-production partner endpoint if one exists. Once a connection, a key and a host key are agreed, a single sensor-and-transfer DAG can usually be proven quickly, but partner response times set the pace. Ask for a written plan that lists each external dependency with an owner, so the schedule reflects who must act and not only the build work.
What must our team provide before the work starts?
You need to provide named partner contacts who can approve key exchange and confirm host key fingerprints, the host names and ports for each server, and the network paths from your Airflow workers to each endpoint. You also need service accounts on each remote host with the minimum directory permissions required, and a decision on where private keys will be stored. Without these, most of the calendar time is spent waiting, so confirm each item with the partner and your network team before scoping the build.
Can our internal team build and run SFTP and SSH transfers alone?
Yes, if your team already runs Airflow in production and has someone comfortable with SSH keys, firewall rules and connection configuration. The providers are separate packages installed alongside Airflow core, so the technical steps are not exotic. The risks sit in the parts that are easy to underestimate: host key verification, key rotation without breaking schedules, and running remote commands without granting broad network access. Teams often do the first connection alone and then bring in outside help for hardening and for multi-partner rollout.
How should host keys be verified and managed?
Host key checking should be enabled, with the expected host keys obtained through a channel you trust, such as a partner's documented fingerprint delivered by a known contact, and recorded in your secrets management system rather than being accepted automatically on first connect. Each host key change should trigger a review, because a silent change can mean a legitimate rotation or an interception. Plan the process before go-live: who receives a rotation notice, who approves the new fingerprint, and how the connection definition is updated across Airflow environments.
What breaks most often with drop-file sensors and partner servers?
The most common failures are partner files arriving late or partially written, file naming changes without notice, expired keys, and network paths that open for one environment but not another. Sensors that wait for drop files should check for stable file completion rather than mere presence where the partner supports it, and timeouts should match the partner's real delivery window. Ask the team to document each partner's naming convention, delivery window and escalation contact so that a missing file produces a clear alert instead of a silent stall.
What should we ask for in a quote for this work?
Ask for the written scope to list every partner and legacy host in scope, the connection types for each (SFTP, SSH, or both), whether remote command execution is included, and which environments will be built and tested. Request the assumptions about partner cooperation, the deliverables such as connection documentation and runbooks, and what is handed over at the end. Also ask which parts recur, such as key rotation, onboarding of new partners and support, so that the one-time build cost is not confused with the ongoing operating cost.
How DeployOpen scopes and delivers SFTP and SSH integrations
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including Apache Airflow SFTP and SSH file transfers. We start from the written scope, build the transfer DAGs and connections in your environment, and hand over documentation and runbooks your team can operate.
How we scope
- Inventory every partner SFTP server, legacy host and object storage target, with its authentication method and host key owner.
- Agree which transfers use sensors, which use direct operators, and where remote commands are permitted.
- Document network paths from Airflow workers to each endpoint so firewall requests are specific and reviewable.
How we build
- Install the SFTP and SSH provider packages with the constraints file pinned to your Airflow version.
- Configure key-based connections with credentials stored in your secrets backend, not in DAG code.
- Set host key checking to a verified known-hosts policy and test drop-file sensors against staging endpoints first.
How we hand over
- Deliver tested DAGs, connection definitions and a runbook covering key rotation, host key changes and failed transfers.
- Walk your operators through retry behavior, sensor timeouts and how to recover a partially copied file.
- Provide a written record of network rules, service accounts and the controls each transfer relies on.
How we support
- Support is fixed-scope by default, and ongoing support, upgrades and continuous testing are available as a retainer.
- Retainers can cover Airflow provider version updates, host key rotation and changes to partner endpoints.
- Ask for a quote that separates one-time build work from recurring support, and confirm response terms in writing.
