Running Azure Data Factory pipelines from Apache Airflow

Trigger, parameterize and monitor Azure Data Factory pipelines from self-hosted Airflow with the Microsoft Azure provider, and decide which work stays in ADF.

On this page

How Airflow can own the schedule while Azure Data Factory runs the pipelines

If your team already runs Azure Data Factory pipelines and wants Airflow to own the schedule and the dependencies between jobs, the supported approach is the Microsoft Azure provider's AzureDataFactoryRunPipelineOperator, paired with AzureDataFactoryPipelineRunStatusSensor where a run started elsewhere needs to be watched. Airflow decides when a pipeline starts, what must finish before it starts, and what runs after it succeeds or fails. Data Factory still executes the pipeline activities, including its copy activities and its own integration runtime, so the work stays where your team already manages it. The operator starts a pipeline with optional parameters and, by default, polls until the run succeeds. Setting wait_for_termination to False returns as soon as the run is submitted, which suits DAGs that only need to kick off work.

The part that most affects capacity is waiting. With deferrable=True, the operator hands the polling to the Airflow triggerer instead of holding a worker slot for the whole pipeline run, which can last hours. The status sensor supports the same deferrable pattern. The sensor takes a run_id, often passed from the operator's XCom output, so a DAG can start a run in one task and check it in another. Before you commit to this design, confirm with your Azure and Airflow administrators that the triggerer is deployed and sized for the number of concurrent waits you expect, and that the Airflow service principal has the Data Factory permissions needed to start and read runs. The Azure Data Factory operators documentation lists the parameters for each operator, and the Azure provider package is documented separately.

What you need before connecting Airflow to Data Factory

The integration depends on four things lining up: a supported Airflow 3 base, a compatible Microsoft Azure provider release, an Azure identity that can run and read pipelines in the factory, and network paths from the components that call Azure. Check each one before writing the first DAG, because most failed connections trace back to one of them.

Airflow 3 base and Python version

Start from Airflow 3.3.2, the current patch release in the supplied version table, and confirm that your Python interpreter falls in the supported range of 3.10 to 3.14. Airflow 2 reached end of life on April 22, 2026, so a new integration should not target it. If you are upgrading an existing Airflow 2 environment, the Airflow 3 upgrade guide requires being on 2.7 or later first. Check that your DAG code passes the Ruff AIR301 and AIR302 rules before you connect Data Factory.

Metadata database and sizing

Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4, or an Innovation release as the metadata database. MariaDB is not supported, and SQLite is for testing only. Ask your team whether the database will be backed up and how it will be restored, because the run history that the sensor reads lives there. Plan at least 4 GB of RAM for the scheduler and components, as the prerequisites recommend.

Microsoft Azure provider release

Install the Microsoft Azure provider at a release that matches your Airflow version. The supplied provider page lists release 15.2.1 with a minimum of Airflow 2.11.0, so confirm the release notes for the version you pin. Pin the provider in your constraints file so that worker and triggerer images carry the same version. The provider brings Azure SDK dependencies for Data Factory, Key Vault and other services, so review that dependency set with your security team.

Azure identity with rights on the factory

Create a dedicated identity, such as a service principal or managed identity, for Airflow. Grant it only the Data Factory permissions needed to start pipeline runs and read run status on the specific factory. Do not reuse a personal account or a broad subscription contributor role. The exact role names and the connection fields that map to tenant, client and subscription values must be confirmed against the Azure provider docs for your chosen release before you store them in Airflow.

Secret storage for connection values

Store the Azure client secret or certificate in a secrets backend rather than in DAG code or plain environment variables. Decide where the Airflow connection lives, whether in the metadata database, a Vault backend or another secrets store, and who can read it. Confirm which connection fields the provider reads from the connection, since field names change between releases. Rotate the credential on a schedule your security team sets and test that a rotated secret reaches both the workers and the triggerer.

Network paths from workers and triggerer

Both the task workers and the triggerer must reach the Azure Data Factory management endpoints and the Entra ID login endpoint over HTTPS. The triggerer is the component that polls when deferrable mode is on, so it needs the same outbound path as the workers. Check proxies, egress firewall rules and private endpoint routing in the network where Airflow runs, and test from the actual pods or hosts, not from a laptop. Ask your cloud network team to confirm whether traffic must stay on a private link.

How to configure the Data Factory connection and first DAG

This sequence takes a pinned Airflow environment from no Azure access to a DAG that triggers an existing Data Factory pipeline, waits for it to finish, and reports its status. Each step ends with a check you can see before moving on.

  1. Confirm the pinned Airflow environment

    The platform team confirms the Airflow 3 version, Python version and constraints file used in the target environment, so the Microsoft Azure provider is installed against the versions your team has already tested. Check the provider's minimum Airflow version in the package documentation before installing. Done means the environment lists the expected Airflow version and the constraints file is recorded in your change record.

  2. Install the Microsoft Azure provider

    The platform team adds the Microsoft Azure provider to the image or virtual environment build, pinned through the same constraints file as Airflow itself. The provider pulls in the Data Factory SDK and other Azure dependencies, so review the resulting package set in your dependency scanner. Done means the provider appears in the installed providers list of the scheduler and worker images, and a DAG import with the ADF operator raises no errors.

  3. Create the Azure connection outside DAG code

    An administrator creates an Airflow connection of the Azure Data Factory type and supplies the service principal or managed identity details through the connection form or your secrets backend, never as literals in a DAG file. Ask your Azure team which authentication method the tenant allows and where the client secret will be rotated. Done means the connection test succeeds from the Airflow UI, and a grep of the DAG repository finds no client secret or tenant credential.

  4. Define factory and pipeline names as configuration

    The data engineering team records the resource group, factory name and pipeline name in Airflow Variables or a configuration file, then reads them in the DAG. This keeps the same DAG code usable across development and production factories. Done means the DAG file contains no hard-coded factory identifiers, and the names resolve to a pipeline visible in the Data Factory authoring view for the target environment.

  5. Trigger a run with the run pipeline operator

    The data engineering team adds an AzureDataFactoryRunPipelineOperator task that passes the pipeline parameters from the DAG run configuration. For a first test, keep the default wait behavior so the task holds until the run completes. Done means a manual trigger of the DAG starts a run, the task log shows the run identifier, and the run appears in the Monitor section of the Data Factory studio under the expected pipeline name.

  6. Add the pipeline run status sensor

    The team adds an AzureDataFactoryPipelineRunStatusSensor downstream of the trigger task, reading the run identifier from the operator's XCom output. Use the sensor to confirm the run status before dependent tasks start, and decide whether it should defer to the triggerer once you have reviewed worker capacity. Done means a failed pipeline in Data Factory marks the sensor task as failed, and a successful one allows the downstream task to run.

Waiting on a run: operator, deferrable mode or separate sensor

By default, AzureDataFactoryRunPipelineOperator starts the pipeline and then keeps polling Azure until the run reaches a successful state. The task stays in the running state for the whole duration, and the worker slot it occupies stays occupied with it. For short pipelines this is the simplest option: one task, one log, and a failure shows up on the task that started the run. The cost appears when runs last hours. Every long run holds a slot, and a burst of concurrent runs can starve unrelated tasks of workers. Use this mode when runs are short, concurrency is low, or your executor isolates the work well enough that an idle slot does not matter.

  • Simplest failure view: the task that started the run also reports its final state, so one task page shows the whole story.
  • Slot cost: the worker remains allocated for the full run time, not only the moments when work is happening.
  • Pool limits: if you use pools to cap ADF-related concurrency, a polling task counts against that pool for its whole duration.
  • Retry behavior: a retry restarts the operator, so confirm whether a retry should start a new pipeline run or re-attach to the existing one.

Before you choose this mode, check the expected run durations for each pipeline and the number of runs that can overlap in a single hour.

How pipeline parameters and run IDs move between tasks

The operator passes parameters to the Data Factory pipeline at trigger time, and the run ID it creates is what the status sensor needs to follow that run. Parameters are supplied through the operator's parameters argument, and they can be templated from Airflow, for example from a DAG run conf value, a Variable, or a logical date. Keep each parameter name identical to the name defined in the pipeline's parameters section in Data Factory, including case. Data Factory will not match a renamed key to its pipeline parameter, so a mismatch shows up as a failed run or a silently ignored value. Before go-live, have the pipeline owner export the pipeline JSON and compare its parameter list with the DAG's templated keys. Only the value types the pipeline expects should be sent, so check that dates and numbers are formatted the way the pipeline's expressions read them.

The run ID is the handoff between the trigger task and the monitoring task. When the operator returns, its run ID is pushed to XCom by default, and a downstream AzureDataFactoryPipelineRunStatusSensor reads it through a templated run_id such as the operator's XCom pull. This is also the place to plan for retries. If the trigger task is retried after the run already started, a second run may be created, so decide whether the pipeline is safe to run twice or whether the trigger should be guarded. XCom should carry only the run ID and other small identifiers. Do not pass the pipeline's output rows, file listings or large JSON bodies through XCom, because the metadata database stores them and every task reading them loads them again. If a downstream step needs the results, have the pipeline write them to storage and pass the path. Confirm the XCom backend and any size limits for your Airflow deployment before relying on larger values.

Which Airflow and Data Factory objects map to each other

This table lines up the Airflow objects you configure with the Data Factory objects they act on, so you can trace a failed run from the Airflow task to the factory run. Read it before writing the connection and before deciding which run identifier each task passes downstream. Field names and state labels come from the Microsoft Azure provider release you install, so confirm them against that release before you build.

Airflow objectData Factory counterpartWhat to confirm
Airflow connection (Azure Data Factory type)Azure identity and factory scope: tenant, client or managed identity, subscription, resource group and factory nameConfirm the connection type and extra fields in your provider release, and whether managed identity is supported in your setup
Operator resource and factory fieldsThe specific Data Factory instance the pipeline lives inConfirm the factory name and resource group match the environment the connection points to
DAG task parameters passed at run timePipeline parameters defined in the factory's pipeline JSONConfirm each name and type matches the pipeline definition, since mismatches surface only at run time
Operator run_id output (XCom)Pipeline run ID returned when the factory accepts the runConfirm the XCom key and value shape in your provider release before a downstream sensor reads it
Sensor run_id inputExisting pipeline run the sensor polls for statusPass the operator's run ID explicitly, and confirm the sensor's parameter names in your release
Airflow task state (success, failed, deferred)Pipeline run status (queued, in progress, succeeded, failed, cancelled)Map each ADF terminal state to an Airflow outcome in your alerting, and confirm how cancelled runs are reported
Task retry and timeout settingsPipeline run lifetime and cancellation in the factoryDecide whether an Airflow retry starts a new factory run or reuses the existing one, and test that behavior

Securing the Azure credentials and the Airflow to Azure path

The Airflow to Azure Data Factory path carries credentials that can start production pipelines, so it needs controls at four points: where the connection lives, what the identity may do, who can change the DAGs that call it, and where workers and the triggerer can connect. The cards below cover those controls and the checks to run before go-live.

Keep the Azure connection out of DAG code

Store the Azure connection in a secrets backend, such as an external vault or a Kubernetes-backed secret store, and reference it by connection ID from the DAG. Hardcoded client secrets in DAG files end up in version control, scheduler logs and every copy of the repository. Ask your team to confirm which secrets backend the Airflow deployment reads from and that no ADF connection exists as plain text in the metadata database. Rotate the client secret on a schedule your security policy sets.

Scope the identity to one factory

Grant the Airflow service identity only the Data Factory role needed to start runs and read run status, assigned at the scope of the specific factory rather than the subscription or resource group. A pipeline-run identity that can also edit pipelines, linked services or triggers turns a scheduling credential into a change path into production. Confirm with your Azure administrators which actions the identity holds and review them after each new pipeline is added.

Separate run rights from edit rights

Use distinct identities for Airflow in development and production, so a test DAG cannot start a production factory. Within Airflow, limit who can edit DAG files and who can trigger runs manually, since a manual trigger with altered parameters can change what a production pipeline processes. Record these decisions as part of the change control for the scheduler, and confirm how your deployment maps Airflow roles to these permissions before go-live.

Protect the DAG repository and review path

Require pull request review and protected branches for the repository that holds DAGs targeting Azure, because whoever merges a DAG change can run code with the Azure connection. Restrict who can push directly to the branch that Airflow syncs from. Ask your team to confirm that DAG deployment is logged, so each production change can be traced to a reviewer and a commit.

Limit egress from workers and triggerer

Allow outbound connections from Airflow workers and the triggerer only to the Azure endpoints the ADF operators and sensors need, and block other internet destinations. Deferrable mode moves polling to the triggerer, so that component needs the same Azure reachability as the workers. Confirm the required endpoints with your network team and your Azure provider documentation, and test them from the actual Airflow pods or hosts, not from an administrator workstation.

Review trust boundaries beyond this path

This path covers only the Azure credentials and the network route. The broader trust model, including how DAG authors are trusted, how worker isolation works and how JWT and API access are controlled, belongs to the Airflow security hardening work. Use that review to confirm that a DAG author cannot read the Azure connection through task code, logs or the API before you rely on the controls above.

Why ADF runs fail, hang or report the wrong status

This table lists the failures teams most often hit when Airflow drives Azure Data Factory, with the likely cause and the fix for each. Read it from the symptom column inward: a run that fails immediately, one that waits forever, or one that reports a status that does not match ADF. The main takeaway is that most of these faults sit in configuration, identity or the deferral setup rather than in the pipeline itself.

SymptomLikely causeFix
Operator fails at the first call with a not-found errorFactory name or resource group in the Airflow connection or operator does not match the ADF resourceCopy the factory name and resource group from the Azure portal, then confirm the subscription ID in the connection matches the same subscription
Authentication or authorization error on the run callThe identity behind the Azure connection lacks a Data Factory role on the factory, or the credential has expiredGrant the identity a role scoped to the factory that allows running and reading pipelines, then test the connection from the Airflow UI
Run starts but uses the wrong values or fails inside ADFParameter names or types passed in the operator do not match the pipeline's declared parametersCompare parameter names and types with the pipeline definition in ADF, and pass only parameters the pipeline declares
Sensor or downstream task reports no run or an empty IDThe run_id was not pulled from the operator's XCom output, or the sensor template points at the wrong task IDPass the run ID through a templated XCom reference using the exact producing task ID, and log the value once to confirm it is set
Sensor times out on a long copy activitySensor timeout is shorter than the pipeline run, so the task fails while ADF is still workingSet the sensor timeout and poke interval to fit the longest expected run, and confirm the run status in ADF before changing retries
Deferred task sits in a deferred state and never resolvesThe Airflow triggerer is not running or cannot reach the Azure endpoints, so deferred tasks have no process to resume themConfirm the triggerer runs as its own process with network access to Azure, and check its logs for trigger errors
Import error or unexpected behavior after an Airflow upgradeThe installed Microsoft Azure provider version does not match the Airflow release it was built forCheck the provider's minimum Airflow version in its package page and pin a compatible provider version in your constraints file
Airflow shows success while the ADF run failed or was cancelledThe operator was run with wait_for_termination set to False, so Airflow never checked the final statusKeep waiting enabled, or add a status sensor downstream that checks the run ID until it reaches a terminal state

When Data Factory should keep the copy work and Airflow owns the schedule

Keep Azure Data Factory as the engine for copy activities and connector-heavy data movement when its integration runtimes, linked services and built-in connectors already do that work well, and let Airflow own the schedule, the cross-system dependencies, retries, calendars and alerting. The split works only if each layer has one job. A practical boundary is that Airflow decides when a pipeline runs and what must finish first, while ADF moves the bytes. Before agreeing to that boundary, confirm which ADF activities your team uses today, which of them depend on ADF triggers or schedules, and which connectors need a self-hosted integration runtime inside your network.

Running two orchestration layers has a cost that is easy to underestimate. Each layer has its own failure modes, its own permissions, its own monitoring and its own change process, and an incident often spans both. Someone must answer where a failed run is investigated, who restarts it, and whether a retry in Airflow can start a copy that ADF is already running. Ask any partner to document the run ID handoff, the retry policy on each side and who holds on-call responsibility for each layer, and to estimate the recurring effort of keeping both current, including Azure provider upgrades and the minimum Airflow version the Microsoft Azure provider requires.

The case for moving logic into Airflow is strongest when most ADF pipelines are thin wrappers that only chain copies, call stored procedures or branch on dates, because those steps are easier to version, test and review as Python in the same repository as the rest of the estate. Moving them also consolidates alerting and audit evidence in one place. Treat this as a migration decision with its own scope: inventory each pipeline, keep the copy-heavy ones in ADF, port the scheduling logic that spans systems, and retire ADF triggers only after the Airflow side has been tested against the same run history.

Questions teams ask before connecting Airflow to Data Factory

Who does the work when Airflow triggers Data Factory pipelines?

DeployOpen plans and builds the Airflow side under a written scope: the Azure provider installation, the Airflow connection to your Data Factory, the DAGs that call the operators, and the monitoring and failure handling around them. Your team keeps ownership of the Data Factory pipelines, their datasets and linked services, and the Azure subscription. The split should be written into the scope before work starts, including who approves changes to pipeline parameters and who is on call when a run fails.

What drives the cost of this integration?

The main drivers are the number of pipelines to trigger, how many parameters each one takes, how many environments (development, test, production) need the same DAGs, and how much dependency logic sits between Airflow tasks and Data Factory runs. Identity design and the authentication method for Azure also add effort. Costs split into one-time work (design, build, testing, cutover) and recurring work (provider upgrades, pipeline changes, run monitoring). Ask for a quote that lists each cost item separately, so you can see what changes if the pipeline count grows.

How long does setup take, and what does it depend on?

Duration depends on your environment more than on the operators themselves. The points that most often set the pace are access to the Azure subscription and Data Factory instance, the state of the pipeline inventory, how identity is handled for the Airflow connection, and whether Airflow is already running in your infrastructure. A small set of pipelines in one environment is a much shorter engagement than many pipelines across several environments. Ask for a timeline in the written scope that names the dependencies on your side, such as access approvals and review dates.

Can our internal team build this integration without a partner?

Yes, if your team has Airflow and Azure experience. The operators are documented in the Microsoft Azure provider and the work follows standard Airflow patterns. What usually takes longer for an internal team is the first pass at deferrable waiting, retry behavior and the run status sensor, plus the access and identity decisions on the Azure side. Many teams start with a short internal proof of concept on one non-production pipeline, then decide whether the remaining build should be done internally or under a fixed-scope engagement.

What do we need to provide before the work starts?

Plan to provide four inputs:

  • Azure access: a role or service principal with rights to the target Data Factory and the resource group that contains it, approved by your Azure administrators.
  • A pipeline inventory: names, parameters, expected run durations, schedules and owners for each pipeline Airflow will trigger.
  • Identity decisions: how Airflow authenticates to Azure, where credentials are stored, and who can rotate them.
  • A test environment and a named approver for production cutover.

Confirm the exact role and authentication method against the current Azure provider documentation and your own Azure security policy before access is granted.

How are upgrades of the Azure provider handled?

The Microsoft Azure provider is a separate package from Airflow, so it is upgraded on its own schedule and tested against your DAGs. The provider release supplied for this page is 15.2.1, which requires Airflow 2.11.0 or later. Upgrades should be checked against the provider changelog, tested in a non-production environment, and pinned in your dependency constraints so production does not change without review. Whether an upgrade is part of a retainer or a one-off task should be stated in the written scope.

Should we keep Data Factory copy activities or move them into Airflow?

Most teams keep copy activities in Data Factory and let Airflow own the schedule and dependencies. Data Factory is already built for managed data movement and its integration runtimes, so moving copies into Airflow tasks adds code to maintain without a clear gain. Airflow is a better place for ordering work across systems, handling retries and alerting on the whole chain. Review each pipeline on its own: the decision depends on where the data lives, how the copy is configured and who owns it today.

Which Airflow version should the integration target?

Target Airflow 3 for new work. Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. Airflow 2 reached end of life on April 22, 2026, so it gets no fixes. If your Airflow is still on 2.x, the Airflow 3 upgrade guide requires being on 2.7 or later first and recommends checking DAG code with Ruff AIR301 and AIR302 rules. Confirm the Azure provider version that supports your target Airflow release before you pin dependencies.

How DeployOpen scopes and delivers this integration

DeployOpen approaches Running Azure Data Factory pipelines from Apache Airflow as a written, fixed-scope piece of work inside your own infrastructure. We agree the pipelines, connections, triggerer capacity and handover materials before build starts, then deliver them against that scope.

How we scope

  • Inventory the ADF pipelines Airflow will trigger, including which parameters come from the DAG and which stay inside Data Factory
  • Confirm the Airflow deployment, metadata database and triggerer placement that deferrable waiting will depend on
  • Agree in writing which ADF copy activities stay in Data Factory and which schedules move to Airflow

How we build

  • Configure the Azure connection with a credential model your security team approves, stored in your secrets backend rather than DAG code
  • Build run-pipeline and run-status DAGs with run IDs passed through XCom and deferrable waiting enabled where the triggerer is sized for it
  • Test failure, timeout and retry paths against a non-production Data Factory before any production schedule is switched over

How we hand over

  • Document each DAG, connection, parameter mapping and the ownership boundary between Airflow and Data Factory
  • Provide a runbook for re-triggering runs, reading ADF run status and handling pipelines that stall or fail
  • Walk your operations and data teams through the handover and record the agreed acceptance checks

How we support

  • Ongoing support, provider upgrades and continuous testing are available as a retainer after handover
  • Review Airflow and Microsoft Azure provider version changes before you adopt them in your environment
  • Confirm with your Azure and Airflow teams any capability, quota or release detail that your own environment depends on

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.