Migrate Azure Data Factory to Apache Airflow

Migrate Azure Data Factory orchestration to Apache Airflow: the short answer

Yes, Airflow can take over scheduling and dependencies from Azure Data Factory while ADF copy activities keep running where they work. The usual pattern is that Airflow owns the schedule, the dependency graph, retries and alerting, and calls ADF pipelines through the Microsoft Azure provider's AzureDataFactoryRunPipelineOperator. Copy activities, linked services and integration runtimes stay in ADF, so the data movement does not have to be rewritten. The work is in deciding which ADF triggers, dependencies and parameter flows move to Airflow, and in sequencing the cutover so no pipeline runs twice or goes unscheduled.

Effort depends on four things you can count before any build starts: the number of ADF pipelines in scope, how complex the existing triggers are (schedule, tumbling window, storage event or chained pipeline triggers), how many pipelines accept parameters from triggers or from other pipelines, and how many pipelines contain Azure-native logic such as Data Flows, Web activities, Until loops or Execute Pipeline chains that would need to be translated into Airflow tasks. Ask your partner or internal team to produce an inventory of each of these four before accepting a quote. Pipelines that are mostly copy activities with simple triggers are the cheapest to move. Pipelines that depend on ADF's internal state or on trigger semantics that Airflow expresses differently need a design decision for each one.

Target Airflow 3. Airflow 2 reached end of life on April 22, 2026, according to the Airflow version support table, so it gets no fixes and no support, and a new migration should not land on it. Airflow 3 also changes how the scheduler, DAG processor and API server are deployed, so the ADF migration and the Airflow platform build are usually planned together. Confirm the exact Airflow 3 patch release and Azure provider version you will pin with your platform team before the first DAG is written, and check the Airflow version life cycle page for the current support status on the day you start.

What maps cleanly from ADF to Airflow and what does not

This table compares each part of an Azure Data Factory estate with its closest Airflow equivalent and marks where the move is a straight translation and where it needs redesign. Read the last column first, then confirm each item against your own pipelines, linked services and identity setup before estimating the work.

Component	In Azure Data Factory	In Airflow	Migration status
Pipelines and scheduling	Pipelines scheduled by triggers and run from the ADF service	DAGs scheduled by Airflow timetables, with the scheduler running in your infrastructure	Maps cleanly, but schedules must be re-entered and checked
Dependencies	Activity dependencies inside a pipeline and pipeline chaining	Task dependencies inside a DAG, plus cross-DAG dependencies that you design	Maps cleanly for simple chains, redesign for cross-pipeline links
Parameters	Pipeline and dataset parameters supplied at trigger time	DAG params and run configuration passed through the operator	Maps cleanly, confirm types and defaults for each parameter
Copy activities	Copy activity running on an Azure integration runtime	Keep the copy in ADF and call it from Airflow, or move to a dedicated transfer operator	Keep in place where it works, confirm connectivity from your workers
Data flows	Mapping and wrangling data flows running on Spark clusters managed by ADF	No direct equivalent; logic is rebuilt in Spark, dbt, SQL or another engine	Needs redesign and separate testing
Tumbling window triggers	Windowed runs with built-in dependency and retry state per window	Timetables and data intervals, with backfill logic you write and verify	Needs redesign, confirm late-data and backfill behavior
Linked services and credentials	Connections stored in ADF and resolved through managed identity or Key Vault	Airflow connections, populated from a secrets backend such as Azure Key Vault	Needs redesign, confirm every connection and its owner
Integration runtimes	Azure, self-hosted or Azure-SSIS runtimes that execute activities	Workers that run tasks, sized and placed in your network	Needs redesign, confirm network paths to each source
Managed identity assumptions	Pipelines authenticate as the factory's managed identity	Airflow workers need an identity or service principal you manage and rotate	Needs redesign, confirm role assignments before cutover
Waits and polling	Activity waits for a run to finish using the ADF service	Operator polls by default, or defers the wait to the triggerer	Maps cleanly with deferrable=True, confirm triggerer capacity
Monitoring and alerts	ADF monitor and alerts configured on the factory	Airflow UI, task logs and alerting rules you configure and operate	Redesign, rebuild alerts and confirm who receives them

Documented ways to run ADF pipelines from Airflow during migration

The documented starting point is AzureDataFactoryRunPipelineOperator from the Microsoft Azure provider. It starts a pipeline that already exists in your Data Factory instance, so the copy activities, linked services and integration runtimes stay where they work today. You pass a pipeline name, the resource group and factory name, and optional parameters that Data Factory maps to pipeline parameters. This lets a DAG act as the scheduler and dependency layer while Data Factory continues to do the data movement.

Pipeline name and factory identity: Name the pipeline, resource group and factory explicitly so each task points at one known pipeline.
Parameters: Pass run-time values such as dates or partition keys from the DAG so one pipeline serves several schedules.
Upstream dependencies: Place the operator after the Airflow tasks that produce its inputs, so ordering that used to live in ADF triggers moves into the DAG.
Run identifiers: Capture the run ID the operator returns and log it, so an on-call engineer can find the matching run in the Data Factory monitoring views.
No converter: The supplied sources do not include an automated ADF-to-DAG converter, so each DAG is written and reviewed by hand.

Check that the provider version in your image supports the operator and parameter names you intend to use, and confirm the pipeline behavior in a non-production factory first.

By default the operator polls Data Factory until the run succeeds or fails, so the Airflow task stays in a running state for the whole pipeline. Setting wait_for_termination=False makes the task return as soon as Data Factory accepts the run. This suits pipelines whose completion does not gate downstream work in Airflow, such as a notification or an archival copy that another team monitors separately.

When to use it: Choose it only where no downstream Airflow task reads the pipeline output, because a failure will not block later tasks.
Failure visibility: Add a separate check, such as a later sensor or an alert on the run status, so failed runs are not silently ignored.
Retries: Retries of the trigger call are not the same as retries of the pipeline itself, so review what a rerun will start.
Ownership: Record which team watches each fire-and-forget run, since Airflow no longer shows its result.
Migration risk: Teams moving from ADF triggers often assume an ADF dependency is still enforced. It is not, unless you rebuild it in the DAG.

Review each pipeline's gating role with its owners before choosing this setting.

With deferrable=True, the operator hands the polling loop to the Airflow triggerer, a separate process that waits on many tasks asynchronously. The worker slot is released while the Data Factory pipeline runs, and the task resumes when the triggerer reports a result. For long pipelines this reduces the number of worker slots held idle, which matters when your executor has a fixed concurrency budget.

Triggerer deployment: Run the triggerer as a component with its own health checks, because deferred tasks wait on it.
Capacity planning: Size worker and triggerer capacity together, and confirm the concurrency limits in your deployment.
Credential access: The triggerer also needs the Azure connection and must reach the Azure management endpoints from its network location.
Monitoring: Alert on triggerer restarts and on tasks that stay deferred longer than the pipelines they wait for.
Behavior change: Test resume and retry behavior after a triggerer restart before relying on it in production.

Confirm the triggerer setup for your Airflow version and the provider documentation before enabling deferrable mode across all tasks.

How Airflow calls ADF pipelines and waits for results

In the coexistence pattern, Airflow owns the schedule, the dependency graph and the retry policy, while Azure Data Factory keeps running the copy activities it already does well. Each ADF pipeline becomes a task in an Airflow DAG, usually through AzureDataFactoryRunPipelineOperator from the Microsoft Azure provider. The decision to settle first is which side owns the business logic that decides whether a run should start at all. If a pipeline is only a wrapper around copy activities, Airflow can call it directly. If it contains branching that other teams rely on, document that logic before moving the schedule, because it will otherwise be split across two tools. Confirm the connection to the factory, its resource group and the Airflow connection that holds the credentials before writing any DAG.

The operator offers three waiting modes, and the choice changes how much Airflow capacity a long copy consumes. By default, the operator polls the Data Factory run until it succeeds, which keeps a worker slot occupied for the whole run. Setting wait_for_termination=False submits the run and returns immediately, which suits fire-and-forget loads but leaves completion checking to a later task, such as a sensor or a downstream task that reads run status. Setting deferrable=True hands the polling to the triggerer, a separate Airflow component that runs lightweight async triggers, so the worker slot is released until the run finishes. Confirm that a triggerer process is deployed and healthy in the target environment, because a deferrable task waits indefinitely without one. Also check how many concurrent deferred waits the triggerer must hold for the peak number of pipelines.

Parameters pass from the DAG to the pipeline through the operator's parameters argument, so one pipeline can serve several schedules, such as a daily incremental load and a monthly backfill. Keep parameter values in Airflow templates and resolve secrets through a configured secrets backend, not in DAG source. On failure, the operator raises an error for the failed run, and Airflow's own retry settings, such as retries and retry_delay on the task, decide whether the run is resubmitted. Be deliberate here: retrying a copy that already wrote partial data can duplicate rows, so make loads idempotent or set retries per task. Route failure notifications through Airflow callbacks or alerting to the team that owns the data, and test that an alert reaches a person during a deliberate failure drill.

How to sequence the move with a parallel run and cutover

This sequence moves ADF scheduling into Airflow in controlled groups, so each pipeline has one owner of its schedule at any moment and the team can prove outputs match before ADF triggers are retired.

Inventory triggers, pipelines and dependencies
Export the list of ADF pipelines, triggers, trigger types and parameters, and the dependencies between pipelines, including tumbling window or event-based triggers that gate other work. Record which pipelines call Azure resources through linked services, which feed downstream systems, and which are run manually. The step is done when every trigger maps to one owning pipeline and one named business owner, and the team agrees which pipelines are in scope for each wave.
Build the Airflow 3 platform
Stand up Airflow 3 on infrastructure you control with a supported metadata database, such as PostgreSQL within the versions listed in the prerequisites, and not SQLite. Run the DAG processor as its own component, since Airflow 3 requires it, and configure the API server in place of the webserver. Set up the Azure connection, secrets backend and alerting, then confirm that a test DAG parses, runs and writes logs. The step is done when the platform passes a health check and a dummy DAG runs end to end.
Pilot a few low-risk pipelines
Choose a small set of pipelines with limited downstream impact, such as reporting copies with clear rerun behavior. For each, write an Airflow DAG that calls AzureDataFactoryRunPipelineOperator with the same parameters, and test it in a non-production Data Factory if one exists. Confirm retries, failure alerts and the handling of ADF run IDs. The step is done when the pilot DAGs have completed several runs with results the owners accept.
Run in parallel and compare outputs
For each migrated pipeline, let the ADF trigger and the Airflow DAG run against the same inputs in a window the owners agree on, writing to separate target locations where the pipeline allows it. Compare row counts, checksums, timestamps and downstream reconciliation results, and record any difference with its cause. The step is done when the owner signs off that outputs match for the agreed number of cycles, and every difference has a fix or an accepted explanation.
Cut over one trigger group at a time
Move pipelines in groups that share dependencies. For each trigger, disable it in ADF first, confirm it shows as disabled, and only then enable the matching Airflow schedule. This ordering prevents double runs, which would duplicate writes or send duplicate notifications. Keep the disabled ADF trigger in place for a rollback window, and record who approved each switch and when. The step is done when the group runs only from Airflow and the change log shows the ordering was followed.
Decommission triggers and linked services
After the rollback window closes, delete the retired ADF triggers and remove any linked services and integration runtime references that no remaining pipeline uses. Keep copy activities in ADF where they remain the right tool, and document the reason. Archive the pipeline definitions and trigger exports for audit. Confirm that access to the ADF factory is reduced to what the remaining pipelines need. The step is done when the inventory shows no active triggers outside Airflow and the archive is stored under your retention policy.

What drives the effort and duration of this migration

Most of the effort in this migration comes from how much orchestration logic sits inside Data Factory and how well-documented it is. The cards below cover the factors that move the scope most, so you can give a partner or your own team a complete picture before asking for a quote.

Triggers and dependency chains

Count every schedule, tumbling window, storage event and tumbling dependency trigger, and map which pipelines chain into which. Each trigger becomes a schedule, sensor or asset dependency in Airflow, and chains that cross resource groups or factories need an explicit ownership decision. Ask for the trigger inventory and a dependency graph exported from each factory before estimating. Chains with conditional branches or failure paths take longer to reproduce and verify than linear ones.

Data flows and Azure-native logic

Mapping Data Flows, Execute Pipeline loops, Lookup and If Condition activities, and custom Azure Function or Databricks activities is the largest variable in the work. Copy activities that already run where they work can stay in ADF and be triggered from Airflow, but Mapping Data Flows usually need to be rebuilt as code, a Spark job or a stay-in-ADF decision. Ask the partner which activities they expect to keep, rebuild or retire, and why.

Parameters, secrets and linked services

Every linked service, global parameter and Key Vault reference must be traced to a new home. Airflow reads secrets through a configured secrets backend, and the provider list includes Azure Key Vault, HashiCorp Vault and Kubernetes Secrets, so the choice of backend affects how connections are named and rotated. Pipelines that pass complex JSON parameters between activities need a documented contract. Ask for a list of linked services and the secret store each one uses.

Integration runtimes and network paths

Self-hosted integration runtimes, managed virtual networks, private endpoints and on-premises gateways each affect where Airflow workers must run and what they can reach. Airflow must be able to call the ADF management API, and any private endpoint or firewall rule needs to be recreated for the new network. Ask how many integration runtimes exist, which ones are shared, and whether the Airflow environment will sit in the same network boundary.

CI and deployment maturity for DAGs

Teams that already deploy ADF through ARM or Bicep templates and Git integration can reuse that discipline for DAG code. Teams that edit pipelines directly in the ADF studio will need to add version control, code review, environment promotion and rollback for DAGs from the start. Ask whether the current ADF factories are Git-backed, what branches and environments exist, and how production changes are approved today.

Team experience with Airflow

An engineer who already runs Airflow in production will move faster on task design, retries, connection handling and the triggerer than a team that is learning the platform during the migration. Plan time for training, or ask the partner to pair with your engineers. Also check how many people will operate the scheduler, workers and metadata database after handover, since that work recurs long after the migration ends.

Questions teams ask before moving ADF orchestration to Airflow

What drives the cost of moving ADF orchestration to Airflow?

The largest driver is how many pipelines, triggers, and dependency chains you are moving and how tangled they are. Cost has three parts: the platform work to stand up self-hosted Airflow in your own infrastructure, the migration of pipeline logic and parameters into DAGs, and the testing and cutover period when both systems run. Ongoing costs are the infrastructure, the metadata database, the people who operate Airflow, and any support retainer. Ask any quote to itemize these parts, to state which pipelines are in scope, and to say what it assumes about copy activities that stay in ADF. Apache Airflow itself is licensed under the Apache License 2.0, so there is no project license fee to budget for.

How long does an ADF to Airflow migration take?

Duration depends on the number of pipelines and triggers, how much custom logic sits in ADF expressions and linked services, the secrets and identity setup in your environment, and how long you need a parallel run before cutover. A small estimate can only be made after an inventory of pipelines, schedules, and dependencies, so treat any early figure as a hypothesis. Fixed-scope work is agreed in writing with scope, deliverables, and timelines before it starts. Ask any partner to show the inventory method they will use and to name the phases and exit criteria for each one.

What does the work need from our data platform and engineering teams?

The work needs named owners who can answer for each pipeline: what it reads, what it writes, who depends on its output, and what a failure should trigger. You also need access to the ADF factories, their linked services, and the Azure resources behind them, plus a place to host Airflow with a production-grade metadata database. Expect to supply the secrets store decision, network paths between Airflow and Azure, and sign-off on the change calendar for cutover. Access should be granted with least privilege, and the team should decide early who may author and deploy DAGs.

Can an internal team migrate ADF orchestration without outside help?

Yes, if the team already runs Airflow in production, has Python and Azure experience, and can staff the work alongside its normal duties. An internal team usually struggles with the parallel run, the trigger retirement order, and the operational setup of schedulers, workers, and backups. Be honest about capacity: the migration needs someone to own inventory, someone to rebuild pipelines as DAGs, and someone to run tests and cutover. A partner can take a defined slice, such as the inventory and first wave, while your team keeps ownership of the platform.

Do copy activities have to be rewritten in Airflow?

No, not in most cases. Copy activities that already work in Data Factory can stay there, and Airflow can call them by running the pipeline that contains them. The AzureDataFactoryRunPipelineOperator runs a Data Factory pipeline with optional parameters. By default it polls until the run succeeds, and with wait_for_termination=False it returns immediately. Setting deferrable=True moves the polling to the Airflow triggerer so workers are not held during long copies. Rewrite only where the logic is hard to keep in ADF, where the platform should no longer depend on Data Factory, or where a copy belongs in a different tool entirely. Confirm the operator behavior against the Azure provider documentation for your provider version.

How do we avoid double runs while ADF and Airflow run in parallel?

Double runs happen when a schedule or trigger remains active in both systems. Avoid them by giving each pipeline exactly one owner at a time and recording that owner in your inventory. Move one pipeline or group at a time: disable the ADF trigger only after the Airflow DAG has passed comparison runs against the same inputs. Use DAG-level pause settings in Airflow so new DAGs do not start until their cutover date. Keep ADF triggers in a disabled state rather than deleting them at first, so a rollback is a single change. Check that any downstream job reads from one output location, and that alerts route to one on-call group during the overlap.

Why build on Airflow 3 instead of Airflow 2?

Start new work on Airflow 3 because Airflow 2 is past its support window. According to the Airflow version support table, Airflow 3 was first released on April 22, 2025, and Airflow 2 (current patch 2.11.2) entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. The Airflow 3 architecture also changes how components run: the webserver is replaced by an API server, the DAG processor runs as its own component, and task code uses the REST API or Task SDK instead of direct database access. Those changes affect design now, so building on Airflow 3 avoids a second migration later. Confirm current patch levels against the version table before you commit.

Which secrets stores can Airflow use for ADF connection credentials?

Airflow supports several secrets backends through its providers, including Azure Key Vault, HashiCorp Vault, Kubernetes Secrets, and others. Choose the store that your security team already operates and audits, then configure Airflow to read connections from it instead of storing them in the metadata database. Confirm during planning which Azure identity Airflow will use to start pipelines, how it is granted access, and how rotation will be handled. Keep the list of connection IDs in your inventory so every DAG maps to a known secret.

What usually goes wrong and how to prevent it

Most failed ADF-to-Airflow migrations fail in the handoff rather than in the DAG code itself. The risks below cluster around who owns scheduling, where credentials live, which metadata database backs the platform, and which Airflow version the team is starting from. Check each one before the first production DAG is enabled.

Double runs during coexistence

While ADF triggers still fire, an Airflow DAG that calls the same pipeline will start a second run of it, often with the same parameters and the same target tables. This is the most common cause of duplicated loads during a migration. Before enabling any DAG, list every ADF trigger and its owner, then disable each trigger in the same change window as the DAG that replaces it. Confirm in the ADF monitoring view that only one run starts per schedule slot.

Lost trigger semantics

ADF tumbling window triggers track window state, dependencies and retries in ways a plain cron-style schedule does not reproduce. Teams that translate them directly often lose backfill behavior and the window boundaries that downstream jobs depend on. Document each trigger's window size, delay, retry and dependency settings, then decide whether the Airflow DAG uses data intervals, explicit parameters or both. Test a backfill over a known past window before cutover.

Secrets left in linked services

ADF linked services often hold connection strings, service principal secrets or account keys that teams assume are migrating with the pipelines. In Airflow they must be recreated in a secrets backend such as Azure Key Vault, HashiCorp Vault or Kubernetes Secrets, and the DAG should reference them by name. Inventory every linked service, rotate credentials that were ever exported, and confirm that the Airflow connection IDs resolve from the chosen backend in a non-production environment.

Unsupported metadata database

A trial Airflow instance often runs on SQLite, and teams sometimes keep it when the pipelines move to production. The production guide warns that the default SQLite backend is for testing only and can lead to data loss, and MariaDB is not supported as the metadata database. Choose PostgreSQL 14 to 18 or a supported MySQL version, size it for the scheduler and API server, and confirm backups and restore tests before the first production run.

Upgrades starting from Airflow 2

If the team starts the project on Airflow 2, the platform may reach end of life before the migration finishes. Airflow 2 entered limited maintenance and reached end of life on the dates in the version support table, after which it receives no fixes. Start new deployments on Airflow 3, and if an existing 2.x environment must be kept, confirm its support window and plan the move to 3.x. The upgrade guide requires Airflow 2.7 or later as the starting point.

DAG code using removed APIs

Airflow 3 blocks task code from using the metadata database sessions and models directly, and SubDAGs have been removed. DAGs ported unchanged from Airflow 2 will fail at import or at runtime, sometimes only in specific tasks. Run Ruff with the AIR301 and AIR302 rules across the DAG repository, replace direct database access with the REST API or Task SDK, and replace SubDAGs with TaskGroups before merging.

What changes for your data and operations teams after cutover

After cutover, the platform team owns the Airflow installation itself: the API server, scheduler, DAG processor, workers and the metadata database that holds run history, connections and variables. Airflow 3 splits the DAG processor out as its own component and replaces the webserver with an API server, so each of these needs a place in your monitoring, patching and capacity plans. The platform team also owns the upgrade calendar, which means tracking the Airflow support windows and rehearsing upgrades in a non-production copy before each one. Confirm with your Azure and Kubernetes or VM teams who holds the database backups and restore tests, because ADF's managed history is no longer there to fall back on.

Data engineers keep ownership of the pipeline logic that matters most to them: the DAG code, the ADF pipelines and their copy activities, and the parameters passed between them. Delivery is the shared seam. Decide who merges DAG changes, how they reach the scheduler (for example, through a Git-based sync or an image build), and who approves a change that alters a schedule or a credential reference. Pipeline definitions that remain in ADF still need a change path of their own, so the team should agree which repository is the source of truth for each.

Logs, access and audit trails move with the orchestration. ADF's activity run history and Azure Monitor diagnostics will no longer record scheduling decisions, so the platform team must decide where task logs are stored, how long they are retained and who can read them. Confirm the authorization model in the Airflow auth manager you choose, map roles to your identity provider groups, and keep a record of who can trigger, pause or clear runs. Also confirm that secrets are read from a supported backend, such as Azure Key Vault, rather than written into DAG code, and that access to that vault is logged. Your auditors will ask for this evidence, so set up the export before cutover, not after.

How DeployOpen scopes and delivers an ADF to Airflow migration

DeployOpen plans and implements the move of ADF scheduling and dependencies into self-hosted Apache Airflow inside infrastructure you control. Engagements are fixed-scope, so the pipeline inventory, deliverables and cutover plan are agreed in writing before build work starts.

How we scope

We inventory ADF triggers, pipelines, linked services and parameters to decide which ones move and which stay in Data Factory.
The written scope names each pipeline to be wrapped, the triggers to be retired and the secrets backend to be connected.
We confirm which Airflow version, executor and metadata database your environment will run before any DAG is written.
Ongoing support, upgrades and testing after handover can be added as a retainer, priced separately from the fixed scope.

How we build

We build DAGs that start existing ADF pipelines with parameters, using the Azure provider operator and your service principal setup.
We configure deferrable waits so long Data Factory runs wait in the triggerer rather than occupying worker slots.
We run old ADF triggers and new Airflow schedules side by side, comparing run results before any trigger is disabled.
We test failure paths, retries and alerting for each migrated pipeline and record the results against the agreed scope.

How we hand over

You receive DAG code in your repository with a dependency map showing which ADF dependencies became Airflow task dependencies.
The runbook covers the cutover order, how to rerun a failed Data Factory pipeline from Airflow, and how to roll back a trigger.
We document the Airflow connections, secrets backend entries and the role each service principal needs, so your team can rotate them.
Your team reviews and accepts deliverables against the written scope before the engagement closes.

How we support

Where you want continued help, a retainer can cover Airflow upgrades, provider package updates and changes to pipeline wrappers.
Before any Airflow upgrade, we check DAG code against the Ruff AIR301 and AIR302 rules, as the Airflow 3 upgrade guide recommends.
Support scope, response expectations and any SLA are written into the retainer, so confirm them in the quote rather than assume them.
Your team keeps ownership of the Airflow environment, and we document each change so operations staff can run it without us.

Migrate Azure Data Factory orchestration to Apache Airflow