What an Airflow 2 to 3 migration involves
An Airflow 2 to 3 migration changes how the platform is put together, not only how DAGs are written. In Airflow 3 the webserver is replaced by an API server started with airflow api-server, and the DAG processor must run as a separate component. Task code can no longer open Airflow database sessions or use Airflow models directly; it must go through the REST API or the Task SDK. SubDAGs are removed in favor of TaskGroups, and the Simple auth manager is the default, so any login setup you rely on needs an explicit decision. Each of these affects deployment templates, service definitions, network rules and DAG code, so inventory them before you plan the work.
Airflow 2 (current patch 2.11.2) entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026. Under the project's support policy, end-of-life versions receive no fixes, so any security exposure found after that date stays open on your side. Three triggers usually start the work: the end-of-life date, a security finding that cannot be patched on 2.x, or a new platform team that wants to standardize on the 3.x architecture. The Airflow 3 upgrade guide requires being on Airflow 2.7 or later first, so confirm your current minor version before you estimate anything.
Whether to migrate in-house or with a partner depends on three things you can measure: the number of DAGs in scope, how much custom operator, hook or plugin code depends on database access or SubDAGs, and how much hands-on Airflow experience your team has. A small estate with standard providers and an experienced team often migrates in-house. A large estate with heavy custom code, or a team that has never run the scheduler and API server separately, often benefits from a fixed-scope partner engagement. Ask any partner to review your DAG inventory and custom code before quoting.
What the work includes
A fixed-scope migration usually splits into six work packages, each ending in a deliverable your team keeps. The order matters: inventory and the 2.x upgrade come before code conversion, and cutover is planned last.
Inventory of DAGs and dependencies
We list every DAG, operator, connection, variable, plugin and deployment, then record which ones depend on database sessions, SubDAGs or custom providers. This list sets the scope and the order of work. The deliverable is a migration inventory with an owner and a risk rating for each item.
Upgrade to the latest 2.x first
The Airflow 3 upgrade guide requires Airflow 2.7 or later before upgrading, so older environments move to the latest 2.x release first. We check provider compatibility and run the database migrations on a staging copy. The deliverable is a tested 2.x baseline with the upgrade steps recorded.
Ruff AIR301 and AIR302 findings
We run the Ruff AIR301 and AIR302 rules across the DAG repository to flag code that Airflow 3 no longer supports. Each finding is fixed, suppressed with a documented reason, or assigned to the owning team. The deliverable is the converted repository with the lint report attached.
Task code moved to REST API and Task SDK
Airflow 3 blocks task code from using database sessions and models directly. We move that code to the REST API or the Task SDK and update imports to airflow.sdk. Teams confirm the behavior of each converted task in a test run. The deliverable is test evidence for each converted task.
SubDAG to TaskGroup rewrites
SubDAGs are removed in Airflow 3, so each one is rewritten as a TaskGroup. We compare the run graph, retries, trigger rules and dependencies before and after the change. Any behavior change is recorded and approved by the DAG owner. The deliverable is a rewrite log with the comparison results.
Infrastructure split and cutover plan
Airflow 3 runs the API server and DAG processor as separate components, and the schedulers need their own sizing and database access. We set up these components, then write the cutover sequence and a rollback path to the Airflow 2 environment. The deliverables are the infrastructure configuration and a runbook.
What drives cost and duration
This table lists the main effort drivers for an Airflow 2 to 3 migration, why each one adds work, and what the team can do before cutover to reduce it.
| Driver | Why it adds effort | How to reduce it |
|---|---|---|
| DAG count and complexity | Each DAG needs review, Ruff AIR checks, rewrites and test runs, so effort grows with volume and branching logic. | Prune DAGs nobody runs, retire duplicates and standardize patterns such as TaskGroups before the move. |
| Direct database access in task code | Airflow 3 no longer lets task code use database sessions or models directly, so each access needs a REST API or Task SDK rewrite. | Inventory every session and model import early, then move reads and writes to the Task SDK or REST API. |
| SubDAG usage | SubDAGs are removed in Airflow 3 and must be rebuilt as TaskGroups, which changes how tasks are scheduled and monitored. | Convert SubDAGs to TaskGroups on the current Airflow 2 release, then test the grouped DAG before upgrading. |
| Custom operators, hooks, plugins and providers | Changed imports and provider versions require code review, unit tests and runs against Airflow 3 for each component. | Standardize custom code, pin provider versions and package custom code so it can be tested on its own. |
| Number of environments | Each environment, such as development, test and production, repeats DAG checks, configuration changes and cutover steps. | Limit environments to what the team needs and promote the same tested artifacts through each one. |
| Metadata database and executor | The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported in Airflow 3.0.0, so those setups need redesign. MariaDB is not supported as a metadata database. | Confirm database and executor versions early, and plan a database migration if moving from MariaDB or SQLite. |
| DAG delivery method | Changing how DAG files reach schedulers, workers and the DAG processor affects every deployment and rollback path. | Choose one delivery method, such as container image, git-sync or shared volume, and automate it before cutover. |
| Secrets and authentication integrations | Connections, secrets backends and the auth manager must be reconfigured, and Airflow 3 allows only one auth manager at a time. | Document each connection and role mapping, then test login and secret retrieval in a staging environment. |
| Compliance evidence requirements | Assessors may expect records of inventory, test results, change approvals and cutover decisions, which add work for every migration step. | Confirm the evidence list with your assessor first, then collect records as each migration step finishes. |
Migration phases and outputs
The sequence moves DAGs and infrastructure from Airflow 2 to Airflow 3 with a checked output at each phase, so cutover happens only after staging results match production.
Inventory DAGs, connections and dependencies
Your team and DeployOpen list every DAG, custom operator, plugin, connection, variable, secrets backend and external system the schedules touch. The list flags direct database access, SubDAG use and executor choices that need attention. Output: an inventory register with an owner and migration path for each item. The DAG inventory sets the schedule, which is confirmed in the written scope.
Upgrade to the latest 2.x release
The current Airflow installation moves to the latest 2.x patch release before any Airflow 3 work starts, because Airflow 3 requires Airflow 2.7 or later and the upgrade guide recommends this step. Output: a running 2.x environment with the DAG test suite passing and deprecation warnings recorded. Confirm your target against the version life cycle table, since Airflow 2 reached end of life on April 22, 2026.
Run Ruff AIR checks on DAG code
Engineers run the Ruff AIR301 and AIR302 rules across every DAG repository to find code that will break on Airflow 3, such as imports that move to airflow.sdk and direct database use. Findings are grouped by DAG and owner. Output: a findings report that sizes the conversion work. The step closes when every repository has a clean scan or a documented exception.
Build the parallel Airflow 3 environment
DeployOpen builds a separate Airflow 3 environment beside the current platform, using PostgreSQL 14 to 18 or MySQL 8.0 or 8.4 for the metadata database. MariaDB and SQLite cannot serve as the metadata database, and SQLite should not be used in production. The API server, DAG processor, scheduler, triggerer and workers run on Python 3.10 to 3.14. Output: an environment passing health checks, with secrets and identity wired in.
Convert DAGs and code to Airflow 3
Code moves to airflow.sdk imports and the Task SDK or REST API in place of direct database sessions and models, and SubDAGs are rewritten as TaskGroups. Engineers work through the findings from the previous step, and DeployOpen can take on conversion under fixed-scope work. Output: converted DAGs in version control with unit tests, and a rerun of the Ruff AIR checks with no blocking findings left.
Validate in staging with side-by-side runs
Converted DAGs run in staging alongside the current Airflow 2 schedules, and task states, outputs, run timing and data results are compared for the same inputs. Each difference is triaged to a code fix, a configuration change or an accepted difference. Output: a validation log per DAG with sign-off from the data owner. The step closes when every critical pipeline has matched results over an agreed set of runs.
Cut over and hand over to operations
Production schedules move to Airflow 3 in an agreed window, the Airflow 2 DAGs are paused, and the old environment stays available for rollback until the agreed exit point. Runbooks, alert routes, backup and upgrade procedures, and access are then transferred to your operations team. Output: a cutover record, a handover package and a documented decision on who runs ongoing upgrades.
What goes wrong and how to prevent it
These failures appear most often when DAGs and infrastructure move from Airflow 2 to Airflow 3. Each card names the check to run before cutover.
Runtime failures from database access
Airflow 3 blocks task code from using database sessions or models directly, so DAGs that import models or open sessions can pass linting and parsing yet fail when a task runs. Run every DAG as a task on an Airflow 3.3.2 worker in staging, and replace direct queries with REST API or Task SDK calls.
Undocumented connections and variables
Connections and variables that exist only in the old metadata database or in one engineer's memory will be missing or renamed after migration. Export every connection and variable with its owner, purpose and consuming DAG, then confirm each secret resolves through your secrets backend before the first Airflow 3 scheduler starts.
Scheduler and DAG processor misconfiguration
In Airflow 3 the DAG processor runs as its own component, and the webserver is replaced by an API server started with airflow api-server. Service definitions carried over from Airflow 2 often leave DAGs unparsed or the API unreachable. Write a component map and verify that each process starts, parses DAGs and reports health separately.
Provider and Python version conflicts
Provider packages pinned for Airflow 2 may not install alongside Airflow 3.3.2, and the image Python must fall within 3.10 to 3.14. Build a locked constraints file for the target release in a clean image, then run the full DAG import suite against it before any shared environment changes.
Unsupported metadata database versions
Airflow 3.3.2 supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, and MariaDB is not supported as a metadata database. An older server version, or a SQLite file left from a test install, can block the upgrade or lose history. Confirm the server version, back up the database and rehearse the migration on a restored copy.
Duplicate runs and missing rollback path
Running Airflow 2 and 3 side by side can trigger the same DAG twice when both systems see an active schedule. Pause each DAG in the old system before unpausing it in the new one, keep the old environment restorable, and write the rollback trigger and steps down before cutover.
What teams keep in-house and what they outsource
Each row names a migration workstream, the skills it needs, whether client teams usually keep it or hand it to a partner, and the signal that suggests moving it out.
| Workstream | Skills needed | Usually kept or outsourced | Signal to move it to a partner |
|---|---|---|---|
| Inventory and business rules | Knowledge of which DAGs feed which downstream consumers, SLAs, data owners and the business logic behind schedules | Usually kept in-house, because only the owning data and business teams know what each pipeline must do | Ownership of DAGs is unclear, few people remember why schedules or sensors exist, or the inventory spans several teams with no catalog |
| DAG code conversion | Airflow 3 authoring, Python, Ruff AIR301 and AIR302 rule fixes, airflow.sdk imports, TaskGroup rewrites for SubDAGs | Often split: client engineers keep logic changes, and a partner handles the mechanical rule-driven fixes at volume | Lint reports show many AIR findings across hundreds of DAGs, custom operators depend on removed internals, or the team has no Airflow 3 authoring experience |
| Platform build | Container images, PostgreSQL or MySQL setup, separate DAG processor and API server, executor configuration, Kubernetes or VM operations | Commonly outsourced in part when the team lacks the infrastructure skills for a new multi-component layout | No one on staff has operated the DAG processor, API server and scheduler as separate services, or the target platform is new to the team |
| Data validation | Pipeline output comparison, data quality checks, reconciliation with source systems, test design for each critical DAG | Usually kept in-house, because the data owners must confirm that outputs are correct | Validation effort is not staffed, comparison tooling does not exist, or outputs feed regulated reporting with no agreed acceptance criteria |
| Cutover | Change planning, scheduler and database switch sequencing, rollback design, communication with downstream consumers | Shared: the client owns the go or no-go decision while a partner may run the runbook | The cutover window is tight, rollback has not been rehearsed, or several pipelines share state across the switch |
| Operations after go-live | Monitoring, on-call response, upgrades to later 3.x patch and minor releases, provider package updates, database maintenance | Usually kept in-house for daily operations, with a support agreement added when internal capacity is thin | Alert volume exceeds on-call capacity, no one tracks the version life cycle, or upgrades keep being deferred |
| Compliance evidence | Change records, access reviews for Airflow roles and connections, secrets handling records, audit log retention, control mapping | Usually kept in-house, because the organization owns its control environment and auditor relationship | Auditors request evidence the team has not collected before, or no owner exists for mapping Airflow settings to existing controls |
How the work differs by starting point
Your installation already meets the minimum version for the upgrade, so the work centers on DAG code and deployment components.
- DAG code: run the Ruff AIR301 and AIR302 checks and fix flagged imports, including moves to the airflow.sdk namespace.
- Database access: find tasks that use Airflow database sessions or models directly, then move them to the REST API or Task SDK.
- SubDAGs: rewrite each SubDAG as a TaskGroup, since SubDAGs are removed in Airflow 3.
- Components: replace the webserver with the API server and run the DAG processor as its own component.
- Confirm: that your current 2.x patch level and metadata database versions match the Airflow 3.3.2 prerequisites.
If your installation runs a 2.x release earlier than 2.7, the upgrade needs an intermediate step before any Airflow 3 change.
- Intermediate hop: move to the latest 2.x patch first, as the upgrade guide recommends, so the Airflow 3 migration starts from a supported base.
- Repeat checks: run the Ruff checks again after the hop, because findings can change once the code runs on the newer 2.x release.
- Provider packages: confirm which provider package versions each DAG depends on and update them during the hop.
- Custom code: review plugins and operators for references to components that Airflow 3 removes.
- Confirm: your exact installed version, the number of hops your path requires, and the owner who signs off on each step.
Moving from cron-based jobs or a commercial scheduler is a new build rather than an upgrade. Each source job must be mapped to a DAG, and that mapping must be confirmed against the source tool.
- Job inventory: list every job with its schedule, dependencies, retry rules, alerts and owner.
- Schedule semantics: confirm how the source tool handles time zones, missed runs and backfill, then set the matching Airflow settings.
- Credentials: move passwords and keys out of scripts and into the secrets backend you choose.
- Parallel run: compare outputs of the old and new jobs for an agreed period before switching off the source.
- Confirm: the source tool's exact behavior for each job type, since commercial tools often add features that have no direct Airflow equivalent.
On Kubernetes, the migration also covers chart and image versions, because the official Helm chart uses the official Docker image.
- Version pairing: match the chart version to the Airflow 3 image and confirm your cluster falls within Kubernetes 1.30 to 1.35.
- New deployments: add the API server and DAG processor to the chart values, and remove webserver settings.
- Executors: choose one executor or several. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported in Airflow 3.
- DAG delivery: confirm how DAG files reach the new DAG processor and that the pipeline deploys the same files to each component.
- Database: confirm that the external PostgreSQL or MySQL version is on the supported list before the database migration runs.
In restricted networks, every package, image and tool used in the migration must come from an internal mirror, so the mirror set becomes part of the scope.
- Python packages: mirror Airflow, its provider packages and the Ruff version used for DAG checks.
- Container images: mirror the Airflow image and any base images the build depends on.
- Helm chart: mirror the chart version and its dependencies if you deploy on Kubernetes.
- Interpreter: confirm that Python 3.10 to 3.14 is available on build and runtime hosts.
- Confirm: who approves each mirror update, how documentation is reviewed offline, and how mirrored versions are checked against the upgrade guide.
What the client needs to provide
The migration team needs access before any DAG is changed. Provide read access to the current Airflow environment, the metadata database and the DAG repository, along with read access to scheduler, worker and task logs. Confirm that the team can run side-by-side schedules in a non-production environment, because comparing old and new behavior on the same DAGs is the main way to find differences in timing, retries and outputs before cutover.
Supply an inventory of what the DAGs depend on. This covers connections, variables, plugins and any custom providers, with the owner of each. Ask your teams to flag any task code that opens Airflow database sessions or imports models directly, since Airflow 3 blocks that pattern and those tasks need rework. Also identify any SubDAGs, which must be rewritten as TaskGroups.
Name an accountable owner for each DAG team who can confirm expected outputs, approve changes to their pipelines and make decisions during testing. Provide test data, or a safe copy of production data, and agree how outputs will be compared, such as row counts, checksums or report totals. Without these, a passing test run does not show that results match.
Make three decisions and a scheduling request early. Choose the target executor, the DAG delivery method and the secrets backend, because each changes how DAGs are packaged and tested. Agree a change window and who approves rollback. Confirm the exact Airflow 2 and Airflow 3 versions, Python and metadata database versions with your platform team, against the supported versions listed in the Airflow prerequisites, before the plan is fixed.
Questions buyers ask
What drives the cost of an Airflow 2 to 3 migration?
Cost follows the written scope. The largest drivers are the number of DAGs, how many use custom operators, hooks or provider packages, and how much code must change to remove direct database access. Who is paid for what: DeployOpen is paid for planning, code changes, testing, cutover support and documentation under a fixed-scope quote. Apache Airflow is licensed under Apache-2.0 and has no paid edition, so the software license cost is zero. Infrastructure such as the metadata database, executors and storage is a recurring cost paid to your cloud or data center provider. Ask every quote to separate one-time migration work from recurring support.
How long does an Airflow 3 migration take?
Duration comes from the written scope and the DAG inventory, so there is no fixed figure to quote in advance. The inventory shows how many DAGs exist, which ones depend on SubDAGs, direct database sessions or custom code, and which ones call external systems that need test access. Those factors, plus the number of environments and the change windows your team allows, set the schedule. Before you commit, ask for a DAG-by-DAG assessment built from Ruff AIR301 and AIR302 findings, since that assessment turns a rough estimate into a plan you can check.
Can our own team migrate DAGs to Airflow 3 without a partner?
Yes. A team with Python and Airflow skills can do the migration alone, provided it has a staging environment that behaves closely enough to production to test DAG runs. Start by moving to the latest 2.x release, because the upgrade guide requires Airflow 2.7 or later before upgrading. Then run Ruff AIR301 and AIR302 checks, rewrite SubDAGs as TaskGroups, and replace direct database access with the REST API or Task SDK. Teams that have not moved DAGs before often underestimate testing and cutover, so plan those steps first.
What do we need to provide for a migration engagement?
Your team provides access and decisions as well as code. Typically that means read access to the DAG repository and the current Airflow 2 configuration, a list of connections and secrets backends with secret values kept out of the handoff, and a staging environment with source systems that tests can reach. You also need named owners for each DAG group who can confirm expected results, a change window for cutover, and someone who can approve production access. The exact list is confirmed in the written scope, so ask for it before work starts.
What happens to Airflow 2 after April 22, 2026?
Airflow 2 reaches end of life on April 22, 2026, and after that date it receives no fixes. Airflow 2.11.2 entered limited maintenance on October 22, 2025, when only security and critical bug fixes were provided. Staying on Airflow 2 after April 22, 2026 means accepting that new vulnerabilities and bugs in the 2.x line will not be fixed upstream, and any alternative fix source must be agreed and documented by your security team. Plan the cutover before that date, or record a time-limited exception with a named owner and an end date.
Does DeployOpen certify or audit the migrated Airflow environment?
No. DeployOpen does not issue audit reports, attestations or certifications. It prepares systems, controls and evidence for compliance assessments, such as configuration records, access settings, logging and backup procedures, and documented cutover results. Independent auditors and certification bodies assess that evidence and issue any outcome. If your frameworks require specific controls, confirm the evidence list with your assessor before the migration starts.
How DeployOpen runs this migration
DeployOpen plans and carries out Apache Airflow 2 to 3 migrations inside your own infrastructure, starting from an inventory of your DAGs, connections and deployment. The work follows a written scope, with each step tested before the cutover is planned.
How we scope
- The written scope lists every DAG, provider package, custom operator and connection that moves to Airflow 3.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before the migration starts.
- The scope records the target Airflow 3 version, database, executor and Kubernetes or VM hosting as confirmed decisions.
How we build
- DAG code is checked with the Ruff AIR301 and AIR302 rules, and findings are resolved by hand where the rules cannot fix them.
- Task code moves to airflow.sdk imports and the REST API or Task SDK, removing direct database session access.
- SubDAGs are rewritten as TaskGroups, and each rewritten DAG is run in a staging environment before any cutover.
How we hand over
- You receive the migrated DAG repository, deployment configuration and a record of every rewritten pattern.
- Runbooks cover the separate DAG processor, API server, scheduler checks and the rollback path used during cutover.
- Documentation states which items you should confirm with the Airflow maintainers or your own environment before go-live.
How we support
- Ongoing support, upgrades and testing are available as a retainer after the migration is complete.
- A retainer can cover keeping Airflow on a release that still receives fixes, as set out in the project's version life cycle.
- Support scope, response expectations and review cadence are written into the agreement before it starts.
