Why SubDAG usage must be rewritten before the Airflow 3 move
If your Airflow 2 DAGs use SubDagOperator, the rewrite is a prerequisite for Airflow 3, not an optional cleanup. SubDAGs are removed in Airflow 3. The upgrade guide names the replacements: TaskGroups for grouping tasks inside a single DAG, and assets or data-aware scheduling when a sub-pipeline is really a dependency between DAGs. Your first job is to inventory every SubDagOperator and decide, per use, which replacement fits. The Airflow 3 upgrade guide is the reference for the removal and should be read alongside your own DAG code.
The supported path is to move to the latest 2.x release first, then to Airflow 3. The upgrade guide requires Airflow 2.7 or later before upgrading and recommends the latest 2.x as the stepping point. Rewriting SubDAGs belongs in that pre-upgrade work, on the 2.x line, where you can run the old and new DAG versions side by side and compare their behavior before any scheduler or metadata database changes. Doing the rewrite during the Airflow 3 cutover combines two sources of failure in one window and makes it hard to tell which change caused a broken run.
Airflow 2 reached end of life on April 22, 2026, after which it gets no fixes, so the rewrite should not wait for a deadline. Teams that postpone it tend to discover the SubDAG usage only when the Airflow 3 upgrade fails on import. Confirm with your platform team which DAG repositories still contain SubDagOperator calls, who owns each one, and whether any downstream consumers depend on the SubDAG task IDs. Those answers set the size of the work before you agree on a scope and timeline.
What Airflow 3 removes and what replaces it
This table lists the Airflow 2 features removed in Airflow 3 that teams typically meet in the same rewrite as SubDAGs, with the impact of each and the action it requires. Read the SubDAG row first, then check the adjacent rows against your DAG code and scripts, since they are usually changed in the same pass.
| Change | Impact | Action needed |
|---|---|---|
| SubDAGs removed in favor of TaskGroups | SubDAG DAGs no longer run, so pipelines that nest work inside SubDagOperator stop in Airflow 3 | Rewrite each SubDAG as a TaskGroup for in-DAG grouping, or as separate DAGs linked by assets where work is shared |
| Data-aware scheduling through assets | Cross-DAG dependencies that relied on SubDAG nesting need an explicit producer and consumer relationship | Decide per SubDAG whether a TaskGroup fits or whether an asset-triggered DAG models the dependency better |
| SLAs replaced by Deadline Alerts | SLA callbacks and SLA miss reporting in SubDAG-era DAGs do not carry over unchanged | Inventory SLA settings and rebuild the alerting intent with Deadline Alerts, then test the alert path |
| SequentialExecutor removed | Single-process test setups that ran SubDAGs sequentially no longer start on Airflow 3 | Move local or test runs to a supported executor, such as LocalExecutor, and confirm the test environment still runs the rewritten DAGs |
| CeleryKubernetesExecutor and LocalKubernetesExecutor removed | Hybrid executor configurations fail to start on Airflow 3.0.0 and later | Choose a supported executor or combination of executors, and update executor settings before cutover |
| --subdir flag removed | Scripts, CI jobs and service definitions that pass --subdir to Airflow commands fail | Search deployment code and runbooks for --subdir and remove the flag from each command |
| REST API /api/v1 removed | Scripts that trigger DAGs or read run state through /api/v1 stop working | Move clients to /api/v2 and confirm authentication changes before the rewritten DAGs go live |
Choosing TaskGroup, assets or data-aware scheduling
Many SubDAGs exist only to bundle related tasks so the graph is easier to read. In this case a TaskGroup is the direct replacement. It keeps the tasks in the same DAG run, under the same schedule and the same retry and trigger settings, so the rewrite is mostly structural. The tasks still execute in one DAG run, and no separate DAG run is created for the group.
- Scheduling stays the same: the group inherits the parent DAG's schedule, and no second DAG appears in the scheduler.
- UI changes: the Grid and Graph views show collapsible groups in place of the SubDAG node, so operators click into the group rather than opening a child DAG.
- Task IDs gain a prefix: tasks inside a TaskGroup are named with the group ID as a prefix, which changes history keys unless you plan for it.
- Dependencies move with the group: set dependencies on the group object, and check that edges into and out of the group still match the old SubDAG.
- Shared state is not isolated: a TaskGroup offers no separate pool or run boundary, so resource limits must be set on the tasks themselves.
Before you rewrite, confirm that the SubDAG never had its own schedule or was not triggered by another DAG. Those cases belong to the other tabs.
Some SubDAGs were used to hand data from one pipeline to another, for example when an upstream load had to finish before a downstream transform started. A TaskGroup does not solve this, because it only organizes tasks inside one DAG. Assets and data-aware scheduling fit better: the producing DAG declares an asset it updates, and the consuming DAG is scheduled when that asset changes. The two pipelines then depend on data, not on a hidden parent-child call.
- Dependency becomes explicit: the producer's outlets and the consumer's schedule name the same asset, which is easier to review and audit.
- Scheduling changes: the consumer runs when the asset event occurs, so its timing follows the producer rather than a fixed cron expression.
- UI changes: operators see asset events and the DAG runs they triggered, instead of a SubDAG node inside the parent run.
- Ownership splits: each team can own its DAG, which can matter for access control and change management.
- Failure behavior differs: a failed producer simply produces no asset event, so monitor for missing events, not only failed tasks.
Before you choose this path, map which data is actually exchanged and whether it is stored somewhere a consumer can read. Passing data through XCom inside a single run is a different question and should be checked separately.
Some SubDAGs ran the same sub-pipeline many times, or had their own schedule and were triggered from several parents. These SubDAGs usually behave like independent workflows, and the decision is whether they should become a separate DAG. A TaskGroup cannot carry its own schedule or be triggered independently, so it only fits when the group always runs as part of its parent. If the sub-pipeline must run on its own, promote it to a standalone DAG and connect it to the parents with a trigger or an asset.
- Separate DAG when it runs alone: if operators start it independently or it has a schedule, give it a DAG ID and its own runs.
- Dynamic generation needs care: a repeated sub-pipeline is often generated in a loop, so confirm that the generated task and DAG IDs are stable across parses.
- Run history is separate: a standalone DAG keeps its own run history, which helps audit but means dashboards and alerts must be updated.
- Triggering is explicit: use a trigger from the parent or an asset event, and record which runs depend on which.
- Concurrency limits move: pools and max active runs apply per DAG, so review capacity once the sub-pipeline has its own run queue.
Ask the owners of each SubDAG who starts it, when it runs and which downstream jobs rely on its output. The answers usually settle which tab applies.
Keeping task IDs stable so run history stays readable
A TaskGroup prefixes the task IDs of its children with the group ID, so a task named load_orders inside a group called sales becomes sales.load_orders. A straight swap from a SubDagOperator to a TaskGroup therefore changes every task ID in the rewritten section. That breaks the link between new runs and the history of the old tasks, and it can silently break alerts, dashboards, SLA rules and any script or query that looks up task instances by ID. Before you rewrite anything, export the task IDs your monitoring and reporting depend on, and list which downstream systems reference them.
You have two workable paths. The first is to keep the old IDs by choosing group and task names deliberately, for example by setting the TaskGroup group_id and the task names so that the resulting full IDs match what the SubDAG produced. This keeps history continuous, but it depends on how the prefixing rules apply to your specific structure, so verify it rather than assume it. The second is to accept new IDs and plan the remapping: record the old-to-new ID mapping, update alert rules and dashboards, and decide how you will report on runs that span the cutover. Choose one path per DAG and write the decision down, because mixing them across a single pipeline makes history harder to read.
Confirm the behavior in a staging environment that contains a copy of your real DAG history, not an empty instance. Trigger the rewritten DAG, then check the task instance IDs it produces against your exported list, and confirm that the views, alerts and queries you depend on still return results for the new runs. Ask your platform team to confirm how the Airflow 3 upgrade guide's removal of SubDAGs applies to your DAGs before you start, since that guide is the reference for the rest of this migration: Upgrading to Airflow 3.
What to inventory and prepare before rewriting
Rewriting SubDAGs is a change to DAG code, task identity and scheduling history at the same time. The six preparation steps below connect the code inventory, the data you must keep, the test environment and the way back if a rewrite goes wrong.
Find every SubDagOperator
Search the whole DAG repository, including shared libraries, dynamically generated DAGs and vendored code, for SubDagOperator and subdag references. Record the parent DAG, the subdag factory function and whether the subdag is generated in a loop, since loops multiply the number of tasks you must rewrite and retest. Note any SubDAG whose internal tasks use pools, queues or custom retries, because those settings must move to the TaskGroup.
Record current task IDs
Export the task IDs of every task inside each SubDAG, along with the parent task ID the SubDAG currently uses. A TaskGroup prefixes child task IDs with the group ID, so load.extract will differ from extract unless you choose the group name deliberately. Keep this list as the mapping you test against when you decide which IDs to preserve.
Map downstream consumers
Identify everything that reads task IDs or task state: XComArg references, ExternalTaskSensor waits, trigger rules, alerting rules, dashboards, runbooks and any script calling the REST API. Each consumer that refers to a renamed task will fail or silently point at nothing after the rewrite. Confirm ownership of each consumer before the change window so that someone can update it.
Back up the metadata database
Take a consistent backup of the metadata database before changing any DAG file, because rewritten DAGs are reserialized and run history is tied to task instances. Confirm that the backup restores into a separate instance and that you can query its DAG run and task instance tables. Your production database should be on PostgreSQL or MySQL, since the production deployment guide warns that SQLite can lead to data loss.
Stage with production-like DAG volume
Run the rewritten DAGs in a staging environment that matches production in Airflow version, executor, number of DAGs, schedule frequency and the number of concurrent scheduler instances. Parse-time problems from TaskGroups or generated tasks often appear only at higher DAG counts. Ask the platform team to confirm that staging uses the same database engine and secrets backend as production, so test results carry over.
Plan a DAG code rollback
Keep the last known good commit of each DAG repository tagged and deployable, and decide whether a rollback reverts DAG files only or also the Airflow version. Test how the scheduler behaves when a DAG file reverts to SubDAG code after runs have started under TaskGroup IDs. Agree on the owner, the trigger conditions and the window in which rollback is still safe.
How to rewrite and verify SubDAG usage in order
This sequence turns SubDAG usage into TaskGroups or asset-based designs on the 2.x line, proves the rewritten DAGs match their baseline behavior, and leaves a clean rollback path before the Airflow 3 upgrade starts.
Inventory every SubDAG in the repository
A platform engineer searches the DAG repository for SubDagOperator and subdag factory functions, then lists each DAG file, the parent DAG it belongs to, the tasks inside it, and the schedule and downstream dependencies. Include DAGs that generate SubDAGs dynamically, because a text search can miss them. The step is done when the inventory covers every DAG that deploys to the scheduler, and a second reviewer has confirmed the count against the production DAG list. Keep a copy of the current inventory as the baseline for later comparisons.
Classify each SubDAG by intent
For each entry, a DAG author and the data owner decide why the grouping exists. A SubDAG used only to make the graph readable usually maps to a TaskGroup. A SubDAG that runs a fixed set of work on one schedule with its own outputs, or that is triggered by data arriving, may fit better as separate DAGs connected through assets or data-aware scheduling. Record the chosen pattern, the reason, and any DAG that needs a design review. The check is a classification table where every row has one pattern and one owner, and the table is approved before code changes begin.
Rewrite each DAG in a feature branch
A DAG author replaces each SubDagOperator with a TaskGroup or splits the work into separate DAGs that publish and consume assets, changing one DAG per commit where possible. Remove imports of the SubDAG factory once nothing references it. Keep the original files untouched on the main branch so the 2.x production DAGs stay as they are. The step closes when the branch builds, a diff review confirms that dependencies and schedules match the classification table, and the team agrees that the branch is the only candidate for promotion. Rollback is simply to delete or abandon the branch.
Decide task ID mapping before the rewrite merges
Task IDs drive history in the metadata database, so the author decides for each task whether the new ID stays identical to the old fully qualified name or moves to a new name with a documented mapping. A TaskGroup adds a group prefix to child task IDs, which changes them, so teams usually reproduce the old names or keep a mapping file that operators and dashboards can read. Confirm with the team that owns alerts and reports which names they query. The check is a written mapping for every renamed task, reviewed by the operations owner. If the mapping is unclear, leave the old DAG running and do not merge.
Run DAG parse checks and unit tests
The CI pipeline imports every changed DAG file, which confirms the scheduler's parser can load it without errors, then runs the unit tests that cover dependency structure, task IDs, and any custom callbacks. Add a test that asserts the expected task IDs and edges from the mapping created in the previous step. The step is done when the pipeline passes on the branch and the test output lists each DAG and its tasks. If a test fails, fix the branch and rerun, and do not relax the expected graph to make the test pass.
Compare run graphs in staging against the 2.x baseline
Deploy the branch to a staging Airflow 2.x environment that holds a copy of production configuration and connections, then trigger the same logical dates in both the baseline DAG and the rewritten DAG. Compare task states, durations, outputs, XCom values, and the generated graph. Investigate every difference, including changed trigger rules caused by different group structures. Keep the baseline DAG deployed in staging so either version can be run again. The check is a signed comparison report showing matching outputs, and the data owner accepts any intended difference in writing.
Promote the rewrite to production on the 2.x line
The release owner merges the approved branch and deploys it to production while the scheduler still runs Airflow 2.x, so the change is isolated from the platform upgrade. Pause the rewritten DAGs during the first deployment if the team uses that control, then unpause them after the first scheduled run has been reviewed. Keep the previous DAG version available in version control so it can be redeployed. The step is done when the first production runs match the staging comparison and the operations owner confirms that alerts and dashboards show the expected task IDs.
Proceed with the Airflow 3 upgrade and keep the fallback
Once the rewritten DAGs have run cleanly on 2.x in production, the team continues with the Airflow 3 upgrade and runs the Ruff AIR checks against the repository to confirm no SubDAG references remain. Keep the pre-rewrite DAG version and the baseline comparison report available until the upgraded environment has completed its verification runs. The step closes when the upgrade checks pass and no removed SubDAG code is found. If a regression appears after cutover, redeploy the archived 2.x DAG version as the rollback point.
How to test rewritten DAGs before the Airflow 3 cutover
Testing a SubDAG rewrite starts with proving that every DAG still parses and loads under the Airflow version you will run after cutover. Run the parse test against the same Airflow 3 environment your staging scheduler and DAG processor use, not only against a developer laptop on Airflow 2. Then compare the old and new structures programmatically: for each DAG that contained a SubDagOperator, export the task IDs, upstream and downstream edges, and trigger rules from the old DAG and from the TaskGroup version, and review every difference. A TaskGroup prefixes child task IDs with the group ID, so a rename that you did not intend will show up here before it reaches production history.
Assert task IDs explicitly in the tests whenever run history, dashboards, alerts or downstream consumers depend on them. Keep the expected IDs in version control next to the DAG, and fail the build if a task that existed before the rewrite no longer appears under the same name. Dependency checks should cover trigger rules as well as edges, because a fan-in that previously ran through a SubDAG may have depended on a trigger rule that you need to carry over. Use the Ruff AIR checks described in the upgrade guide as a first gate, but treat them as a partial control: they flag known breaking patterns and can apply some rewrites, yet they will not tell you that a TaskGroup now runs its children in a different order or that a sensor's timing changed.
Finish with a staging run over a representative data window, including at least one period with late or partial source data and one retry scenario. Confirm with your own data owners which window is representative, since that choice is specific to each pipeline. Compare outputs, row counts and task states against the production run for the same logical dates, and keep the old SubDAG DAG paused but available until the comparison is signed off. Record the results as cutover evidence so that the decision to switch is traceable.
What makes the SubDAG rewrite take longer
How many SubDAGs and how deep do they nest?
The count and nesting depth are the biggest effort drivers, because each SubDAG is a separate DAG object with its own dependencies, retries and failure paths. Start by listing every SubDAGOperator across the repository, then record how many levels of nesting each one has and how many tasks sit inside. A flat set of a few SubDAGs usually converts in a straightforward pass. Deeply nested SubDAGs, or SubDAGs that call other SubDAGs, need the outer structure redesigned before the inner tasks can be moved. Measure the number of SubDAG instances, the maximum depth, and the total task count inside them, and track which DAGs share a SubDAG factory function, since one change then affects several pipelines.
Do SubDAGs pass data through XCom or external state?
Data flow between a parent DAG and its SubDAG decides whether the rewrite is a mechanical swap or a design change. When tasks exchange small values through XCom, a TaskGroup can usually keep the same dependencies and data passing. When SubDAGs read and write external state, such as a table, a file path or a object store prefix that another pipeline also reads, the team must decide what that state represents now. In that case, assets or data-aware scheduling may fit better than a TaskGroup, and each producer and consumer needs an explicit asset definition. Count the XCom keys each SubDAG pushes and pulls, list every external location it touches, and note which downstream consumers depend on those outputs.
Which alerts, dashboards or scripts depend on existing task IDs?
Task IDs change or gain a group prefix when SubDAG tasks move into a TaskGroup, so anything that looks up tasks by name can break. Alert rules, dashboard queries, SLA or monitoring filters, runbooks, backfill scripts and CI jobs are the usual casualties. Search for every hard-coded task ID, including string matches in metrics labels and in calls to the REST API, and map each old ID to its new ID before any code changes. Plan how run history will remain readable, and decide whether to keep the old IDs for continuity or to accept a clean break with a documented cutover. The effort grows with the number of consumers that were never documented.
Do custom operators wrap SubDAGs internally?
Custom operators that build SubDAGs in their execute or constructor logic hide the pattern from a simple code search, so they often take longer than the visible DAG files suggest. The team needs to find every operator class that imports SubDAGOperator or builds a DAG object internally, then decide whether the operator should emit a TaskGroup, a set of ordinary tasks or an asset-producing task. Each custom operator also has its own tests, versions and consumers across teams. Ask for a list of internal operator packages, their owners, and whether they are published as shared libraries, because a change in a shared library requires coordinated releases.
How many teams own the DAGs being changed?
Ownership affects how fast decisions get made and how many review cycles the rewrite needs. When one team owns every affected DAG, the work can follow one review path and one release schedule. When several teams share SubDAG factories or depend on each other's outputs, the rewrite needs agreement on task naming, asset names, change windows and who signs off on cutover. Map each DAG to an owning team and a named contact, note which DAGs are shared across teams, and record which teams must test their pipelines before the change is merged. Unclear ownership is often the longest delay in these projects.
What test data and staging environment exist for these DAGs?
Staging coverage determines how much of the rewrite can be verified before production. Teams with a staging Airflow that runs representative data, with comparable schedules and source connections, can compare old and new runs side by side. Teams without that environment need to build or borrow one, and synthetic data may not reproduce edge cases such as late-arriving files or partial failures. Check whether each SubDAG has input fixtures, expected outputs and known failure cases, whether staging can read production-like sources without exposing regulated data, and how long a full DAG run takes there. Record these answers as part of the estimate, because missing test data usually adds more time than the code changes themselves.
What should the team measure before estimating the rewrite?
Measure a short inventory that covers each driver above, so the estimate rests on counts rather than impressions. Useful measurements include the number of SubDAG instances and their maximum depth, the total tasks inside them, the XCom keys and external locations each one uses, the custom operators that wrap them, the teams and named contacts for each DAG, the task IDs referenced outside the DAG code, and the staging runs available for comparison. Ask the internal team or a partner to convert these counts into a written scope with deliverables and a timeline before work starts. Any cost or duration figure should come from that agreed scope and the vendor's upgrade documentation, not from an assumption.
Can the SubDAG rewrite be scoped as a fixed-price engagement?
DeployOpen works mostly on fixed-scope engagements, so the rewrite can be quoted once the inventory is complete and the scope is agreed in writing. The quote depends on the drivers above, along with the number of DAGs in scope, whether testing and staging setup are included, and how many handoffs to other teams are required. Ask for a written scope that lists the DAGs covered, the testing approach, the staging needs and the client's responsibilities, such as approvals and data access. DeployOpen does not publish a price list, so the quote follows from that written scope.
What a fixed-scope SubDAG rewrite covers
A fixed-scope SubDAG rewrite starts with a written scope that names every DAG in the repository that uses SubDagOperator, the replacement chosen for each one, and the task ID policy that applies to the rewrite. Each SubDAG gets a line in the scope document showing the parent DAG, the child logic it wraps, whether the replacement is a TaskGroup or an asset-based design, and the reason for that choice. The list also records DAGs that only look like SubDAG candidates, such as those that call a shared function, so the reader knows what was reviewed and left alone. Because the scope is agreed in writing, the reader can check it against their own inventory before any code changes, and the quote follows from that document rather than from an estimate of the code base.
The task ID policy is the part most likely to affect history, so it should be decided before the build begins. The scope should state whether existing task IDs are preserved, how TaskGroup prefixes are applied, and how renamed tasks are mapped to their previous IDs for monitoring and alerts. Test deliverables usually include unit tests for task logic, DAG import checks, and comparison of the task graph before and after the rewrite, with the run results in a staging environment that uses the same metadata database engine as production. The scope also names the Ruff AIR checks run on the repository and the Airflow version the work targets. DeployOpen plans and implements the changes in the client's own repository and environment, and handover notes describe the final graph, the test suite, the staging evidence and open decisions for the client's team.
How DeployOpen works on this
DeployOpen scopes the SubDAG removal as a fixed-scope engagement, starting from an inventory of every SubDagOperator in your DAG repository. The work is planned, rewritten, tested and documented in your own infrastructure, with ongoing support available as a retainer.
How we scope
- Inventory every SubDagOperator in your DAG repository and record which ones hold independent scheduling or pool settings.
- Classify each SubDAG as a TaskGroup rewrite, an asset or data-aware scheduling candidate, or a design decision for your team.
- Agree in writing which DAGs, Airflow versions and environments the scope covers before any rewriting starts.
How we build
- Rewrite selected DAGs with TaskGroups, keeping task IDs stable so existing run history and dependencies still resolve.
- Run Ruff AIR checks on the rewritten code and record the remaining findings against your target Airflow 3 version.
- Test each rewritten DAG in a staging environment against representative runs, comparing task graphs and outputs with the original.
How we hand over
- Deliver the rewritten DAG code, a mapping of old SubDAG names to new TaskGroup IDs, and the test results.
- Document the decisions taken for each asset or data-aware scheduling change, so your engineers can review the reasoning later.
- Walk your team through the changes and confirm which owners will maintain the DAGs after the engagement closes.
How we support
- Support continues after handover through a retainer, covering questions about rewritten DAGs and follow-up fixes if agreed in scope.
- Retainers can also cover the later Airflow 3 upgrade steps, such as the upgrade path from 2.11 and checks on newer 3.x releases.
- Confirm the support hours, response expectations and change process with DeployOpen before signing, since these are set in the written agreement.
