Airflow metadata database migration: why the database is the upgrade's riskiest part
Teams moving a metadata database through an Airflow upgrade should back up consistently, trim history, reserialize DAGs, run `airflow db migrate`, and plan a rollback before anything changes. The database holds DAG runs, task instances, connections and variables, so a failed or partial migration puts the whole scheduling state at risk, not just one pipeline. Airflow 2 reached end of life on April 22, 2026, and the supported path to Airflow 3 requires Airflow 2.7 or later, with the latest 2.x release recommended first as the stepping stone. If your estate is still on an older 2.x line, confirm that step as the first item in your plan.
The Airflow 3 upgrade guide calls airflow db migrate the biggest part of an Airflow upgrade, which is why the database work deserves its own plan rather than a line item in a deployment checklist. Schema migrations can be slow on large databases, and the guide recommends trimming old data with airflow db clean beforehand and running airflow dags reserialize to surface errors such as AirflowDagDuplicatedIdException while the old system is still running. Confirm with your own environment how many rows your largest tables hold, how long a migration took in a staging copy, and whether your backup tooling can produce a consistent snapshot within that window.
Treat the database as the rollback boundary. Once the migration has run, the schema no longer matches the Airflow 2 code, so restoring the pre-upgrade backup is usually the only clean way back. That makes the backup itself a control you must test, not assume. Decide who owns the go or no-go call, how long the scheduler may stay paused, and what evidence proves the restored database is consistent. Airflow supports more than one scheduler using database row-level locks, so confirm that all scheduler and worker components are stopped during the snapshot and migration. The sections that follow cover each step in order.
What changes in the metadata database and runtime during the upgrade
This table compares each change that touches the metadata database or the runtime layout during an Airflow 3 upgrade, with its impact and the action to take before the cutover. Read it top to bottom as a pre-upgrade checklist, since the backup and history trim come before the migration step. The main takeaway is that most upgrade failures come from skipped preparation rather than from the migration command itself.
| Change | Impact | Action needed |
|---|---|---|
| Mandatory metadata database backup | The upgrade guide asks for a backup first. A hot backup taken during active scheduling may not be consistent. | Back up before migration. If a hot backup cannot be made consistent, stop Airflow so the backup is clean. |
| Trimming history with airflow db clean | Schema migrations can run slowly on large databases with years of task and log history. | Review the retention window, run airflow db clean on a copy first, and time it against your downtime window. |
| airflow dags reserialize before upgrading | Stale or conflicting DAG records can make the migration or the new scheduler fail. | Run airflow dags reserialize on the Airflow 2 side and fix errors such as AirflowDagDuplicatedIdException first. |
| airflow db migrate as the largest step | The upgrade guide calls this the biggest part of the upgrade, and it rewrites the schema in place. | Plan it as the main outage window, rehearse it on a restored copy, and confirm rollback steps before starting. |
| Separate DAG processor | Airflow 3 requires the DAG processor to run as its own component rather than inside the scheduler process. | Provision and monitor the DAG processor, and confirm it can reach the same DAG files and metadata database. |
| Webserver replaced by API server | The webserver is replaced by an API server started with airflow api-server, which changes the process layout and ports. | Update service definitions, health checks and load balancer routes to point at the API server. |
| No direct database sessions in task code | Tasks can no longer use Airflow database sessions or models directly and must use the REST API or Task SDK. | Find tasks that query or update the metadata database and move them to the REST API or Task SDK before cutover. |
| PostgreSQL 14 to 18 or MySQL 8.0, 8.4 and Innovation | These are the supported metadata database versions for Airflow 3.3.2. Older database versions need their own upgrade first. | Confirm the engine version against the prerequisites before the migration, and plan any database upgrade as a separate step. |
| MariaDB and SQLite not supported for production | MariaDB is not supported as a metadata database, and SQLite should not be used in production. | Migrate off MariaDB or SQLite before the upgrade, since the target platform cannot run on them. |
| At least 4 GB RAM recommended | The prerequisites recommend at least 4 GB of RAM, which the migration host or component sizing should respect. | Confirm memory for the scheduler, API server and DAG processor hosts, and the database host if it shares resources. |
Which upgrade path fits your starting version
Airflow 3 requires you to be on Airflow 2.7 or later before you upgrade, so this is the direct starting point. The upgrade guide still recommends moving to the latest 2.x patch first, which is 2.11.2 at the time of writing, so that the migration runs against the schema and tooling you will check in Airflow 3. Airflow 3 first shipped on April 22, 2025, and its current patch is 3.3.2.
- Move to 2.11.2 first: Upgrade within the 2.x line so the metadata database schema matches the latest Airflow 2 release before you start the Airflow 3 migration.
- Run the cleanup and reserialize steps on 2.x: Trim old history with
airflow db cleanand runairflow dags reserializewhile the old version still runs, so problems surface before the cutover. - Back up before each version change: Take a consistent backup before the 2.x step and again before the Airflow 3 step.
- Check DAG code early: Use the Ruff AIR301 and AIR302 rules to find breaking changes before you change any binaries.
- Plan the downtime window around `airflow db migrate`: This is the largest part of the change, so measure it on a staging copy of your database.
Confirm your exact current patch version and the schema state in a staging environment before you schedule the production window.
If you run Airflow 2.11 or an older 2.x release, the first task is to determine whether you can reach Airflow 3 directly. Releases earlier than 2.7 do not meet the minimum in the upgrade guide, so they need an intermediate 2.x upgrade before any Airflow 3 work begins. Upgrading to 2.11.2 is the path the guide points to, and it also gives you the most time to test the metadata database changes on a version that still receives limited maintenance.
- Confirm the minimum version: Airflow 3 requires 2.7 or later, so releases older than that must step up first.
- Plan the intermediate upgrade as its own change: Treat the move to 2.11.2 as a separate release with its own backup, migration and rollback point.
- Expect more schema drift on older databases: The further behind you are, the more migrations run during the upgrade, which affects the downtime window.
- Check provider packages during the intermediate step: Installing the standard provider on 2.x helps the DAGs you carry into Airflow 3.
- Document the starting state: Record the exact version, database engine and size so the migration timing can be estimated for your environment.
Ask your team to confirm the current version and the number of intermediate steps required before scheduling.
Airflow 2 reached end of life on April 22, 2026, and end-of-life versions receive no fixes or support. Limited maintenance, which covers security and critical bug fixes only, began on October 22, 2025, so any 2.x release you run now is outside the supported window. Your database work becomes a security and continuity issue as well as an upgrade task, because the metadata database holds the state of every run and any compromise or failure has no vendor fix path on that release.
- Expect no security fixes: Any vulnerability found in your 2.x release will not receive a patch from the project.
- Upgrade to the last 2.x line first where possible: Moving to 2.11.2 is the recommended stepping stone toward Airflow 3.
- Prioritize a verified backup: Restore a backup to a test environment before you rely on it for the migration.
- Set a rollback plan with a named owner: Decide who makes the rollback call and what evidence triggers it.
- Involve the security team early: Access to the database and the backup store should be reviewed during the window.
Confirm with your security and compliance leads how long an unsupported release may run in your environment before you commit to a migration date.
How to prepare the metadata database before you touch the schema
Schema migration is the step most likely to fail or run long, so the work before it decides how safe the upgrade is. The cards below cover the sequence we would expect a team to follow: take a consistent backup, reduce what the migration has to process, clear DAG errors that block reserialization, and rehearse everything on a copy of production.
Take a consistent backup
The Airflow 3 upgrade guide asks you to back up the metadata database first. If the database cannot be backed up hot without risking an inconsistent copy, stop the scheduler, API server, DAG processor and workers before the dump so no task writes during the backup. Confirm with your DBA whether point-in-time recovery with continuous archiving is already in place, since it lets you restore to a moment just before the migration rather than only to the last nightly dump.
Trim history on a copy first
Run airflow db clean against a restored copy to remove old task instances, logs and other history before the production migration. The upgrade guide notes that schema migrations can be slow on large databases, so the trimmed size is a direct input to your downtime estimate. Agree a retention window with the data owners, record the rows removed, and keep the pre-trim backup until the upgrade is accepted.
Reserialize DAGs and fix duplicate IDs
Run airflow dags reserialize before the migration and fix errors such as AirflowDagDuplicatedIdException first. A duplicate DAG ID found during reserialization is a repository problem, so it should be resolved in code rather than suppressed in the database. Treat the reserialize output as a go or no-go input and attach it to the change record.
Rehearse on a staging clone
Restore the production backup into a staging database that matches the production engine and version, then run the full sequence: trim, reserialize, airflow db migrate and a smoke test of the API server and scheduler. Record the measured migration time and compare it with the window you plan to request. Confirm that PostgreSQL versions and any PgBouncer settings in staging match production, because a pooler setting that differs can hide connection problems until cutover.
Check pooling and engine support
If PgBouncer sits between Airflow and PostgreSQL, confirm how it handles session state and transaction pooling during the migration and any connection drain, and test that the migration user can run DDL through it. Also confirm the engine is supported for Airflow 3: MariaDB is not a supported metadata database and SQLite should not be used in production. A failed engine check found in staging is far cheaper than one found during the window.
The upgrade sequence with checks and rollback points
This sequence moves the Airflow metadata database to the Airflow 3 schema while keeping a verified backup and a defined way back at each stage. Each step closes with a check that tells the team whether it can safely continue.
Confirm version and prerequisites
The owning platform engineer confirms the current Airflow version is 2.7 or later, ideally the latest 2.x, and that the metadata database engine is PostgreSQL or MySQL on a supported version. Confirm Python, Kubernetes if used, and at least 4 GB of RAM for the components against the Airflow 3.3.2 prerequisites. Done when a written compatibility sheet lists each check as passed, with the target Airflow patch release named. Rollback point: none yet, since nothing has changed.
Rehearse the upgrade on a restored copy
Restore a recent production backup into a separate, isolated database and point a staging Airflow at it. Run the trim, reserialize and migration steps there first, and record how long the migration takes and which errors appear. Done when the rehearsal completes without schema errors and the team has a timed runbook. Rollback point: the staging copy is discarded, production is untouched.
Take the backup and verify the restore
Stop writes if a hot backup cannot guarantee consistency, then take a full logical or physical backup of the metadata database. Restore it to a scratch instance and run a row count and a sample query against key tables such as dag_run and task_instance. Done when the restore test matches the source and the backup is stored outside the Airflow host. Rollback point: this verified backup is the restore target for every later step.
Trim history with airflow db clean
Run airflow db clean with a retention cutoff agreed with the data owners, first as a dry run and then for real on a rehearsal copy before production. Keep the cutoff documented, since trimmed run history cannot be recovered from Airflow itself. Done when the dry run output matches expectations and the table sizes drop as expected on the live database. Rollback point: restore the verified backup if the trim removed records that auditors or downstream teams still need.
Reserialize DAGs and fix errors
Run airflow dags reserialize against the current Airflow 2 environment and resolve reported failures, including any AirflowDagDuplicatedIdException errors, by renaming or removing duplicate DAG IDs. Run the Ruff AIR301 and AIR302 checks on the DAG repository in the same pass. Done when reserialization finishes with no errors and the Ruff output is clean or each finding has an owner. Rollback point: revert the DAG repository commit that introduced the fixes.
Stop the scheduler, workers and API components
Drain running tasks where possible, then stop the scheduler, all workers, the webserver and any triggerers so nothing writes to the metadata database during migration. Confirm no task instances are in a running state and no processes still hold database connections. Done when the database shows no active connections from Airflow components. Rollback point: restart the Airflow 2 components against the restored backup if the downtime window must end early.
Run airflow db migrate
Run airflow db migrate with the Airflow 3 installation, using the same database connection the rehearsal used. Capture the full output and any errors, since the upgrade guide describes this step as the largest part of the upgrade. Done when the command exits cleanly and the migration version recorded in the database matches the target release. Rollback point: if the migration fails or corrupts state, restore the verified backup and restart the Airflow 2 components.
Deploy Airflow 3 components and verify runs and logs
Start the API server with airflow api-server, run the DAG processor as its own component, then start the scheduler and workers. Trigger a sample of representative DAG runs and confirm task logs load in the UI and API. Done when scheduled runs, manual runs and log retrieval work and the team signs off on the agreed checks. Rollback point: until sign-off, the restored backup and the prior Airflow 2 release remain available for return.
How to time the downtime window and plan rollback
The downtime window is set by two things: how long airflow db migrate takes on a rehearsed copy of your production metadata database, and how long the cutover around it takes. Measure the first on a restored copy of the backup, not on an empty test database, because schema migrations are slow on large histories and the trimming done with airflow db clean changes the result. Airflow supports running more than one scheduler at once, but that does not remove the need for a controlled stop: the migration changes the schema the schedulers, the API server and the DAG processor all read, so stop the components, run the migration once, then bring them back on the new version. Confirm the exact sequence for your deployment against the Airflow 3 upgrade guide, including where airflow dags reserialize fits.
Rollback means restoring the verified backup to the previous Airflow version, not reversing migrations by hand. Write down the restore procedure, the owner who can start it, and the point at which you stop trying to fix forward. Keep the previous Airflow image and the matching constraint files available so the old version can start against the restored database. Any run that completed on the new version after cutover is lost on restore, so decide in advance how you will handle those runs and which DAGs you will pause during the decision window. PostgreSQL facts shape the database side: minor releases never change the internal storage format, so a minor update means replacing executables and restarting. Major upgrades use pg_dumpall and restore, pg_upgrade, or logical replication. Logical replication works across major versions and allows a major upgrade with only several seconds of downtime, but it needs its own rehearsal, monitoring and cutover steps. Confirm the supported PostgreSQL version against the PostgreSQL versioning policy before you choose the path.
Automatic failover is not part of PostgreSQL itself. The database provides hot standby and synchronous or asynchronous replication, but it does not detect a primary failure and promote a standby, so any automatic promotion during the upgrade window depends on external tooling that you select, test and operate. Do not let an unplanned failover happen in the middle of a migration. Freeze the topology before the window, record which node is primary, and decide whether a standby will be used for rollback. Ask the team responsible for PostgreSQL to confirm the point-in-time recovery procedure covers the upgrade day, since restoring a base backup alone may leave you short of the state you need.
What makes a metadata database upgrade take longer
This table lists the factors that stretch a metadata database upgrade and what to check for each one before scheduling the downtime window. Read it as a pre-upgrade inventory: a large history table, many DAG files and a database engine outside the supported range each add time or risk, so measure them in your own environment first.
| Driver | Why it matters | What to check |
|---|---|---|
| Database size and history volume | The Airflow 3 upgrade guide notes that schema migrations can be slow on large databases, so the migrate step grows with stored history. | Measure table sizes and row counts for task instances, DAG runs and logs, then run airflow db clean on a copy to estimate the reduction. |
| Number of DAG files and duplicate IDs | Serialization must succeed for every DAG before migration, and duplicate DAG IDs raise AirflowDagDuplicatedIdException, which can halt the upgrade. | Run airflow dags reserialize in a rehearsal and count errors, including duplicate IDs, across every DAG folder. |
| Custom plugins and direct database access in tasks | Airflow 3 removes direct database session and model access from task code, so tasks and plugins that query the metadata database need rewriting. | Search DAG and plugin code for direct database imports and sessions, and confirm each one has a REST API or Task SDK replacement. |
| Database engine and version | Airflow 3.3.2 supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, while MariaDB and SQLite are not supported for production metadata. | Confirm the engine and major version are on the supported list, and plan any engine upgrade as a separate step before the Airflow change. |
| High availability and replication setup | Replicas and synchronous commits add steps to the backup and cutover, and PostgreSQL does not detect primary failure or fail over on its own. | Document whether the primary has standbys, how writes are paused during the window and which tool performs any failover. |
| Number of environments to rehearse | Each environment has its own data volume, DAG set and integrations, so a single rehearsal rarely covers production behavior. | List development, staging and production, and confirm each one is restored from a recent backup and migrated before production. |
Questions teams ask before upgrading the Airflow metadata database
What drives the cost and effort of an Airflow metadata database upgrade?
The biggest driver is the size and age of the metadata database, because the Airflow 3 upgrade guide calls airflow db migrate the biggest part of an Airflow upgrade and notes that schema migrations can be slow on large databases. Effort also rises with the number of DAGs that need reserialization, the number of open errors such as AirflowDagDuplicatedIdException, and how much of the stack must be rebuilt around the upgrade. Costs split into internal staff time, a partner's fixed-scope work, staging capacity for rehearsals and backup storage. Ask any quote to itemize the rehearsal runs, the backup and restore test, the rollback plan and the post-upgrade checks as separate lines, so you can see which items recur and which are one-time.
How long is the downtime during an Airflow metadata database upgrade?
Downtime depends mainly on how long the schema migration takes on your data, how long the backup takes if you must stop Airflow for a consistent copy, and how much history you trim first. The upgrade guide says to shut Airflow down when hot backups are not possible so the backup stays consistent, and it recommends trimming old data with airflow db clean before migrating. Measure the real window in a staging copy of production data rather than relying on a general estimate. Confirm with your own team how long your largest tables take to clean and migrate, and agree on a maintenance window that includes time for the rollback decision.
What does the upgrade need from our DBA, platform and DAG owner teams?
Three teams carry distinct work. The DBA team owns backup and restore testing, database sizing, connection limits, and any PostgreSQL version change, and should confirm that a consistent backup can be restored. The platform team owns the scheduler, API server, DAG processor and worker deployments, the maintenance window, and the order in which components are stopped and started. DAG owners run airflow dags reserialize, fix errors such as duplicate DAG IDs, and confirm that their DAGs load and run after the change. Agree on who makes the go or no-go call before the window opens.
Can our internal team run the Airflow metadata database upgrade alone?
Yes, if you have people who can operate PostgreSQL or MySQL, deploy Airflow 3 components, and test restores, because the upgrade is a documented sequence of backup, trim, reserialize and migrate steps. The risks come from the less documented parts: a backup that cannot be restored, a long migration on a large table, or DAG code that breaks after reserialization. Teams without recent experience with these steps often add an outside review of the rehearsal plan or hand over the whole upgrade under a fixed-scope written agreement. Decide early which of these you need, and ask any partner to state exactly what they will test and hand over.
Does the Airflow upgrade require a PostgreSQL major version upgrade at the same time?
No. The Airflow 3 prerequisites list PostgreSQL 14 to 18 as supported, so a team already on a supported version can upgrade Airflow without changing PostgreSQL. Check the version you run against the PostgreSQL versioning policy, because PostgreSQL 14 is supported only until November 12, 2026, and versions 13 and earlier are unsupported. If you plan both changes, treat them as separate projects with separate rollback points. PostgreSQL documents pg_dumpall with restore, pg_upgrade and logical replication as major upgrade methods, and logical replication allows a major upgrade with only several seconds of downtime, but it adds moving parts you must test first.
How does rollback work after the metadata database migration?
Rollback starts with the backup taken before the migration, which is why the upgrade guide tells you to take it first and keep it consistent. Once airflow db migrate has changed the schema, you generally cannot run the Airflow 2 code against the new database, so the practical rollback is to stop Airflow, restore the pre-upgrade backup and restart the previous release. Any run, state change or DAG edit made after the migration is lost unless you account for it. Rehearse this path on a copy, record the time it takes, and decide in advance the point after which you commit to Airflow 3. Confirm the exact restore steps for your database version with your DBA team and the vendor documentation.
What your team provides and what you receive
Your team supplies the access and decisions that make a database upgrade safe to run. The work needs named owners on both sides: one person accountable for the metadata database (usually the platform or DBA team) and one for the DAG repositories (usually the data engineering lead), plus an approver for each change window. Engineers need access to the metadata database and to the backup tooling you already use, with rights to take a backup, restore it to a scratch instance and run the airflow db clean and airflow db migrate commands. You also provide the DAG repositories, a staging environment that mirrors your production Airflow version, executors and database engine as closely as you can manage, and a change window long enough to cover the backup, migration and smoke tests. Confirm with your own team which backup method you use for PostgreSQL or MySQL and whether it can run hot, because that answer decides whether Airflow must be stopped during the backup.
The engagement delivers an upgrade runbook written for your environment, a backup and restore record that shows a backup was taken and restored successfully into a non-production instance, and a rollback procedure with its entry and exit conditions. The runbook covers the order of steps: backup, history trimming, DAG reserialization, airflow db migrate, component restarts and post-upgrade checks. The rollback procedure states who decides to roll back, what state the database must be restored to, and how scheduled runs are paused and resumed. All of this runs inside infrastructure you control, including your cloud accounts, data centers or air-gapped networks, so database credentials, backup storage and logs stay under your governance. Ask any implementation partner to name which of these artifacts they will hand over, in what format, and who on your side signs off on each one before work starts.
How DeployOpen scopes, delivers and supports the upgrade
DeployOpen plans the metadata database work as a separate, written part of the Airflow upgrade, with backup, history trimming, reserialization, migration and rollback each scoped and tested before the production window. The work runs inside your own database platform and infrastructure, and the same method applies whether the backend is PostgreSQL or MySQL.
How we scope
- Written scope names the Airflow and database versions, the source and target releases, and the environments covered.
- Scope records which backup method is used and whether Airflow can be stopped for a consistent copy.
- Scope lists the history trimming, reserialization and migration steps, with the owner of each decision and sign-off.
- Scope states the downtime window, rollback criteria and the evidence you receive at the end.
- Engagements are fixed-scope, and ongoing support after the upgrade can be arranged as a retainer.
How we build
- We rehearse the backup, restore and migration on a copy of your metadata database before production.
- We run airflow db clean and airflow dags reserialize on staging and fix duplicate DAG ID errors first.
- We measure migration duration on the copy so the production window is based on your data volume.
- We test the restored database by starting the upgraded scheduler, API server and DAG processor against it.
- We check that the rollback path works, including restoring the backup to a clean database server.
How we hand over
- You receive the runbook with ordered commands, owners, checkpoints and the go or no-go decision points.
- You receive backup and restore records, restore test results and the migration logs from rehearsal and production.
- Your team is walked through the rollback procedure and the conditions that trigger it.
- Configuration for database backups, retention and access is documented in your own repositories.
- Open items, known risks and the database platform upgrades you still need to schedule are listed in writing.
How we support
- Support after cutover covers questions about the migration, post-upgrade database health and cleanup schedules.
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped in writing.
- Before the next Airflow or database upgrade, we can refresh the rehearsal and runbook against current versions.
- Confirm with your database team who owns backups, point-in-time recovery and failover tooling after handover.
- Support boundaries, response expectations and access arrangements are agreed in the written scope.
