Apache Airflow backup and disaster recovery

The Airflow 3 upgrade guidance tells you to back up the metadata database first, and to shut Airflow down if a hot backup cannot be kept consistent. The metadata database holds DAG run state, task instances, connections, variables and the audit trail, so a copy taken while schedulers, workers and the API server are writing can capture a state that never existed. Decide before the first backup which approach you will use, and write it into the runbook so the on-call engineer does not improvise it during an incident.

A hot backup keeps Airflow running but needs a database that can produce a point-in-time-consistent copy while writes continue. A planned stop window is simpler and easier to verify: you drain the schedulers, stop the workers and API server, take the backup and restart. The cost is downtime for every scheduled run in that window, so schedule it where missed runs can be caught up. PostgreSQL and MySQL are the supported production backends in the prerequisites and deployment guidance, and they provide the native backup, replication and point-in-time recovery options that make hot backups practical. SQLite is for testing only and should not be the source of a production backup plan.

Run airflow db clean before the backup when history has grown large. It removes old rows according to your retention settings, which shrinks what you back up and speeds up the schema migrations the upgrade guide warns can be slow on large databases. Confirm the retention window with your audit and compliance owners first, because cleaned rows cannot be restored from the backup you take afterward. Ask your database team which backup tool, retention schedule and point-in-time recovery window your platform supports, and test a restore into a separate instance before you rely on the procedure.

The main risk after a recovery is that the restored metadata database describes work that already happened, and a scheduler that wakes up against it decides that work is still pending. Tasks that finished after the backup point may be scheduled again, and intervals that were recorded as running may be skipped or re-queued. The first control is to keep the recovered environment paused until someone has checked its state: pause the DAGs, keep the scheduler and workers stopped, and only start them once the checks below pass. Airflow supports running several schedulers against one database using row-level locks, which is why a stale scheduler left over from the failed site must never be pointed at the restored database. Confirm that every scheduler, DAG processor and API server in the recovered environment uses the restored connection string, and that nothing in the old region still reaches it.

Before unpausing, review the task instances that were queued or running at the backup point and compare them with what the external systems actually did. Look at the dag_run and task_instance state in the restored database, then check the target systems (warehouses, APIs, file drops, message topics) for the same work. Where a task already wrote output, decide whether to mark it successful, clear it for a rerun, or leave it alone. Treat this as a practice to validate in your own environment rather than a guarantee: the exact states depend on your executor, retry settings and how far the backup lagged behind the failure. Record each decision so the audit trail shows why a run was rerun or skipped.

The most durable protection is idempotent task design, which Airflow cannot provide on its own. Tasks that write with upserts keyed on a business identifier, overwrite a partition for a given logical date, or check for existing output before writing will tolerate a replay without corrupting data. Confirm with each pipeline owner which tasks are safe to rerun and which need a manual step, and document that list alongside the recovery runbook. Include a catch-up review for schedules that could backfill a large window once the scheduler resumes, and confirm how your DAG catchup and max_active_runs settings limit that burst. Validate these controls in a rehearsal, as described in the testing section, before you rely on them in a real failover.

A backup that has never been restored is an assumption, not a recovery plan. Run a restore drill at least once before the platform goes into production, then repeat it on a schedule. Restore the metadata database backup into an isolated environment that cannot reach production task workers, message brokers or source systems. Redeploy the same Airflow version you backed up, using the same Helm chart or image tag and the same configuration files. Matching the version matters because the metadata schema and the Airflow 3 components must agree with the restored data. Start with the database restore and the schema check, then bring up the API server, scheduler and DAG processor as separate components, since Airflow 3 requires the DAG processor to run on its own.

Once the stack is up, confirm three things in order. First, confirm that DAGs import cleanly from the restored DAG source, whether that is the image, the git-sync repository or the externally populated volume. An import error at this stage usually means the DAG source revision does not match the database state, so record which revision each environment used. Second, confirm that secrets resolve in the documented order: the alternative secrets backend first, then environment variables, then the metastore. This order is not configurable, so test a connection that depends on each layer and confirm which one answers. Also check any separate worker backend configured under [workers] secrets_backend. Third, confirm that the scheduler does not fire duplicate runs. Compare the restored dag_run records against the schedule and the logical dates of the last completed runs before enabling any DAG, and keep the drill environment's schedulers paused so it cannot write to production systems.

Record each step as evidence: the backup identifier, the Airflow version, the DAG source revision, the time of each check and the person who signed off. Store these records with your change and recovery documentation so auditors and internal reviewers can see that the plan works. Repeat the drill after every Airflow upgrade, every Helm chart change, every secrets backend change and every significant airflow db clean run, because each of these can alter what a restore needs. Ask your partner or internal team to state in writing how long the drill took and which steps were manual, and treat any step that needed undocumented knowledge as a gap to close before the next drill.