Airflow backfills in Airflow 3: setup and planning

When an Apache Airflow backfill is the right way to reprocess history

A backfill is a scheduler-managed rerun of past intervals of a DAG, so you reprocess history without hand-triggering each run. You need one when a DAG logic fix changes the output of past windows, when a missed window has to be recovered, or when a new column needs historical data. Those are the situations where dozens or hundreds of logical dates must run, in order, with the same code and the same guarantees as normal scheduling. The Apache-2.0 project has no paid edition, so the license costs nothing. What you pay for is the planning of which dates to rerun, the capacity to run them, and the operating effort to watch the results.

A simpler action is often enough. If only one or two runs are wrong, clearing those task instances or triggering a single run reprocesses them without a backfill. Use a clear when the DAG's current schedule already covers the window and you only need a retry. Use a backfill when the range is large, when you must control how many runs execute at once, or when you need the scheduler to skip work that is already in progress. Before you start, confirm which reprocess behavior fits your case (missing, failed or all runs) and whether the DAG's current version is the one you want applied to history, since that determines whether old outputs are overwritten.

Plan the backfill before you start it. Estimate the number of intervals, check that downstream systems can absorb the extra load, and decide whether the work should run backwards so the most recent intervals finish first. Ask your platform team to confirm the capacity your executor and database can carry during the run, and ask your data owners to confirm which downstream tables or APIs will receive rewritten data. Record who approved the date range and the reason for the rerun, because that record becomes part of your change history. The Apache Airflow backfill documentation describes the options you will be choosing between.

Starting a backfill from the UI, CLI or REST API

The UI suits operators and data owners who need to reprocess a known window of history for one DAG and watch the result in the same console they use for daily runs. The form asks for the DAG, the date range and the reprocess behavior, so the operator can review the parameters before anything is scheduled. It is the easiest path to read, but the action is only as traceable as the user account that performed it.

Form-driven input: Operators pick a DAG and date range, then choose how existing runs are treated before submitting.
Visible progress: Created runs appear in the standard grid and graph views, which helps operators confirm what is being reprocessed.
Individual accountability: Actions are tied to a named user, so confirm that shared or generic accounts are not used for backfills.
Role limits: Confirm which Airflow 3.3.2 roles can see the DAG and trigger backfills, since access is controlled through the configured auth manager.
Change records: Ask whether your process needs a ticket reference recorded alongside the UI action, because the UI does not create one for you.

Before granting UI access, confirm the auth manager in your deployment and test that a non-administrator can start and cancel only the backfills you intend.

The CLI suits engineers who repeat a backfill pattern, test it in staging, or wrap it in a runbook script. The core command is airflow backfill create with --dag-id, --from-date and --to-date, and the same command can be checked and reused across environments. Because the command is text, it can live in version control and be reviewed like any other change, which is valuable when a reprocessing run touches production tables.

Reproducible commands: Store the exact airflow backfill create invocation with its date range in a runbook or repository so the run can be repeated or explained later.
Scripted guardrails: Wrap the command with checks for the DAG state and date window before execution, rather than relying on memory.
Scheduler dependency: Backfills are managed by the scheduler, so confirm the scheduler is healthy and reachable from the host where the command runs.
Credentials on hosts: Confirm which account the CLI runs under and where it is stored, because a shell session on a worker or admin host can start backfills for any DAG it can reach.
Concurrency settings: Set backfill-level concurrency deliberately in the command or runbook so it does not reuse the DAG's own limit by accident.

Before relying on scripts, run the exact command against a staging environment and confirm the resulting runs match the date window you intended.

Reprocess behavior and concurrency options compared

This table compares the three reprocess behaviors and the concurrency settings that control a backfill. Read each row by asking what state your existing runs are in and how much load the rerun will put on downstream systems. The main takeaway is that the reprocess choice decides which runs change, while the backfill max_active_runs value decides how hard those runs hit your systems.

Option	When it fits	Main risk
Reprocess: none	Filling gaps where no run exists yet for a date, such as after a schedule was added or a period was never run	Existing failed or incorrect runs stay as they are, so a bad earlier result is not corrected
Reprocess: failed	Repairing dates whose runs failed, for example after a bug fix or an outage in a source system	Successful runs produced with old code or bad input are not rerun, so they may still be wrong
Reprocess: completed	Full recomputation after a logic change or corrected source data across a whole date range	Every date in range runs again, which multiplies load on sources and sinks and overwrites existing outputs
Backfill max_active_runs	Capping how many backfilled dates run at once, set separately from the DAG's own value	A high value lets the backfill compete with live scheduled runs and overload downstream systems
DAG max_active_runs	Limiting scheduled runs of the DAG during normal operation, which the backfill value does not replace	Lowering it alone does not limit a large backfill, so confirm the backfill-level value before starting
Overlap protection	Backfill never creates a run for a date that is already running or queued, so duplicate work is avoided	Runs queued from earlier activity may hold up the backfill, so check the queue before starting
Run direction (--run-backwards)	Processing the most recent intervals first when current data matters most	If the window is cut short, the oldest gaps may remain open, so confirm the full range is covered

Protecting downstream systems from backfill load

A backfill replays many historical intervals at once, so every task that reads from a source database or writes to a warehouse sees a burst of load it never saw in normal scheduling. The decisions below set limits on how much work the backfill can push downstream, when it runs, and who is told before it starts.

Lower the backfill max_active_runs

A backfill has its own max_active_runs, separate from the DAG's own setting, so you can cap how many historical runs execute at once without changing the schedule that serves production. Start with a low value for the first pass and raise it only after you have watched the downstream effect. Check that the value you set is the one the backfill is actually using, since the DAG-level setting does not govern it.

Schedule runs outside business hours

Historical reprocessing competes with the daytime workloads your analysts and customer-facing systems depend on. Plan the backfill window for overnight or weekend periods when source systems have spare capacity, and confirm that the window is long enough to finish the planned intervals. Pause or stop the backfill if it runs into a window with heavier production traffic.

Coordinate with source database owners

The teams that own operational databases and APIs usually know their peak periods, rate limits and maintenance windows better than the pipeline team does. Share the intervals, the expected number of queries and the planned concurrency before the backfill starts, and agree on a stop condition they can trigger. Record the agreement with the backfill plan so the change is traceable.

Limit task-level parallelism

Concurrency at the run level is not the only control. Multiple tasks inside each run can hit the same system at once, so the total load is the product of both settings. Review the task concurrency limits and pool assignments for the tasks that touch shared resources, and confirm that they hold during the backfill as well as during normal runs.

Stage the backfill targets first

Writing reprocessed output directly into production tables makes it hard to validate results and to undo a bad run. Point a test copy of the backfill at a staging schema or bucket, compare row counts and key totals against the current output, and promote the results only after the comparison passes. This also shows whether the load pattern on sources is what you expected before you run it at full scale.

Check warehouse quotas and limits

Analytical warehouses often enforce concurrency slots, query quotas or cost budgets that a large backfill can exhaust, delaying or failing production queries. Ask the warehouse owner which limits apply to the service account Airflow uses, and whether a separate workload group or reservation can isolate backfill queries. Confirm these limits in your own environment, since they differ by platform and by contract.

How running backwards changes what finishes first

Passing --run-backwards to airflow backfill create changes the order in which intervals are processed. Instead of starting with the oldest date in the range, the backfill starts with the most recent interval and works toward the start date. Use this when the newest data matters most, for example when a corrected transformation must reach current reports before older history is rebuilt. Analysts can then see the corrected figures for the current period while the historical reprocessing continues in the background. The order changes only which runs happen first. The backfill still applies the same rules about runs that are already queued or running for the same date, and it still honors the backfill-level max_active_runs setting you choose.

The trade-off is that a recent interval may finish before the older intervals it depends on. If a DAG reads its own previous output, a rolling total, a slowly changing dimension or a model trained on prior periods, the newer run can complete using data that has not been rebuilt yet. The run shows as successful, but its results may be wrong until the older intervals catch up. Before you choose this order, check every DAG in the backfill for dependencies on earlier logical dates, including depends_on_past settings, sensors that wait on prior runs and any tables that are overwritten rather than appended. Confirm these assumptions in a staging environment with a short, known date range, and compare the output against a forward run. Where a downstream consumer must not see partial results, plan a gate that holds publication until the full range is complete.

Planning and running a backfill safely

This sequence takes a backfill from a proposed date range to a verified reprocessing run, so each decision about idempotency, active runs, reprocess behavior, concurrency and downstream load is checked before the full range starts. The result is a documented run that can be paused, explained and repeated if needed.

Confirm the DAG and target tables are safe to rerun
The pipeline owner and the DAG author confirm that each task produces the same output when run again for the same interval, and that the target tables or partitions are keyed by data interval so a rerun replaces one interval rather than appending duplicates. Write down which tables each interval touches and who owns them. Done means the owner has signed off on idempotency for every task in the backfill path, and the list of affected tables and partitions is recorded with the change ticket. If any task writes to a shared table without interval keys, stop here and redesign that write before continuing.
Check for active or queued runs in the date range
Before creating anything, the operator lists runs for the DAG in the target date range in the UI or through the CLI or REST API and notes any run that is still running or queued. Airflow does not create a backfill run for a date where a run is still running or queued, so these intervals will be skipped or will wait, and the team needs to know which ones. Done means the list is captured, the owners of any in-flight runs have agreed to the window, and the scheduler is not about to start conflicting work for those dates.
Choose the reprocess behavior for the range
The operator selects the reprocess behavior for the backfill: none, failed or completed. Choose none when only missing intervals should run, failed when existing failed runs should be rerun, and completed when every interval in the range must be rerun, including successful ones. Record the choice with the reason, because completed reprocessing touches intervals that already delivered data and is the option most likely to affect downstream consumers. Done means the behavior is written in the change ticket and matches the list of runs from the previous step.
Set backfill concurrency separately from the DAG
The operator sets the backfill-level max_active_runs, which is independent of the DAG's own setting, so the historical load can be limited without changing how the scheduled DAG behaves. Start with a low value and confirm the value shown on the created backfill matches what was requested. Check the connection limits, API rate limits and warehouse capacity that the DAG's tasks use, and confirm the number of runs that can execute at once stays within them. Done means the value is recorded, the limit has been compared with the capacity owners' figures, and the live scheduled DAG has not been changed.
Rehearse a short date range in staging
The team runs the same backfill command, with the same reprocess behavior and concurrency, against a staging environment and a short date range chosen to cover the variety of intervals, including any month or quarter boundaries and any intervals with known failures. The DAG version and the staging data should match production as closely as possible. Compare row counts, partition contents and output checksums with the expected values, and review task logs for retries and errors. Done means the staging output matches the expected results and the team agrees the full range can be run on the same settings. If the outputs differ, fix the DAG or the write logic and repeat this step.
Start the production backfill with a monitoring plan
The operator creates the production backfill with the agreed date range, reprocess behavior and concurrency, using the UI, the CLI or the REST API, and records the backfill identifier. Before starting, the team confirms who watches the scheduler, the task queue, the database and the downstream systems, and agrees the conditions under which the backfill is paused or cancelled. Done means the backfill is created with the recorded settings, the monitoring owners have confirmed their dashboards, and the first intervals are running as expected.
Watch scheduler and downstream load through the range
While the backfill runs, the operator tracks scheduler health, queued task counts, database load and the load each downstream system sees, such as warehouse queries, API calls and file deliveries. Confirm that the scheduler keeps processing the live DAG on time and that the reprocessing does not push downstream systems past their agreed limits. If either is at risk, reduce the backfill concurrency or pause the backfill rather than let the load grow. Done means every interval in the range has reached a final state, the downstream owners confirm their systems are stable, and the outputs have been checked against the validation criteria from staging.
Record the outcome and close the change
The operator records the backfill identifier, the date range, the reprocess behavior, the concurrency value, any intervals that failed or were skipped, and the validation results in the change record. Failed intervals are rerun or handled under the same controls, and the downstream owners are told which dates were reprocessed so their reports can be checked. Done means the change record shows every interval as complete or explained, the downstream teams have acknowledged the reprocessed dates, and the evidence is stored with the team's operational records.

Access control and audit for backfills

A backfill reprocesses history, so it can write data and load downstream systems at scale. Access control and audit decide who can start one, what credentials they use, and how each run is traced back to a request and a reason.

Who may start a backfill

Backfills can be started from the UI, the CLI, or the REST API, and each entry point is governed by the auth manager your deployment uses. Airflow 3 defaults to the Simple auth manager, and only one auth manager is active at a time, set by the auth_manager option. Confirm which manager runs in your environment, then map the roles that allow DAG run creation to the teams that own each pipeline. Test that a non-privileged user cannot start a backfill from any entry point.

Scoping API credentials

Automation that calls the REST API to start backfills should use a dedicated identity with only the permissions it needs, not a shared administrator account. Check whether the credential can also pause DAGs, change variables, or read connections, because a backfill job rarely needs those rights. Rotate and revoke these credentials through your identity process, and confirm with your auth manager documentation how tokens are issued and how long they remain valid.

Recording the command and reason

Each backfill should be tied to a change or incident record that states the DAG, the date range, the reprocess behavior, and the max_active_runs value used. Capture the exact airflow backfill create command or API payload in that record, along with the requester and approver. This record is what lets an auditor match a burst of historical runs to an authorized decision, so store it outside the Airflow metadata database.

Reviewing reprocess choices before approval

The reprocess setting determines which existing runs are touched, so reviewers should see it before a backfill starts. Choosing a behavior that reruns completed runs overwrites more history than one that fills only missing or failed runs. Ask the requester to state the expected run count and the downstream systems affected. A reviewer who understands these choices can reject a broad request before it reaches production data.

Monitoring the metadata database

Large backfills create many DAG runs and task instances in the metadata database, which can slow the scheduler and the UI. Watch database connection use, lock waits, and table growth while the backfill runs, and keep an agreed stop procedure. Confirm your database sizing and the cleanup plan with your operations team before a multi-year reprocess starts, since the production guide recommends PostgreSQL or MySQL rather than SQLite.

Keeping audit logs and alerts

Airflow records run and task activity, but you should confirm which of those events your auth manager and logging setup export to your central log platform. Route alerts for backfill starts, failed runs, and runs stuck in queued states to the team that approved the request. Test that a backfill start from each entry point appears in your logs with the user identity attached.

Questions teams ask before running an Airflow backfill

How much effort does an Airflow backfill take, and what drives it?

Effort is driven mainly by three things: the date range, the complexity of the DAG, and how much load the downstream systems can absorb. A backfill creates one run per interval in the range, so a long history with a daily schedule means many more runs than a short window with the same DAG. Complex DAGs with many tasks, sensors, or external calls take more planning because each historical run repeats every task. Downstream capacity sets the ceiling on how fast you can safely go, since a backfill can flood warehouses, APIs, or source databases that were sized for normal daily loads. Ask for a written review of the date range, the reprocess choice (none, failed or completed), and the backfill-level max_active_runs setting before anyone starts the job.

Can our own team run an Airflow backfill without a partner?

Yes, if your team already runs Airflow 3 in production and has a staging environment where the same DAGs can be tested against historical data. Backfills use built-in features: you can start one from the UI, the CLI with airflow backfill create, or the REST API, and you choose the reprocess behavior and concurrency yourself. The harder questions are operational. Someone needs to know which downstream systems will feel the load, how to stop a job safely, and how to check that results match what the original runs would have produced. If any of those are unclear, a partner can plan and review the run, but the decision to proceed stays with your team.

What does a backfill need from our team before it starts?

A backfill needs four things from your side. First, access to the Airflow environment with permission to create backfills, plus read access to the DAG code and task logs. Second, test data or a staging copy of the source and target systems, so you can confirm that reprocessed runs write correct results without touching production tables. Third, a list of downstream owners who must be told about the load window. Fourth, change approval: many regulated organizations require a ticket, a reviewer, and a rollback note before a historical rerun touches production data. Your change process should define who can start a backfill, who can stop it, and where the outcome is recorded.

How long does an Airflow backfill take to finish?

The duration depends on the number of intervals in the range and the concurrency you allow, and on the run time of each historical run in your environment. More intervals and longer task runtimes extend the job, while a higher backfill-level max_active_runs can shorten it only as far as the downstream systems allow. Airflow does not run a backfill for a date while a run for that date is still running or queued, so overlapping work is skipped rather than duplicated. The most reliable way to estimate duration is to run a short slice of the range in staging, measure the per-run time, and multiply by the number of intervals. Confirm that estimate with your own data volumes and infrastructure before committing a window.

What does a partner add to an Airflow backfill project?

A partner adds planning and checks that are easy to skip under deadline pressure. Typical work includes reviewing the DAG code for dependencies on the current date, confirming that the reprocess behavior matches the intent of the rerun, choosing a concurrency level that protects downstream systems, and writing the runbook for starting, pausing and verifying the job. Where the backfill is part of a larger change, a partner can also prepare evidence for change records and audit reviews. DeployOpen plans and implements self-hosted open-source platforms inside your own infrastructure, so the work runs in your environment, under your access controls. Scope and deliverables are agreed in writing before work starts.

What should a quote for an Airflow backfill include?

A useful quote states the scope in writing: the DAGs and date range in scope, the reprocess behavior, the target concurrency, and the environments used for testing. It should list the deliverables, such as a reviewed run plan, a staging test record, a runbook, and a verification report comparing reprocessed outputs with expected results. It should name who does what, including which tasks depend on your team's access or approvals, and state the assumptions behind the schedule. Ask whether the quote covers only the backfill or also any DAG changes, and whether follow-up support is a separate retainer. DeployOpen does not publish a price list, so a quote follows from the written scope.

Official Airflow backfill documentation

Before you run a backfill in production, read the official Airflow backfill documentation, since it is the reference for how reprocess behavior, backfill-level concurrency and run direction are defined. Pay particular attention to how the backfill's max_active_runs setting interacts with the DAG's own setting, because the backfill value is separate and applies only to the backfill. Treat the examples in that page as the starting point for your own runbook, and confirm each flag and API field against the exact Airflow version your team runs in its non-production environment before the first real run.

If you are still moving from Airflow 2, read the Airflow 3 upgrade guide alongside the backfill page. It covers the architectural changes that affect how backfills run, including the separate API server and DAG processor components, and the rule that task code no longer uses Airflow database sessions or models directly. Those changes determine which components must be healthy while a backfill is consuming capacity, so check them before you schedule a large reprocess. The guide also lists the Ruff AIR301 and AIR302 checks to run on DAG code.

Because backfill behavior depends on your version, record the exact Airflow release in your change ticket and compare it with the version support table before you plan the work. Confirm with your platform team whether your deployment is on a supported patch release, and ask your DeployOpen contact to check the backfill behavior against that version during scoping. If your environment differs from the documentation, for example a custom executor setup or a restricted network, verify the behavior with a short test backfill on a copy of a representative DAG before you rely on it.

How DeployOpen plans and runs backfills

DeployOpen plans Apache Airflow backfills as part of the wider Airflow 3 deployment in infrastructure you control, so reprocessing rules, concurrency limits and downstream load protection are agreed before anyone starts a historical run. We implement and document the approach, then hand it to your team with the runbook they need to repeat it safely.

How we scope

Written scope lists the DAGs, date ranges and systems that a backfill may touch.
We agree which reprocess behavior applies to missing, failed or all runs for each pipeline.
Scope names the downstream systems whose load limits and maintenance windows must be respected.

How we build

We set backfill-level concurrency separately from each DAG's own max_active_runs setting.
We test backwards runs and confirm the most recent intervals are processed first where that order matters.
We run trial backfills in a non-production environment to check run creation and downstream effects.

How we hand over

You receive a runbook covering the UI, CLI and REST API options and when to use each.
Documentation records the approval steps, owners and checks required before starting a backfill.
Your team walks through a supervised backfill so they can run the next one without us.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer for later backfill work.
Confirm with DeployOpen during scoping which Airflow versions and upgrade paths the retainer covers.

Apache Airflow backfills in Airflow 3