Tuning the Apache Airflow scheduler for throughput

Tuning the Apache Airflow scheduler for throughput when tasks wait too long to start

Scheduler tuning covers the settings and capacity decisions that determine how quickly Airflow turns a due run into queued task instances, and a team needs it when runs start late, task instances sit in a scheduled state for longer than expected, or the scheduler falls behind on a growing estate of DAGs. Scheduling delay rarely has one cause. It can come from scheduler capacity, from DAG file parsing that competes with scheduling, from concurrency limits that hold tasks back on purpose, from executor or worker capacity that never picks tasks up, or from a metadata database that answers each scheduling loop too slowly. Tuning starts by identifying which of these applies, because raising parallelism on a database-bound scheduler or adding schedulers to a deployment that is blocked by a pool will not shorten the wait. The first diagnostic step is to separate waiting in the scheduler from waiting for capacity. Task instances that stay in a scheduled state usually point to the scheduler loop itself, the database, or limits such as parallelism, max_active_tasks_per_dag and pool slots. Task instances that reach queued and then stall point toward the executor, the workers or the broker. Compare the scheduler heartbeat, the DAG parse time metrics and the pool slot metrics over the same period before changing any setting. Ask your team to confirm which metrics your Airflow version exposes and which metric names your telemetry pipeline keeps, since metric names and availability depend on the version and the metrics configuration you run.

Once the bottleneck is located, the decisions are concrete. Adding a second scheduler is worthwhile only when the first one is saturated and the database has headroom, because Airflow relies on database row-level locks to let schedulers share work safely. Per-DAG and per-task limits should be reviewed separately from global ones, since a single DAG with a low max_active_runs can delay its own runs while the cluster sits idle. Database load deserves a direct check: slow queries against the metadata database and connection pressure often show up as scheduler lag before anyone looks at the database. Plan a load test or a controlled change in a staging environment, record the before and after values, and agree with your database owner what headroom is acceptable. Ask any partner to show the measurements that justified each change, not only the final configuration.

Scheduler and concurrency settings and when each fits

This table compares the global, per-DAG, per-task and scheduler-level settings that control how many tasks Airflow runs at once and how quickly the scheduler places them. Read each row as a separate lever: the first column names the setting, the middle column says what it limits, and the last column says when it is the right tool.

Setting	What it limits	When it fits
parallelism	The maximum number of task instances running at once across the whole Airflow installation	Set as a ceiling for total executor capacity so one busy environment cannot exhaust workers or the metadata database
max_active_tasks_per_dag	Running task instances for each DAG, applied globally as a default	Use when one large DAG could otherwise monopolize shared capacity and starve other pipelines
max_active_runs_per_dag	Concurrent DAG runs of the same DAG, applied globally as a default	Use when overlapping runs of one pipeline would write to the same tables or files at the same time
max_active_tasks (per DAG)	Running task instances for one specific DAG, overriding the global default	Use for a DAG that calls a fragile API or database and needs a tighter cap than the rest of the environment
max_active_runs (per DAG)	Concurrent runs of one specific DAG, overriding the global default	Use for backfills or catch-up-heavy DAGs where many runs could start together and flood downstream systems
max_active_tis_per_dag (per task)	Concurrent instances of one task across all runs of its DAG	Use for a single expensive or rate-limited task inside an otherwise fast DAG
pool and slots	Shared slots that any task assigned to the pool must acquire before running, across DAGs	Use when unrelated DAGs share one external system, such as a warehouse or API, and the limit must hold across all of them
priority_weight and queue	Order in which queued tasks are chosen, and which worker queue receives them	Use priority_weight to favor urgent pipelines when capacity is scarce, and queue to route work to specific workers
Multiple schedulers	Scheduling throughput and resilience, coordinated through database row-level locks	Consider when scheduling delay grows with DAG count and the database has headroom; confirm the database can handle the added load first

Which limit is slowing your runs: scheduler, parsing, concurrency or database

Tasks sit in queued while the executor or workers appear to have room. The usual cause is a limit that applies before the executor sees the task: global parallelism, a DAG's max_active_tasks, a pool with no open slots, or a task-level max_active_tis_per_dag. Free workers do not help if the scheduler will not hand them more work. Check the pool slot metrics first, since pool.open_slots at zero with tasks waiting points straight at a pool. Then compare the DAG's active task count against its own limit in the UI. The Airflow FAQ lists every concurrency setting and the level where each one applies.

Pool exhaustion: A shared pool can be full even when workers are idle, so check pool.open_slots and pool.queued_slots together.
Per-DAG caps: max_active_tasks and max_active_runs can hold back a DAG that looks healthy.
Global parallelism: The parallelism setting caps running tasks across the whole installation.
Task-level limits: max_active_tis_per_dag restricts one task across runs, which often surprises teams that expected more throughput.
Queue routing: A task sent to a queue with no workers will wait indefinitely, so confirm the queue name matches a running worker.

Confirm which limit is binding before raising any value, because raising the wrong one only moves the queue elsewhere.

Runs are created late, so the gap appears between a DAG's scheduled time and its first task starting. This points at the scheduler loop itself: it is slow to evaluate DAG runs, or it is competing with other work for database time. Scheduler lag can come from too few scheduler instances, a loop that spends long periods on each DAG, or a metadata database that answers slowly. Start with scheduler_heartbeat to confirm the scheduler is alive and recording on time. A healthy heartbeat with a growing start delay suggests the loop is slow rather than stopped. The scheduler documentation explains how the loop evaluates runs and how multiple schedulers share work.

Heartbeat gaps: Missed heartbeats mean the scheduler process is restarting, starved or blocked, which is a different problem from a slow loop.
Single scheduler: One scheduler serializes all decisions, so a second instance can help once the database can handle the extra locking.
Long scheduling cycles: Many DAGs with many active runs lengthen each pass, so compare the number of active runs against the observed delay.
Database latency: Slow commits from the scheduler show up as delay across all DAGs, not just one.
Parsing competition: Heavy DAG parsing on the same host can consume the resources the scheduler needs.

Compare the start delay with the heartbeat and the active run count before adding schedulers.

A large number of small DAGs usually strains the scheduler through volume rather than size. Each DAG adds parsing work, scheduling checks and database rows, so the total cost grows even when each DAG is quick. The symptom is steady slowness across the whole installation, often with DAG files appearing or refreshing late. Here the first check is the parse time for the whole set of files, not any one file. Look at dag_processing.total_parse_time over time and at dag_processing.import_errors to rule out files that fail and retry repeatedly. If parse time rises with file count, the bottleneck is the DAG processor and the number of files it must read.

Parse volume: Total parse time grows with the number of files, so count DAG files before tuning the scheduler.
Repeated imports: Failing files that re-import on every pass add cost without adding runs.
Database row growth: Each DAG adds run and task instance rows, which slows queries that the scheduler repeats.
Scheduling loop overhead: Many DAGs with frequent schedules produce more work per pass than a few large ones.
Consolidation options: Some teams merge related DAGs or generate them from fewer sources, which the parsing guidance covers in more depth.

Watch total parse time and import errors together, since a rising parse time with a flat run count points at parsing rather than scheduling.

A single DAG can dominate the scheduler when it creates a very large number of runs or dynamically mapped tasks. Its tasks fill the shared pools and parallelism, and every other DAG waits behind it. The symptom is uneven delay: most DAGs start on time, but one or two are consistently late, or a single DAG's task count jumps in the UI. The fix is usually a per-DAG or per-task limit so one workload cannot claim all capacity. Check the active task count for that DAG against its max_active_tasks and max_active_runs settings, and check which pool its tasks use. If the flood comes from dynamic mapping, confirm the number of mapped instances the DAG creates in a single run.

Run accumulation: A backlog of queued runs from catchup or a long outage can flood the scheduler once the DAG resumes.
Mapped task expansion: Dynamic mapping can create a large number of task instances from one run.
Unbounded concurrency: Without max_active_tasks, one DAG can take every free slot in a pool.
Shared pool contention: Other DAGs using the same pool see queued slots rise while the flooding DAG runs.
Catchup settings: Confirm whether the DAG backfills missed intervals, since that can create many runs at once.

If one DAG's active task count tracks the delay on other DAGs, cap that DAG before adding scheduler capacity.

Tuning steps for a self-hosted scheduler with checks

This sequence moves a self-hosted Airflow scheduler from an unmeasured setup to a tuned one, with a check at each step so every change is tied to a measured result. Each step names what to confirm before the next one starts.

Confirm a production metadata database
The platform team confirms that the metadata database is PostgreSQL or MySQL and not SQLite, which the production guidance says is for testing only. Check the supported database versions for your Airflow release in the prerequisites before you change anything, because MariaDB is not supported as a metadata database. The step is done when the connection string, engine version and backup status are written down and match a supported combination.
Baseline heartbeat and scheduling delay
Before tuning, the operations team records the scheduler_heartbeat metric and the gap between when tasks become ready and when they are queued, over a representative period of normal load. Also capture dag_processing.total_parse_time and dag_processing.import_errors so parsing problems are not mistaken for scheduling delay. The step is done when you have a dashboard snapshot that shows the current scheduling lag and whether any scheduler stops reporting heartbeats.
Set global concurrency limits
The platform team reviews the global parallelism setting and the default per-DAG limits, then sets them to match what the database and workers can handle at once. Keep the change small and record the old and new values so the effect is traceable. The step is done when the new values are in the configuration, deployed, and the baseline metrics are captured again under similar load to confirm that the scheduler is no longer the limiting factor.
Add per-DAG and per-task limits with pools
DAG authors and platform engineers set max_active_runs and max_active_tasks on heavy DAGs, and use pools for shared resources such as databases or APIs, so that one pipeline does not occupy every slot. Track pool.open_slots, pool.running_slots and pool.queued_slots to see where tasks wait. The step is done when a pool queue no longer starves unrelated DAGs and the slot metrics show the expected usage pattern.
Review database load from scheduling loops
The database administrator checks connection counts, query latency and lock waits while the scheduler runs at its new limits. If the database is saturated, adding more scheduler capacity will not reduce delay, so the team should address the database first (indexes, connection pooling, cleanup, or more capacity). The step is done when database metrics stay within the limits your team has agreed on during peak scheduling, and any bottleneck is recorded.
Add a second scheduler
The platform team starts a second scheduler, which Airflow supports for performance and resiliency using database row-level locks. Confirm that both schedulers point to the same metadata database and run the same Airflow version, then watch for lock contention and duplicate scheduling behavior in the database logs. The step is done when both schedulers report heartbeats and the lock waits remain acceptable under load.
Re-measure and compare against the baseline
The operations team repeats the measurements from the baseline step under comparable load, including scheduler_heartbeat, scheduling lag, parse time and pool slot usage. Compare each figure with the earlier snapshot and decide whether each change is kept, reverted or still under review. The step is done when the results are written up with the configuration in place, so the next tuning change starts from a known state.

When the metadata database is the real bottleneck

Adding schedulers or raising parallelism only helps while the metadata database keeps up. The cards below cover the database-side causes that most often show up as scheduling delay, and what to confirm with the team that owns the database before changing Airflow settings.

Row-level lock contention

Several schedulers coordinate through database row-level locks, so they compete for the same task instance and DAG rows. When throughput flattens after adding a scheduler, lock waits are a likely cause. Check lock wait time and blocked queries on the metadata database during scheduling peaks, and confirm with the database owner whether a second scheduler is adding waits rather than capacity.

Connection limits

Each scheduler, worker, API server and DAG processor opens connections to the metadata database. When the total approaches the server's maximum, new connections fail or queue, and scheduling loops stall without an obvious Airflow error. Compare the sum of configured pool sizes across components with the database connection limit, and ask the database owner how connections are pooled and what headroom is reserved for administration.

Slow queries on large history tables

Task instance, DAG run and log-related tables grow with every run, and scheduling queries that scan them slow down over time. A system that was fast at launch can degrade months later with the same DAG count. Review the slowest queries and their execution plans with the database owner, and confirm that retention and cleanup for these tables are scheduled and running, as covered in the sibling cleanup topic.

Disk and memory pressure

Index bloat, write-ahead logs and temporary files consume disk, and large result sets or many open connections consume memory. When the database starts swapping, checkpoints slow down, or disk fills, scheduling latency rises for every component at once. Track disk growth rate, free space and memory use on the database host, and agree with the database owner on alert thresholds and who responds.

Connection pooling and cleanup

Pooling and metadata cleanup both reduce database load, but they are separate topics with their own setup steps. Here the question is only whether they are in place before you blame the scheduler. Confirm with the database owner that a pooler is in front of the metadata database if connection counts are high, and that cleanup has run recently, then return to the sibling topics for configuration details.

Scheduler metrics to watch before changing any setting

Before you change parallelism, add a scheduler, or raise a per-DAG limit, read the metrics that show where scheduling time is going. Scheduler delay has several possible sources, and each one points to a different fix. The first signal is scheduler_heartbeat, which shows whether each scheduler process is alive and completing its loop. If you run more than one scheduler, check each one separately so that a single stalled instance does not hide behind a healthy average. Next, look at dag_processing.total_parse_time and dag_processing.import_errors. A growing parse time means the DAG processor is spending its cycle reading files, so the scheduler may be working from stale DAG definitions even when it looks busy. Import errors mean some files are not producing DAGs at all, which can make a pipeline appear to run late when it is actually missing. The Airflow metrics documentation lists these names as examples, and you should confirm the exact names, tags and units that your chosen emission route produces, whether that is StatsD or OpenTelemetry, before you build alerts on them.

The pool and task failure metrics tell you whether tasks are waiting for capacity or failing once they run. pool.open_slots, pool.running_slots and pool.queued_slots should be read together for each pool. If pool.queued_slots rises while pool.open_slots is zero, the limit is deliberate and working, and raising global settings will not help. If the queue grows while open slots remain, the scheduler is not handing work out fast enough, and the database or scheduling loop becomes the suspect. The ti_failures metric, tagged by dag_id and task_id, shows which tasks fail, so you can tell a slow scheduler from a noisy DAG that is retrying heavily and consuming slots. Plan a review step where someone reads these signals side by side for at least one representative busy period and records the baseline. Changing a setting without that baseline makes it impossible to tell later whether the change helped. Ask your team to confirm that the tags are emitted with the expected cardinality, because high-cardinality tags can add monitoring cost of their own.

Operational risks of tuning the scheduler

Scheduler tuning changes how hard Airflow leans on the metadata database, workers and downstream systems. The risks below share one root cause: a change that makes the scheduler faster on paper can overload a component or hide the symptom you were trying to fix, so each change needs a baseline, a record and a staged rollout.

Raising parallelism past database and worker capacity

Raising parallelism lets the scheduler queue more task instances than the metadata database and workers can serve. Each extra queued task adds scheduling-loop queries, heartbeats and state writes, so throughput can fall while the scheduler looks busy. Before raising the value, check database connection headroom, CPU and lock wait times, and confirm worker slots can actually run the added tasks. Make the change one step at a time and watch for the point where extra concurrency stops increasing completed tasks.

Duplicate scheduling after adding schedulers

Running more than one scheduler is supported because Airflow uses database row-level locks to coordinate them, but that safety depends on the database and configuration behaving as the documentation describes. Adding a scheduler without verifying locking under load can produce duplicate task instances or uneven scheduling. Test with a representative DAG load, confirm no task instance is run twice, and check the database engine and version you actually run against the supported list before you add replicas.

Tuning in production without a baseline

Without a recorded baseline, you cannot tell whether a change helped, hurt or did nothing. Capture scheduling delay, scheduler heartbeat, parse time, database load and task queue time for at least one normal business cycle before the first change, then repeat the same measurements after each step. Include peak periods, since the worst scheduling delay often appears at a batch window rather than in average load. Keep the before and after figures with the change record.

Masking failures behind queue symptoms

Raising concurrency limits or pool sizes can make a queue disappear from dashboards while the underlying fault remains. A task stuck in queued may be a symptom of a full pool, an exhausted executor or a broken worker, and removing the queue hides which one it was. Track pool.queued_slots and ti_failures alongside the scheduler metrics, and record which queue symptom each tuning change was meant to address so the cause is still visible afterward.

Change records and staged rollouts

Each scheduler setting change should have a record that names the owner, the reason, the baseline figures, the expected effect, the rollback value and the environment where it was tested first. Apply changes in a staging environment with comparable DAG volume, then roll them to production one setting at a time. Review the record after the change window closes so the team can decide whether to keep, adjust or revert the setting.

Per-DAG limits that starve other teams

Raising global limits without adjusting max_active_runs and max_active_tasks on individual DAGs can let one large pipeline consume the scheduler's attention and shared slots, delaying others. Review which DAGs own which pools and queues, and set per-DAG limits so that one team's backfill cannot push another team's production runs out of the queue. Confirm the ownership map is current before any shared limit changes.

Questions teams ask before tuning or hiring help

How much effort does tuning the Airflow scheduler usually take?

Effort depends mostly on how many DAGs, tasks and parsed files the deployment carries, and on whether the bottleneck is the scheduler itself, the metadata database or the workers. A small deployment with a few dozen DAGs may need little more than setting sensible limits and watching a few metrics. A large one usually needs measurement across several weeks of normal and peak load, changes made one at a time, and a review of how scheduling loops hit the database. Ask any partner to break the work into discovery, measurement, changes and verification, and to state what each phase produces.

Can our own team tune Airflow scheduling without outside help?

Yes, if the team can read scheduler and pool metrics, has access to the metadata database and can test changes outside production. The work is mostly systematic: find where tasks wait, separate scheduling delay from queueing and worker capacity, then change one setting at a time. Teams without database or Airflow operations experience often find it slower, because the cause of delay can sit in query load, DAG parsing or executor capacity rather than in a scheduler setting. Outside help is most useful for a one-time review or a difficult bottleneck.

What access and environments does a tuning project need from us?

Plan for read access to Airflow metrics, the scheduler and DAG processor logs, and the Airflow configuration in each environment. You will also need a way to view the metadata database load, such as database monitoring for your PostgreSQL or MySQL instance, and a staging environment that resembles production in DAG count, schedule shape and worker capacity. Without staging, changes must be made live, which makes it hard to tell cause from effect. Confirm change windows, rollback steps and who approves configuration changes before the work starts.

How long does scheduler tuning take in general terms?

Duration is driven by how long it takes to collect a representative picture of load and by how many changes must be tested. Measurement is often the longest phase, because peaks such as month-end or batch windows must appear in the data before conclusions are safe. Deployments that already have good metrics and a staging copy can move faster. Ask for the phases, the evidence each one produces and the dependencies on your team, such as access approvals and change windows. Do not accept a duration that is not tied to a written scope.

Does adding more schedulers always reduce scheduling delay?

No. Airflow supports running more than one scheduler at the same time for performance and resiliency, using database row-level locks. Extra schedulers help when a single scheduler cannot keep up, but every scheduler adds queries against the same metadata database. If the database is already saturated, a second scheduler can increase lock contention and make delay worse. Confirm the database capacity, connection limits and scheduler heartbeat behavior before adding replicas, and measure the result.

Which costs drive the price of a scheduler tuning engagement?

The main cost components are the time spent on measurement and analysis, the design of configuration changes, the staging work needed to test them, and the verification and handover. Costs rise with the number of environments, the number of DAGs and teams sharing the cluster, the complexity of the executor setup and the amount of database work involved. Costs fall when metrics are already collected, staging exists and ownership of DAG code is clear. Ask any quote to list these phases separately, to say which ones are one-time and which need a recurring retainer for ongoing review.

What should we check before changing parallelism or per-DAG limits?

Start by confirming where tasks actually wait. Airflow concurrency is set globally with parallelism, max_active_tasks_per_dag and max_active_runs_per_dag, per DAG with max_active_tasks and max_active_runs, and per task with max_active_tis_per_dag, pool, priority_weight and queue. Raising a global limit without checking pools, worker slots and database headroom can move the bottleneck rather than remove it. Record the baseline metrics, change one limit, and compare queue time and database load before moving to the next.

How DeployOpen scopes and supports scheduler tuning

Scheduler tuning at DeployOpen starts with a written, fixed scope agreed before any change is made to your Airflow environment. The scope names the Airflow version in use, the metadata database engine, the executor, the number of DAG files and the scheduling delay you want to reduce. It also states which settings may be changed, which are out of bounds, and what the deliverables are, such as a tuning report, a set of configuration changes and a dashboard or alert set. Because most engagements are fixed-scope, the quote follows from this document, so the questions you answer here shape cost more than any later change request.

Before tuning, DeployOpen measures a baseline inside your own infrastructure, on your clusters, databases and monitoring stack. The baseline records scheduler heartbeat behavior, DAG parsing time, pool slot usage, task wait time and database load during normal and peak scheduling. Your team stays in control of access: DeployOpen works with the metrics and logs you expose, and no production data leaves your environment. Each configuration change is recorded with its reason, the value before and after, and the metric it was meant to move, so that you can roll it back and audit it later. Verification repeats the same measurements after each change rather than relying on a single before and after snapshot.

After the agreed work closes, ongoing support is available through a retainer. Retainers typically cover re-measurement after Airflow upgrades, review of new DAG batches that change scheduling load, and reviews of database growth and cleanup schedules. Ask for the retainer terms in writing, including response expectations and what is excluded. Your own team should confirm that its on-call staff can read the metrics and understand the change log before the engagement ends, because scheduler behavior shifts as your DAG count grows.

Tuning the Apache Airflow scheduler for throughput