Apache Airflow pools, priority weights and concurrency limits

A pool is a named count of shared slots that tasks draw from while they run. Each task assigned to a pool takes one or more slots (the number is set per task), so every task using that pool competes for the same budget regardless of which DAG it belongs to. This is what makes pools useful for protecting a database or API: the limit applies across the whole deployment, not just inside one pipeline. The three pool metrics show the state of that budget. pool.open_slots is the capacity still free, pool.running_slots counts slots held by running tasks, and pool.queued_slots counts slots wanted by tasks that are waiting for the pool. Together they tell you whether the pool is the thing holding tasks back.

The key distinction to make when reading these numbers is between a full pool and a queued task. When pool.open_slots falls to zero and pool.queued_slots climbs, tasks are waiting on the pool, and the fix is a design decision: raise the pool size if the downstream system can take more load, or accept the wait. When the pool still has open slots but tasks sit in the queued state, the wait is somewhere else, usually executor or worker capacity, global parallelism limits, or a per-DAG limit. Check the scheduler and executor metrics and the task instance reason before changing any pool. Changing a pool when the real limit is worker capacity changes nothing and hides the diagnosis.

Confirm the exact metric names, tags and available units in the metrics configuration for your Airflow version and your metrics backend (StatsD or OpenTelemetry), because naming can differ between emitters. Before you alert on pool slots, agree with the owners of the protected system what a sustained full pool means for them, and review the metrics page linked from the sources for the current list. Pool metrics are most useful when they are plotted next to the task failure and duration metrics for the same tasks, so a team can see whether waiting tasks are slowing a pipeline or simply queuing harmlessly.

Set both the size of a mapped fan-out and the number of copies allowed to run at once. A task that calls expand() creates one mapped task instance per item in its input, so a list of 500 customer IDs becomes 500 task instances. Airflow caps the size of a single expansion with [core] max_map_length, which defaults to 1,024 tasks. That cap protects the scheduler and metadata database from a runaway input list, but it does not limit how hard the work hits a downstream system. A fan-out that stays under the default cap can still open hundreds of simultaneous connections to a shared API.

The control for concurrency is max_active_tis_per_dag on the mapped task. It limits how many mapped copies run at once across all active runs of the DAG, not only within one run. This matters when several runs overlap, for example a catch-up backfill running alongside the current schedule, because each run would otherwise get its own allowance. Pools also apply to mapped tasks, so a pool sized for the API can work alongside the per-task cap. Confirm in your Airflow 3.3.2 environment how the two interact for your DAG layout before relying on either alone.

For any fan-out that calls a shared database or third-party API, write down three numbers before the DAG goes to production: the largest expected input length, the length you will accept before failing the run, and the number of simultaneous calls the provider tolerates. Set max_map_length or validate the input list upstream so an unexpected large list fails early instead of being truncated silently. Then set max_active_tis_per_dag to a value below the provider limit, leaving room for other DAGs. Test the setting with a staging list at production size and watch the pool slot metrics while it runs, since the per-task cap alone does not show how much of a shared pool the fan-out is consuming.

Before you change any concurrency setting, read the official Airflow references side by side with your own deployment. The FAQ entry on concurrency is the most direct starting point, because it lists where each limit lives: global settings such as parallelism, max_active_tasks_per_dag and max_active_runs_per_dag, DAG-level settings such as max_active_tasks and max_active_runs, and task-level settings such as max_active_tis_per_dag, pool, priority_weight and queue. Read it with a specific question in mind, such as which database or API each task calls, so that every limit you set maps to a real downstream capacity. The Airflow FAQ on concurrency is the natural next read after this page.

Once limits are in place, you need to see whether they hold. The metrics configuration reference documents the pool gauges pool.open_slots, pool.running_slots and pool.queued_slots, which show whether tasks are waiting on a pool rather than on the executor. Confirm with your platform team which metrics backend your deployment emits to, because the metric names and tags you will query depend on whether you use StatsD or OpenTelemetry. Also confirm that the pool metrics are not blocked by your metrics allow or block configuration, since a filtered metric looks like a healthy pool.

Dynamically mapped tasks need their own check. The dynamic task mapping guide explains how expand() and partial() create many task instances from one definition, and it sets the default cap of 1,024 tasks per expansion through [core] max_map_length. It also describes how max_active_tis_per_dag limits how many mapped copies run at once across all active runs. Before you raise either value, estimate how many mapped instances a single run can create and confirm how the cap interacts with your database connection budget and pool sizes, because a large expansion can exhaust a shared resource in seconds.