Why Airflow tasks stay in queued and how to find the cause
Tasks that sit in queued, or fail with no log output, usually point to a gap between the scheduler and the workers that run tasks. The first place to look is the path a task takes after the scheduler hands it off: the executor, the worker slots behind it, and any pool or concurrency limit that holds it back. In Airflow, queued means the scheduler has handed the task to the executor, but no worker slot has picked it up yet. The Airflow FAQ notes that a task queued longer than scheduler.task_queued_timeout is marked failed without logs, so a missing log is often a timeout symptom, not a crash inside your code. Some tasks never reach the queue at all because of start_date and schedule, upstream task states, or max_active_runs and concurrency limits, so confirm that a task is actually queued before you debug the workers.
This diagnosis matters most at predictable points: when the DAG count grows, when a team first runs CeleryExecutor with KEDA autoscaling, and after an upgrade to Airflow 3, where the scheduler, DAG processor and API server run as separate components. Each of these changes how capacity is shared, so a setup that worked before can start queuing tasks without any code change. To run the diagnosis, you need read access to the Airflow UI and metadata database, the scheduler and worker logs for the same time window, the executor and pool configuration, and the metrics pipeline if one exists. Ask the platform team to confirm which executor is in use and how many worker replicas were running when the queue built up.
Which cause fits your symptom: scheduling limits, pools, executor, broker or scaling
Why is my DAG not scheduling any run at all?
Usually the DAG is waiting on its own scheduling inputs, not on capacity. Airflow may not create a run because of the start_date and schedule combination, because upstream task states have not met the dependency, or because max_active_runs is already reached. Nothing reaches the queue in these cases, so the symptom is a missing run rather than a task stuck in queued. Check the DAG's run list first: if no run exists for the expected logical date, look at the start_date, the schedule interval and the catchup setting before touching executor settings. Then check whether an existing run is blocking the new one through max_active_runs, which is the setting to inspect when runs pile up behind one long run.
How do I tell a per-DAG limit from a global limit?
A per-DAG limit holds back tasks in one DAG while other DAGs keep running, and a global limit slows everything at once. The Airflow FAQ lists the global settings as parallelism, max_active_tasks_per_dag and max_active_runs_per_dag, and the per-DAG settings as max_active_tasks and max_active_runs. Per-task limits such as max_active_tis_per_dag, pool, priority_weight and queue control individual tasks. If only one DAG is slow and others run normally, check the DAG-level values. If the whole installation stalls once a threshold is crossed, the global parallelism value is the first suspect. Compare the number of running task instances with each configured ceiling before changing anything.
What does pool slot exhaustion look like in the metrics?
Pool exhaustion shows up as tasks queued while the workers are idle. The metrics docs list pool.open_slots, pool.running_slots and pool.queued_slots, so you can watch them per pool. When open_slots falls to zero and queued_slots keeps rising, the pool is the bottleneck. Confirm that the tasks in the pool are not holding slots after they finish, and check whether a pool was sized for an old workload. Pools protect shared databases and APIs, so the fix is usually a deliberate change to the pool size or a change to how many tasks use it, not removing the pool. Ask your team which downstream system the pool protects before raising it.
Can priority_weight or queue make tasks wait longer?
Yes. Priority weight and queue assignment decide which waiting tasks get the next free slot, so a task with a low priority_weight can wait behind others even when capacity exists. A task sent to a Celery queue that no worker listens to will never start, and it will sit queued until scheduler.task_queued_timeout marks it failed without logs. Check the queue name in the task definition against the worker startup command, and check that the workers consuming that queue are running. If the queue is correct and workers are idle, the issue is more likely a pool or a limit. Priority changes affect ordering only, so they do not add capacity.
How do I know whether the Celery broker or workers are the bottleneck?
Look at whether tasks reach the broker and whether workers pick them up. If queued tasks increase while worker slots are free and no worker logs show activity on the expected queue, the problem is in delivery: the broker connection, the queue name or a worker listening on the wrong queue. If workers are busy and their slots are full, the problem is capacity. Check the worker process count, the worker_concurrency value, and whether the broker itself is healthy, including its memory and connection limits. Because a stuck broker can look like an idle scheduler, check worker heartbeats and broker queue depth together before restarting anything.
Why does KEDA scale my workers too slowly or too much?
A mismatch between the KEDA query and worker_concurrency is the usual cause. The Helm chart's default query counts running and queued task instances per Celery queue, divides that number by worker_concurrency and rounds up to get the worker count. If worker_concurrency in the scaling settings differs from the value the workers actually run with, the replica count will be wrong. Scaling to zero also means queued tasks wait for the first worker to start, so cold starts look like stuck tasks. Also confirm that KEDA and a Horizontal Pod Autoscaler are not both managing the same workers, since the KEDA documentation says they must not both be used on them. Review the KEDA guidance for the exact query before tuning.
What should I ask a partner before paying for a stuck-queue investigation?
Ask for the written scope to name which symptoms are covered, which metrics and logs will be reviewed, and which environments are in scope. Ask what access is needed to the Airflow configuration, the metadata database, the broker, the Kubernetes cluster or worker hosts, and the metrics backend. Ask whether the work is a one-time diagnosis with a written report or includes applying fixes and retesting. Costs come from the number of components involved, how many DAGs and queues are affected, and whether your team has the metrics in place already. Your own team can handle many of these checks with the metrics above, so a partner is most useful when the cause spans the broker, the scaler and the database. Quotes follow from that written scope.
Configuration settings that control queue wait and when to change each
This table lists the Airflow settings that decide how long a task waits in queued and what limits hold it there. Read each row for the symptom a low value causes and the load a higher value places on the scheduler, database, workers and downstream systems. Change one setting at a time and confirm the bottleneck with metrics first.
| Setting | What it limits | Symptom when too low | Trade-off when raised |
|---|---|---|---|
| scheduler.task_queued_timeout | How long a task may stay queued before Airflow marks it failed without logs | Tasks fail with no log output during short capacity spikes, which looks like a crash | Tasks may wait longer before failing, so real capacity problems surface more slowly |
| parallelism | Maximum running or queued task instances across the whole installation | Tasks queue even when workers are idle, because the global cap is reached | More concurrent tasks increase scheduler, database and worker load and can overwhelm shared systems |
| max_active_tasks_per_dag | Global cap on concurrently running tasks for each DAG | One busy DAG holds all slots and other DAGs appear stuck in queued | A single DAG can use more capacity, which may starve other pipelines |
| max_active_runs_per_dag | Global cap on concurrently active DAG runs for each DAG | New runs wait behind older runs and show no task progress | Overlapping runs increase load on sources and targets and can complicate data ordering |
| max_active_tasks and max_active_runs (DAG level) | Per-DAG overrides of the global caps | A DAG with a low override queues tasks even when the global pool is free | Higher overrides let one pipeline consume more shared worker and database capacity |
| pool and pool slots | Named slot budget that tasks draw from, such as a database or API limit | Tasks in a pool show queued while pool.open_slots stays at zero | More slots raise parallel calls to the protected system, so confirm its limits first |
| priority_weight | Order in which queued tasks receive free slots | Critical tasks wait behind low-priority backfill or bulk tasks | Higher weights can delay other work during peaks and should be set with a clear policy |
| queue (Celery) | Which worker queue receives the task | Tasks sit in a queue with no workers listening to it | Separate queues isolate workloads but split capacity, so idle queues waste workers |
| max_active_tis_per_dag | Concurrent instances of one task across DAG runs, including mapped tasks | Dynamically mapped tasks trickle through slowly and stay queued | Higher values fan out more work at once and can flood downstream APIs |
| worker_concurrency | Task slots each Celery worker process runs at the same time | Workers appear busy while queued tasks grow, because each worker has few slots | More slots per worker raise memory and CPU use per host and may slow every task on it |
How to diagnose a stuck queued task step by step
This sequence moves from the task itself outward to the scheduler, pools, executor, broker and autoscaler, so you find the layer where tasks are waiting before you change any setting.
Confirm the task state and whether logs exist
Start in the Airflow UI or the REST API and open the task instance that is stuck. Note its state, its try number and whether a log file exists for that try. Tasks that sit in queued for longer than scheduler.task_queued_timeout are marked failed without logs, so a failed task with no log points to a queue wait rather than a code error. A healthy result for a task still waiting is a queued state with a recent queued timestamp that is still moving forward. Record the timestamps, because every later check depends on how long the task has waited.
Check the DAG run and schedule gates
Confirm the task is eligible to run at all. Review the DAG run's start_date and schedule, whether upstream tasks have reached success, and whether max_active_runs or the DAG-level concurrency limits are blocking new runs. If the task is not yet queued, the cause is upstream of the executor. A healthy result is a task whose upstream tasks are complete and whose run is within the active-run limit. If the task is queued but the gates are open, continue to the pool checks.
Read pool open, running and queued slots
Open the pool that the task uses and read the pool.open_slots, pool.running_slots and pool.queued_slots metrics, or the equivalent figures in the Pools view. Compare the pool's total slots with what is running and queued. A healthy pool shows open slots available when the task is waiting, or queued slots that drain as running tasks finish. If open slots are zero and queued slots keep growing, the pool is the bottleneck, and the fix is a pool size or task-level limit change rather than more workers.
Check executor and worker capacity
Confirm the executor has a free place to run the task. For CeleryExecutor, check that workers are online and that their worker_concurrency is not fully occupied. For KubernetesExecutor, check that pods can be scheduled and that the namespace is not hitting quota. For LocalExecutor, check the scheduler host's free capacity. A healthy result is at least one worker or pod free for the task's queue, with no scheduling errors in the logs of the component that starts the task. If workers are all busy, capacity is the constraint.
Inspect the Celery broker queue depth and KEDA scaler activity
If you run Celery workers with KEDA, check the broker queue depth for the task's Celery queue and the KEDA ScaledObject status. The default KEDA query counts running and queued task instances per queue, divides by worker_concurrency and rounds up. Confirm the scaler is active, that it reports the expected count, and that the Deployment is actually scaling. A healthy result is a queue depth that falls as workers come up. If the depth grows while the scaler stays inactive, check the scaler's connection to the metadata database and that HPA is not also managing the same workers.
Check scheduler heartbeat and parsing health
Confirm the scheduler itself is running the loop that moves tasks from scheduled to queued. Check the scheduler_heartbeat metric and the scheduler logs for recent activity. Also check dag_processing.import_errors and dag_processing.total_parse_time, since a parser that is falling behind can delay new task instances. A healthy result is a heartbeat that updates on schedule and no growing import error count. If the heartbeat has stopped, the fix is in the scheduler process or its database connection, and no pool or worker change will help until it recovers.
Metrics that show where tasks wait and what to alert on
Tasks stuck in queued usually leave a trail in a few metric groups: scheduler health, pool slot pressure, DAG parsing, task failures and broker or KEDA queue depth. Read them together to tell whether a task is waiting for a slot, waiting for a worker, or never reaching the queue at all. The metric names below are examples from the Airflow metrics documentation, so confirm the exact names your deployment emits through its StatsD or OpenTelemetry settings before writing alerts.
Scheduler heartbeat for scheduler health
The scheduler_heartbeat metric shows whether a scheduler is still looping. If it stops advancing, tasks never leave scheduled or queued, and nothing else on the page matters until it recovers. Alert on a heartbeat gap rather than a single missed sample, and confirm the metric name and its tags in your StatsD or OpenTelemetry configuration. In multi-scheduler setups, track each scheduler separately so one stalled instance is not hidden by a healthy peer.
Pool slot pressure
Open, running and queued slot counts for each pool show whether tasks wait because a pool is full. A sustained non-zero pool.queued_slots alongside zero pool.open_slots means the pool limit, not the workers, is the constraint. Compare pool usage against the pool size you configured, and check whether one DAG is consuming most of the slots. Confirm the exact emitted names and tags for your version.
DAG import errors and parse time
dag_processing.import_errors and dag_processing.total_parse_time explain why some tasks never reach the queue. A DAG file that fails to import or parses slowly can leave its runs without new task instances, so the queue looks empty while the schedule is late. Alert on a rising import error count, and trend parse time so a slow file is found before it delays scheduling. Check the DAG processor logs for the failing file.
Task failures tagged by DAG and task
The ti_failures metric, tagged by dag_id and task_id, helps identify tasks that were marked failed after waiting too long in the queue. When the queued timeout fires, the task fails without logs, so a failure count with no matching task log is a clue. Group failures by DAG and task to find which workloads are starving, and set your queued timeout setting, scheduler.task_queued_timeout, deliberately so it matches your expected wait. Confirm tag names in your metrics configuration.
Broker and KEDA queue counts
In Celery deployments, the depth of the broker queue and the counts of running and queued task instances per Celery queue show whether workers are the bottleneck. KEDA's default scaling query uses these counts, divided by worker_concurrency and rounded up, so a queue that grows while worker replicas stay flat points to a scaling or capacity problem. Ask which broker metrics your chart exposes and whether KEDA scaling events are logged, since the broker and the KEDA controller each hold part of the picture.
Celery broker and KEDA scaling issues that leave tasks queued
When Celery workers are scaled by KEDA, a task can sit in queued even though the cluster has capacity, because the scaler is counting the wrong thing or no worker is listening on the queue the task was sent to. The Helm chart's default KEDA query counts running and queued task instances per Celery queue, divides that number by worker_concurrency, and rounds up to get the replica count. If a task is routed to a queue name that no worker consumes, the query still sees it as demand, but no pod picks it up, so the backlog grows while workers stay idle. Check that every queue named in DAG code, pools and default_queue has at least one worker started with a matching queue, and confirm the queue names in your own values file match what the workers consume.
Scaling from zero is the second common cause. If the scaler's query returns zero for a queue, KEDA keeps the workers at zero and any task waiting for them remains queued. Symptoms include a broker queue that keeps growing, idle worker deployments with no running pods, and a KEDA object that reports no active trigger. Also verify that the scaler can reach the metadata database with the credentials it was given, because a failed query often looks like a quiet queue. KEDA and the Kubernetes Horizontal Pod Autoscaler must not both scale the same Celery workers. Confirm in your own cluster that only one controller owns the replica count for each worker deployment, since two controllers fighting over the same target produce oscillating replicas and tasks that wait between scale events.
Treat the scaling path as a chain to verify, one link at a time: the scheduler queues the task, the broker holds it, KEDA reads the count, and a worker pulls it. Compare the broker queue depth with the database count of queued task instances, and compare the KEDA replica target with the number of worker pods actually running. A gap at any link tells you where to look. For the sizing rules, cooldown and grace settings and scale-to-zero behavior, use the sibling topic on autoscaling Airflow Celery workers with KEDA and its details in the Helm chart documentation, and confirm the chart version you run before applying any setting from the documentation.
Security and operational risks when you raise limits to clear the queue
Raising parallelism, pool sizes or worker counts is often the fastest way to clear a queue, and each change moves load somewhere else. The cards below cover the places that load lands, the data that leaks through queues and logs, and the change records you need to explain a later incident.
Metadata database overload
Every extra running task adds scheduler and worker writes to the metadata database, so a higher parallelism can move the bottleneck from the queue to the database. Connection counts rise with worker count as well. Before raising limits, check database connections, lock waits and scheduler loop time, and record the baseline values so you can show the effect of the change.
External API and warehouse saturation
Tasks that were queued for a reason often protect a shared system such as a source API, a warehouse or a file transfer endpoint. Clearing the queue by raising max_active_tasks or max_active_tis_per_dag can send bursts of requests that trigger rate limits or lock contention. Use pools to cap these tasks explicitly, and confirm each downstream owner has agreed to the new ceiling.
Exposed broker and metrics endpoints
Scaling Celery workers or adding executors often means exposing a broker such as Redis or RabbitMQ, a Flower UI or a metrics port to more network segments. Keep the broker on a private network, restrict Flower and metrics endpoints to monitoring hosts, and confirm the chosen setup in your own environment before you widen access.
Credentials in queue and log data
Broker messages and task logs can capture connection strings, tokens or parameters when a task is retried or rerouted to a new worker. Check that secrets come from a secrets backend, that rendered templates do not echo credentials into logs, and that broker data is encrypted and access-controlled. Ask your security team to review the log destinations before more workers write to them.
Unrecorded configuration changes
Editing airflow.cfg, Helm values or environment variables directly on live clusters leaves no record of who changed parallelism, why, and when it was reverted. Keep limit and scaling settings in version-controlled values files, link each change to a ticket, and note the before and after values. This record is the evidence an auditor or incident reviewer will ask for.
Duplicate runs after rollback
Scaling down or reverting a limit while tasks are running can leave some tasks marked failed or retried, and a manual clear can trigger the same work twice. Before reverting, confirm which task instances are running, pause affected DAGs if needed, and record every manual clear or retry in the change ticket. Idempotent tasks reduce the impact, but the record shows what ran and why.
How much effort does fixing stuck queued tasks take, and what do we need to provide?
How long does diagnosing stuck queued tasks usually take?
The length depends on how many DAGs and task types you run, which executor you use, and how much observability you already have. A small LocalExecutor installation with working logs and pool metrics is usually quicker to diagnose than a CeleryExecutor deployment with KEDA, several queues, shared pools and a busy metadata database, where each layer must be checked in turn. Before any work starts, ask for a written scope that names the executor, the number of DAGs in scope and the metrics already collected, because those three facts drive the effort more than anything else. The diagnosis itself is scoped in writing and does not begin until both sides agree on it.
Can our own team diagnose queued tasks without a partner?
Yes, for many setups. A single-scheduler LocalExecutor installation can usually be checked by an experienced platform engineer using the scheduler logs, the pool metrics and the scheduler.task_queued_timeout setting described in the Airflow FAQ. Effort rises with CeleryExecutor and KEDA, because the broker, worker counts, queue names, scaling query and database connections all interact. Teams that lack Kubernetes or Celery operations experience often find that the time spent isolating the layer at fault is the expensive part, not the fix itself. A partner can help with that isolation, and your team keeps the knowledge afterward.
What access and information do we need to provide?
Plan to provide read access to the scheduler, worker and broker logs for the period when tasks waited, plus access to the metrics store that holds the pool, scheduler and failure metrics. The work also needs the Helm values or configuration files for the deployment, the DAG code for the affected pipelines, and the concurrency settings at global, DAG and task level. Share capacity figures for the metadata database, including connection limits and recent load, and details of the executor and worker sizes. Confirm with your security team how that access is granted, since logs and DAG code can contain sensitive data.
What does a quote for this work depend on?
A quote follows from a written scope, so the inputs that change the price are the ones to settle first. These include the executor type, whether KEDA autoscaling is in use, the number of Airflow environments and clusters involved, the number of DAGs and queues to review, and whether the work stops at diagnosis and written recommendations or includes implementing the fixes. DeployOpen does not publish a price list. Ask for a quote that separates the diagnosis, the implementation and any follow-up validation, and that lists the access you must provide as an assumption.
Which cost components should we expect to pay for?
There are usually two. The first is the one-time effort to find the cause, change configuration, adjust limits and verify the result under real load. The second is the ongoing effort to watch queue wait metrics, revisit limits as DAGs grow and handle upgrades that change scheduling behavior. Infrastructure costs are separate and recur, including the workers, broker and database capacity that a fix may require you to add. Ask any quote to state which costs are one-time, which recur and which fall on your cloud or data center bill.
Is ongoing tuning of queued tasks available as a retainer?
Yes. Ongoing support, upgrades and continuous testing are available as retainers. A retainer suits teams whose DAG count keeps growing, whose pools and concurrency limits change often, or who run Celery and KEDA workers that need regular review after Airflow upgrades. For a one-off stuck-task problem, a fixed-scope engagement is the usual starting point. If you expect recurring tuning, ask for the retainer terms in writing, including what is covered, how requests are prioritized and how changes are recorded for your change management process.
What should we check before asking for help?
Gather the basics first so the diagnosis starts from facts. Note the Airflow version, the executor, the value of scheduler.task_queued_timeout, and whether the tasks that wait belong to one DAG, one pool or one queue. Record when the waiting started and whether it matches a deployment, a scaling event or a spike in DAG runs. Check the pool.queued_slots and pool.running_slots metrics and the scheduler heartbeat if they are collected. Airflow 2 reached end of life on April 22, 2026, so confirm your version before you plan any change, since a fix on an unsupported release may need an upgrade first.
When to bring in a partner for queued-task incidents
Bring in an outside review when queued-task incidents keep returning after the obvious fixes are done, or when nobody can say with confidence which component owns the wait. A single stuck run during a busy month is usually a capacity or limit question that your own team can close. A pattern of tasks timing out against scheduler.task_queued_timeout, across several DAGs and several weeks, suggests the cause sits in the interaction between the scheduler, the executor, pools and the worker fleet. That interaction is hard to see from inside one team's habits, and an outside reviewer can trace it without the history that has shaped the current configuration.
Ownership is the second trigger. In a Celery or KEDA setup, the scheduler team, the platform team and the data engineers often each own a piece: the broker, the worker images, the Helm values and the DAG code. When a task waits in queue, each group can point to a neighbor's setting. If you cannot name one owner for the executor and broker path, or the person who understands the KEDA query has left, outside help is worth considering before the next incident. Similarly, if you have no metrics pipeline that shows pool slots, queued task instances and worker counts over time, the team is diagnosing from memory. Ask any partner how they would reconstruct the timeline of a past incident, and what they would need you to export or collect first.
Scale-out plans are the third trigger. Adding DAGs, teams, or worker capacity changes where the bottleneck sits, and a fix that works at today's volume can move the constraint to the metadata database or the broker. Before growth, confirm which metrics you will watch, which limits you will set at the global, DAG and task level, and who will test the change under load. Whether you handle that internally or through a fixed-scope review, the written scope should name the components covered, the evidence delivered and the change window agreed in advance.
How DeployOpen works on this
DeployOpen approaches tasks stuck in queued as a diagnosis first: we trace where each task waits, then fix the limit, capacity or broker problem that causes it in an Airflow deployment you run in your own infrastructure. Work is agreed in a written, fixed scope before it starts, and ongoing support is available as a retainer.
How we scope
- Review the executor, worker and broker setup and which queues tasks are routed to
- Collect queued-task samples, scheduler metrics and pool slot counts from a representative period
- Agree in writing which limits, pools, queues and scaling settings are in or out of scope
How we build
- Reproduce the waiting pattern in a staging environment before changing production settings
- Adjust concurrency limits, pool sizes and queue assignments one change at a time with recorded results
- Test Celery broker connectivity and KEDA scaling behavior under a controlled load before handover
How we hand over
- Document each changed setting, its reason and the metric that shows the effect
- Deliver dashboards and alert rules for queued, running and pool slot metrics agreed in scope
- Provide a runbook for task_queued_timeout failures and steps to confirm the cause
How we support
- Offer retainer support for capacity reviews after DAG changes, Airflow upgrades or traffic growth
- Re-test scaling and limit settings when the Helm chart, executor or broker configuration changes
- Confirm support hours, response expectations and escalation paths in the written agreement
