How KEDA autoscaling for Airflow Celery workers works and when to use it
A platform team running CeleryExecutor usually sizes its worker fleet for the busiest hour of the day. Outside that hour, the workers sit idle and still cost money, and during a peak such as a month-end batch the queue grows faster than the fixed workers can drain it. The Helm chart can scale Celery workers with KEDA, so the number of worker pods follows the work waiting in the queue instead of a constant guess. The chart's KEDA integration is the place to start, and the Helm chart autoscaling with KEDA guide sets out the supported configuration.
By default, the scaling query counts the running and queued task instances for each Celery queue. It divides that count by worker_concurrency, the number of task slots each worker runs at once, and rounds the result up. A queue with 25 queued tasks and a worker_concurrency of 16 would therefore ask for two workers (illustrative). This means the scaling signal depends on three settings that must agree: the queue names used by your DAGs, the worker_concurrency value in the worker configuration, and the pool and parallelism limits that decide how many tasks can be queued or running at all. Confirm these values with your own DAG and queue layout before you tune anything.
Fixed capacity fits when the load is steady and predictable, or when the cluster cannot release nodes quickly enough for scaling to pay off. KEDA-driven scaling fits better when demand arrives in bursts, when off-hours idle capacity is expensive, or when you can accept that a freshly scaled worker takes time to start before it runs tasks. Do not run KEDA and a Kubernetes HPA against the same worker deployment, because the two controllers will fight over the replica count. Check the chart documentation for the exact KEDA settings, including idleReplicaCount, minReplicaCount, and the cooldown and grace period values, before you rely on scale to zero in production.
How the default query sizes Celery workers
The Helm chart's KEDA integration sizes Celery workers from a query that counts task instances per Celery queue. By default, the query adds the running and queued task instances for that queue, divides the total by worker_concurrency, and rounds the result up. That number becomes the target replica count for the worker deployment serving the queue. Because the division is rounded up, a single queued task still asks for one worker, and the replica count rises only when the backlog exceeds the slots the current workers already provide.
worker_concurrency sets how many task slots each Celery worker offers. Raising it lowers the replica count for the same backlog, since each pod absorbs more tasks, but it also raises the load on that pod's CPU, memory and any shared connections. Lowering it produces more, smaller workers that scale in finer steps. Each queue gets its own scaling target, so a default queue and a dedicated queue for heavy transforms can grow and shrink independently. Confirm which queues your chart version scales, and whether each queue maps to its own worker deployment, before you assign pools or priority weights to them.
The query is the default, not a fixed contract. Teams should confirm in their chart version exactly what it counts, including whether running tasks, queued tasks, deferred tasks or tasks in other states are included, and whether the query can be overridden. The Helm chart KEDA documentation is the place to check this. Test the result in staging by queueing a known number of tasks and comparing the replica count KEDA reports against the arithmetic you expect.
Which KEDA settings to change and when each fits
This table compares the settings that shape how KEDA scales Celery workers, what each one controls, when the default is enough, and what it costs in idle capacity versus task start latency. Read each row against your own queue pattern: bursty pipelines push you toward scale to zero, while steady daytime load favors warm minimums.
| Setting | What it controls and when to keep the default | Trade-off between cost and task start latency |
|---|---|---|
| worker_concurrency | Task slots per Celery worker pod. KEDA divides queued and running task instances by this value to size replicas. Keep the chart default until you have measured how many tasks one worker handles without memory or CPU pressure. | Higher values mean fewer pods and lower cost, but a crowded worker slows every task on it. Lower values start more pods faster but raise idle overhead. |
| idleReplicaCount | Replica count when no tasks are queued or running. Set to 0 for scale to zero, which the chart's KEDA integration requires for this setup. Keep at 0 when queues are idle for long periods. | Scale to zero saves the most idle cost, but the first task after a quiet period waits for a worker pod to start and pull its image. |
| minReplicaCount | Floor for worker replicas above zero. Raise it when a few workers must always be ready for short or time-sensitive DAGs. Keep at 0 unless you can show that cold starts break an SLA or schedule. | Each warm worker is paid for even when idle, but tasks start without pod scheduling delay. Confirm the minimum with your own latency targets. |
| cooldownPeriod | Seconds KEDA waits after load drops before scaling down. Use a longer period for pipelines that arrive in waves, so workers are not removed between bursts. Keep the default if load is steady. | Longer cooldowns keep workers available for the next wave and raise idle cost. Shorter cooldowns reduce cost but can cause repeated scale-up and scale-down churn. |
| terminationGracePeriodSeconds | Time a terminating worker has to finish running tasks and clean up. Set it to fit your longest typical task and shutdown hook. Keep the chart default only after confirming your longest task runtime. | A longer grace period protects running tasks from being killed, but slows scale-down and keeps pods alive longer. Pair it with cooldownPeriod so cooldown is slightly longer. |
| Per-queue scaling targets | Each Celery queue can have its own worker set and scaling behavior, so heavy queues do not starve light ones. Define queues that map to workloads with different runtimes or resource needs. Keep a single default queue for small installs. | More queues isolate workloads but add worker sets to run, monitor and size. Each queue is a separate capacity decision, and idle queues still cost something if their minimums are above zero. |
| Autoscaler ownership (KEDA or HPA) | Only one autoscaler should control a given worker deployment. Disable the HPA for workers that KEDA manages, and confirm this in the rendered chart values before go-live. | Running both causes replica counts to fight each other. Keep the choice documented so upgrades do not re-enable a second scaler. |
Cooldown period versus termination grace period
The Helm chart's KEDA guidance sets a timing rule for scale-in: cooldownPeriod should be slightly larger than terminationGracePeriodSeconds. The grace period is how long Kubernetes lets a Celery worker shut down after it receives a stop signal, so a warm shutdown can finish current work. The cooldown is how long KEDA waits after the trigger reads zero before it scales the workers to zero. If the cooldown is shorter than the grace period, KEDA can remove a pod while its shutdown is still running, and the worker loses the chance to finish or hand back its work cleanly. Set the two values together in the same change, and keep the cooldown longer, not equal.
When a worker is stopped too early, the tasks it is running are the main risk. A task that is still executing can be interrupted mid-run, and Airflow then depends on its retry and task-state handling to recover it. Retries are only safe if the task is idempotent, so check which DAGs on each queue write to external systems before you shorten any timer. Also note that idleReplicaCount must be 0 for scale-to-zero, and a minReplicaCount above 0 keeps idle workers running, which changes how often this shutdown path runs at all. Confirm with your own team which tasks are long-running and whether they can resume safely.
Test the values against your longest real tasks, not against a default. Pick the queue with the longest-running tasks, trigger a run that sends work to it, and let the queue drain while you watch the worker pods and the task instances in the Airflow UI. Confirm that a pod marked for removal stays up until its last task finishes or the grace period expires, and that the cooldown is longer than that grace period. Repeat the test during a scale-in event with work still queued, since a new task arriving during shutdown is a common source of surprises. Record the chosen values in your change record so a later change to either setting is reviewed together.
Setting up KEDA scaling for Airflow Celery workers
This sequence moves a CeleryExecutor deployment from fixed worker replicas to queue-driven scaling, so workers add capacity when tasks wait and scale down when the queue is empty. Each step ends with a check you can run before moving on.
Confirm the executor and production backends
Confirm with your platform team that the deployment uses CeleryExecutor and that the metadata database and Celery broker are production grade rather than test instances. The production guide recommends PostgreSQL or MySQL for the metadata database, because the default SQLite backend is for testing only. Done means you can name the metadata database engine, the broker, and the Airflow version in use, and that the Helm values show executor set to CeleryExecutor. KEDA reads queue state from these components, so an unstable database or broker will make scaling decisions unreliable.
Install KEDA in the cluster
A cluster administrator installs KEDA into the Kubernetes cluster that hosts Airflow, following the KEDA installation method your organization approves. Install it in its own namespace with cluster-scoped permissions reviewed by your security team, since KEDA creates autoscaling objects for workloads. Done means the KEDA operator and metrics components are running and healthy, and the KEDA custom resource definitions are present. Confirm these with your cluster tooling before changing any Airflow release.
Disable HPA on the Celery workers
Find any HorizontalPodAutoscaler that targets the Airflow Celery worker deployment and remove it before enabling KEDA. KEDA and HPA must not both control the same workers, because two autoscalers writing to one replica count produce unpredictable scaling. Done means no HPA object references the worker deployment, and only KEDA will set its replica count. Check this with your Kubernetes tooling and record the change in your deployment repository.
Enable KEDA in the Helm values
In the Airflow Helm chart values, enable the KEDA autoscaling section for the Celery worker and keep the default query, which counts running and queued task instances per Celery queue. Apply the change through your normal release process, using the chart version that matches your Airflow version. Done means the rendered manifests contain a ScaledObject for the worker set and the worker deployment no longer has a fixed replica target that conflicts with it. Review the rendered output before applying it to a production namespace.
Set worker concurrency and queue targets
Set worker_concurrency to the number of task slots one worker pod can run safely, given its CPU and memory requests and the resource profile of your tasks. The default query divides the count of running and queued tasks per queue by this value and rounds up, so the value sets how many workers each queue asks for. Map each Celery queue to the workers that should serve it and confirm that queue names in DAGs match the scaled set. Done means a worked calculation for each queue matches what you expect for a known number of tasks.
Set idle and minimum replica counts
Set idleReplicaCount to 0 so that workers scale to zero when no task is running or queued. Set minReplicaCount above 0 only if you need warm workers for latency-sensitive queues, because each idle worker keeps consuming node capacity. Confirm the maxReplicaCount ceiling with your capacity owners, since it caps how many workers can run at once. Done means the ScaledObject shows the idle and minimum values you chose, and you can explain the cost of idle workers for each queue.
Align cooldown with the termination grace period
Set cooldownPeriod to a value slightly larger than terminationGracePeriodSeconds on the worker pods. The Helm chart documentation advises this so that a worker is not removed before it finishes cleanup, such as completing running tasks during a graceful shutdown. Check the grace period value in the rendered pod spec and confirm the cooldown in the ScaledObject are consistent. Done means both values are recorded together and a scale-down event in a test environment does not stop tasks mid-run.
Verify with a burst of tasks
Trigger a DAG that submits a burst of tasks to a scaled queue, then watch the worker replica count, the queue depth of running and queued task instances, and task completion in the Airflow UI and your metrics dashboards. Expect replicas to rise as the queue grows and to fall back toward idleReplicaCount after the queue drains and the cooldown passes. Done means every queued task completes, replicas return to the expected idle count, and no task fails during scale-down. Repeat the test after any change to worker_concurrency or the cooldown values.
Why KEDA and HPA must not compete on the same workers
KEDA and the Kubernetes Horizontal Pod Autoscaler both change replica counts, and on the same Celery worker Deployment they will disagree. The cards below cover the failure modes that follow from that conflict and the checks that confirm which controller owns the workers before you change anything.
Two controllers, one replica count
KEDA creates and manages an HPA behind the scenes, so an HPA you define yourself on the same Deployment is a second writer for spec.replicas. Each controller reads its own metric and writes its own target. Neither one is wrong on its own terms, but the Deployment only has one replica count, so the last write wins on each reconciliation loop. Check whether more than one HPA targets the worker Deployment before you tune any thresholds.
Flapping worker replicas
When two controllers chase different signals, the worker count can swing up and down between reconciliation cycles. A queue-depth signal from KEDA may ask for more workers while a CPU-based HPA asks for fewer, and the Deployment oscillates. Watch the replica count over an hour during a normal scheduling burst. Repeated scale-up and scale-down events on the same Deployment are the first visible sign of this conflict.
Scale-down during running tasks
A scale-down that the second controller requests can terminate a Celery worker that is still executing task instances. Airflow then has to recover those tasks, which can leave them in an unclear state and delay the DAG run. The Helm chart's guidance on terminationGracePeriodSeconds and cooldownPeriod only protects workers when one controller is in charge of when they leave. Confirm that a termination grace period is set long enough for the longest task you run on those workers.
Scale-to-zero breaks under contention
KEDA's scale-to-zero behavior depends on idleReplicaCount being 0 and on KEDA deciding when the queue is empty. An HPA that enforces a minimum replica count of one or more keeps workers alive even when KEDA wants them gone, or it removes workers KEDA expects to keep. Either outcome defeats the reason for autoscaling Celery workers in the first place, so check minReplicaCount and idleReplicaCount on the ScaledObject after any change to the HPA.
Find which controller owns the workers
Run kubectl get hpa -n <airflow-namespace> and kubectl get scaledobject -n <airflow-namespace> to list every autoscaler in the namespace. Then inspect the target reference on each HPA and confirm that only the KEDA-generated object points at the Celery worker Deployment. Check the Deployment's events for scaling messages from more than one source. Record the result in your change log so the next engineer does not add a second autoscaler by mistake.
Remove HPA from Celery workers
Recommended approach: keep KEDA as the single autoscaler for Celery workers and remove any HPA you created for them. KEDA already reads the metric that matters for Airflow, which is running and queued task instances per queue, so a CPU-based HPA adds a second signal without adding information. Before removing the HPA, confirm that no other workload depends on it. Tuning the thresholds of both controllers is not a substitute for choosing one.
Questions buyers ask about KEDA autoscaling for Airflow
Does KEDA autoscaling work with the KubernetesExecutor?
No. This page covers CeleryExecutor workers only. The Helm chart's KEDA integration scales Celery workers based on task instances that are running or queued per Celery queue. With the KubernetesExecutor, each task already runs in its own pod, so there is no long-lived worker pool for KEDA to scale. If you are choosing between executors, the executor choice drives the architecture before any scaling question matters. Confirm your executor with the Airflow executor documentation before planning a KEDA setup.
What does a KEDA autoscaling engagement cost, and what drives the effort?
Most engagements are fixed-scope, so the quote follows from a written scope agreed before work starts. The main effort drivers are the number of Celery queues, how varied your task durations and resource needs are, whether your broker and metadata database already run in a highly available setup, how many environments need the same configuration, and how much load testing you want to see before production. Ask for the quote to split one-time work (design, Helm values, load tests, runbook) from recurring work (upgrades, tuning reviews and any retainer for ongoing support). DeployOpen does not publish a price list.
How long does a KEDA setup take, and what does the timeline depend on?
The timeline depends on the state of the platform you start from rather than on KEDA itself. Teams with a working Kubernetes cluster, a healthy Celery broker, a stable metadata database and a documented worker configuration usually move faster than teams that still need to fix queue design, resource requests or observability. Other factors include the cluster's change process, how many queues need separate scaling rules, and how long load tests take to reproduce your real peaks. Ask for a timeline in the written scope that lists the dependencies and the environments covered.
What must our team provide before a KEDA engagement starts?
Plan to provide the following before work begins:
- Kubernetes access with rights to install the Helm chart, create KEDA objects and read pod and node metrics
- Access to the Celery broker and the metadata database, including a view of connection limits and current sizing
- Your current DAG workload profile: peak hours, queue names, typical task durations and resource requests
- Sign-off from the people who own the cluster, database and monitoring stack
- A non-production environment where load tests can run without affecting production schedules
The more accurate this profile is, the less time goes into guessing thresholds during tuning.
Can our own platform team set up KEDA autoscaling without a partner?
Yes, if your team already runs Airflow on Kubernetes with Celery and understands the Helm chart values and KEDA objects. The hard part is rarely the first install. It is choosing the scaling thresholds, confirming that KEDA and any HPA do not compete for the same workers, setting cooldown and termination grace periods that match your task shutdown behavior, and proving the result under load. A partner is most useful when the team lacks that experience or lacks time for load testing and a runbook. Either way, document who owns each Helm value after handover.
How does scale to zero affect task start latency?
When idleReplicaCount is 0, workers are removed when there is no queued or running work, so the first task after an idle period waits for a new worker pod to be scheduled, pulled and started before it can run. The delay depends on node availability, image size and pull cache, and whether the cluster must add nodes. Keeping minReplicaCount above 0 avoids that cold start but keeps idle workers running. The trade-off is cost against responsiveness for scheduled work that must begin on time. Measure queued time in a test before deciding, and confirm the start-up behavior of your own cluster.
Why must KEDA and the Kubernetes HPA not both manage the same Celery workers?
Two controllers that both set the replica count for the same workload fight each other, causing workers to scale up and down unpredictably and sometimes removing workers that still hold running tasks. The Helm chart's KEDA docs state that KEDA and HPA must not both be used on the same workers. Confirm that no HPA targets the worker deployment before enabling KEDA, and review this during every upgrade. Also set cooldownPeriod slightly longer than terminationGracePeriodSeconds so workers finish cleanup before removal.
How DeployOpen scopes and supports Airflow autoscaling
DeployOpen scopes autoscaling work in writing before any change is made to your cluster. The scope starts with the current state: which Airflow version you run, which executor is in use today, how many DAGs and task instances a typical day produces, and which Celery queues carry which work. It then covers the constraints that shape the design, including the Kubernetes version and cluster capacity, whether KEDA is already installed or permitted by your platform team, how the broker and metadata database are sized, and whether any Horizontal Pod Autoscaler already targets the worker StatefulSet or Deployment. Because the Helm chart's KEDA integration and an HPA must not both control the same workers, that check is one of the first items in the written scope. The output is a short design decision record, agreed with your team, that names the queues in scope, the worker sizing approach and the scaling limits.
The test plan is agreed as part of the scope and runs before any production cutover. It normally includes burst tests that queue a large number of short tasks to see how quickly workers scale out and back, and long-running task checks that confirm a worker is not removed while it still holds work. Those long-running checks are where the cooldown period and the termination grace period are validated against your real task durations. The scope should also state the environment used for testing, the success criteria for scale-out latency and task completion, and who signs off. Handover documentation covers the final KEDA ScaledObject settings, the chosen worker_concurrency, the queue-to-scaler mapping, the runbook for a stuck scaler, and the metrics your team should watch.
Ongoing tuning and upgrades are available as retainers. A retainer is useful when task mix changes, when Airflow or the Helm chart is upgraded, or when KEDA or Kubernetes versions move. Costs come from who is paid for what: the initial scoped build is a one-time fee, while a retainer recurs for agreed hours or response terms. Costs rise with the number of queues, the number of clusters, the strictness of your change control and the amount of testing you require. DeployOpen does not publish a price list, and a quote follows from the written scope. When you ask for a quote, request the list of queues and clusters in scope, the test plan with its environments, the handover deliverables, and the retainer terms for support and upgrades, so the figures can be compared on the same basis.
