When CeleryExecutor or KubernetesExecutor fits your Airflow 3 estate
CeleryExecutor fits steady workloads on worker hosts your team already operates, and KubernetesExecutor fits teams that need per-task isolation and already run a Kubernetes platform. Since the hybrid executors were removed in Airflow 3.0.0, that choice is no longer hidden behind a single hybrid class. An Airflow 3 platform team should decide per deployment, using the trade-offs in the Airflow executor documentation as the starting point. CeleryExecutor suits a predictable load with long-running shared workers, where a team is comfortable patching and scaling worker hosts and accepts that tasks can compete for resources on those hosts. Confirm that your team can own the worker fleet, its broker and its capacity planning before choosing it.
KubernetesExecutor suits workloads where one task must not affect another, and where a few minutes of pod startup per task is acceptable. It is not an optional extra on Kubernetes. The Airflow 3.3.2 prerequisites list Kubernetes 1.30 to 1.35, so a supported cluster must exist before you plan the rollout. Check that your cluster version falls in that range, that your platform team can supply namespaces, service accounts and resource quotas for task pods, and that your image build pipeline can produce the task images. If any of these are missing, the cluster work becomes part of the executor decision and should be scoped with it.
The third path is the multi-executor configuration, available since Airflow 2.10.0, where the first configured executor is the default and individual tasks or Dags can select another. This lets you run short, isolated tasks on Kubernetes while heavy, steady jobs stay on Celery workers. Treat the outcome as reversible rather than final: start with the executor that matches your dominant workload, record which Dags need isolation, and review the split after the first full cycle of production runs.
How the two executors compare on the factors that drive the decision
This table compares CeleryExecutor and KubernetesExecutor on the factors that most often decide the choice, using the wording of the Airflow executor documentation. Read each row as a trade-off: the same property can be a benefit in one team's environment and a cost in another's, so check each row against your own workloads before deciding.
| Factor | CeleryExecutor | KubernetesExecutor |
|---|---|---|
| Task placement | Tasks go to a central queue and are pulled by remote workers | Each task runs ad hoc in its own container or pod |
| Worker lifecycle | Workers are long-running and shared across many tasks | A pod is created for each task and exists for that task's run |
| Noisy neighbor exposure | Shared hosts have the noisy neighbor problem, with tasks competing for resources | Each task is isolated in its own container, so the documented noisy neighbor problem does not apply |
| Startup latency | Tasks start on workers that are already running | Tasks wait while the pod deploys before the task can begin |
| Scheduler decoupling | Workers are decoupled from the scheduler, which the docs describe as more robust; test this under your own failure scenarios | Tasks are launched through the Kubernetes API, so cluster health becomes part of the scheduling path |
| Kubernetes requirement | Does not require Kubernetes; workers run on hosts you manage, plus a message broker | Requires a Kubernetes cluster you operate or have access to, with the cluster skills that implies |
| Multi-executor support | Can run alongside other executors when Airflow 2.10.0 or later is used | Can run alongside other executors when Airflow 2.10.0 or later is used |
| Status in Airflow 3 | Supported; the statically coded CeleryKubernetesExecutor hybrid is no longer supported from Airflow 3.0.0 | Supported; the statically coded LocalKubernetesExecutor hybrid is no longer supported from Airflow 3.0.0 |
How a task reaches a worker under each executor
This sequence shows the path one task takes from the scheduler to a final state under CeleryExecutor and KubernetesExecutor, so you can see where each executor adds delay or contention before you commit to one.
Scheduler queues the task
The scheduler reads the Dag, works out which task instances are ready to run, and marks them queued in the metadata database. This happens the same way under both executors, so the scheduler's role does not change your choice. You know this step is done when the task instance shows a queued state in the metadata database and the scheduler has handed it to the executor. Airflow supports more than one scheduler at a time through database row-level locks, so confirm how many schedulers your version runs before you size the control plane.
Executor takes over the queued task
The configured executor receives the queued task instance from the scheduler. Under CeleryExecutor, the executor places the task on a central queue that remote workers read. Under KubernetesExecutor, the executor asks the Kubernetes API to start a pod for that single task. The Airflow executor docs group these as queued remote executors and containerized executors respectively. You know this step is done when the task has left the scheduler's hands and is visible to either the queue or the Kubernetes API. Confirm the broker and result backend settings for your Airflow version in the Airflow documentation before you test this path.
Worker or pod picks up the task
Under CeleryExecutor, an already running worker pulls the task from the queue, so no new process has to start for the task itself. Under KubernetesExecutor, the pod is scheduled, the image is pulled if needed, and the container starts before Airflow can run anything. This is where the startup latency that the Airflow executor docs describe for KubernetesExecutor appears. You know this step is done when a worker has acknowledged the task or the pod reports a running container.
Task runs on shared hosts or in its own container
Under CeleryExecutor, the task runs as a process on a shared worker host alongside other tasks, so CPU, memory and I/O may compete with them. The Airflow executor docs call this the noisy neighbor problem. Under KubernetesExecutor, the task runs inside its own container, which isolates it from other tasks at the cost of the pod startup delay. Check the resource requests and limits you set for workers or pods against the workloads you expect, because that is where contention or idle capacity shows up. You know this step is done when the task process exits and reports its return code.
Worker or pod reports the final state
When the task finishes, the worker or the Kubernetes pod reports the outcome back to Airflow, which records success, failure or retry in the metadata database. Under KubernetesExecutor, the pod is cleaned up after the task ends, subject to your cluster and Airflow settings. Under CeleryExecutor, the worker process stays up and picks up the next task. You know this step is done when the task instance state in the metadata database matches what the logs show and no orphaned pod or stuck queue entry remains.
Compare the observed delay and contention
Run one representative Dag under each executor in a test environment and record the time from queued to running, along with resource use on shared workers. For CeleryExecutor, look for tasks slowing each other down on the same host. For KubernetesExecutor, look for the time the pod spends deploying before the task begins. These measurements come from your own environment, not from published figures, so decide on the executor only after you have them. You know this step is done when you have written measurements for both paths and can explain which delay matters more for your workloads.
Where noisy neighbors and startup latency actually hurt
Each executor fails in a different place. CeleryExecutor puts tasks on shared long-running workers, so contention shows up when workloads differ in weight. KubernetesExecutor gives each task its own pod, so the cost shows up as startup delay and cluster capacity. The cards below cover the risks that matter most and the conditions under which each one applies.
Shared-host contention on Celery workers
CeleryExecutor sends tasks to a central queue, and remote workers pull them. The Airflow executor docs note that workers are decoupled from the scheduler, which makes the setup robust, but tasks on the same shared host compete for CPU, memory and I/O. This matters most when heavy transforms and light sensor or API tasks share one worker pool. Check it in a pilot by running the heaviest DAG alongside the lightest and watching whether the light tasks slow down. Queue assignment is the control to review: separate queues for heavy and light work reduce the overlap.
Pod startup delay on Kubernetes
KubernetesExecutor runs each task in its own container, which removes the noisy neighbor problem. The trade-off is startup latency, because the pod must deploy before the task can begin. This matters most for short tasks on tight schedules, where startup time can be a large share of total runtime. Measure the gap between a task being queued and its first log line in a pilot, under realistic cluster load rather than an empty cluster. Image size, node autoscaling and image pull policy all affect that gap, so ask your platform team to confirm how each is configured.
Scheduler load and database connections
Both executors depend on the scheduler and the metadata database, and the load grows with task volume. Airflow supports running more than one scheduler at the same time for performance and resiliency, using database row-level locks, according to the Airflow scheduler documentation. This applies to both CeleryExecutor and KubernetesExecutor. The Airflow prerequisites say SQLite should not be used in production, and PostgreSQL or MySQL is recommended. In a pilot, track scheduler loop time and connection counts at the highest task volume you expect, and confirm the database connection limits with your DBA before you scale.
Sizing worker pools for peak load
CeleryExecutor workers run continuously, so you pay for capacity whether or not tasks are queued. Sizing the pool for peak leaves idle workers most of the day, while sizing it for average load builds a queue backlog during bursts. KubernetesExecutor shifts that decision to the cluster, where node pools scale with pending pods, but it needs headroom and quota that your cluster team must provide. This matters most for workloads with sharp daily or month-end peaks. Pilot with a replay of your real peak schedule, and record queue wait time and idle capacity for each option before you commit to a pool size.
Choosing an executor by team, workload and existing platform
Choose this path when your organization already runs Kubernetes and has a platform team that owns clusters, namespaces, quotas and image pipelines. KubernetesExecutor runs each task in its own pod, so isolation comes from the cluster you already operate. The Airflow executor docs describe the trade: each task is isolated from noisy neighbors, but the pod must deploy before the task can begin. Confirm your version first. A new deployment should target Airflow 3.x, since Airflow 2 reached end of life on April 22, 2026.
- Deciding condition: your platform team already operates the cluster, and mixed workloads (Python, dbt, custom containers) benefit from per-task images.
- First test: run one representative Dag with pod-per-task execution and measure the time from task queued to task running.
- Main risk: cluster capacity and quota contention now set task start times, and pods that fail to schedule can look like Airflow failures.
- Version fit: the Airflow 3.3.2 prerequisites list Kubernetes 1.30 to 1.35, so confirm your cluster version falls in that range before you pilot.
Check which team owns pod-level logs and who responds when a task pod stays pending.
Choose this path when you run many short tasks on a steady schedule and have no Kubernetes cluster to build on. CeleryExecutor sends tasks to a central queue, where long-running remote workers pull them. The Airflow executor docs say workers are decoupled from the scheduler, which makes this model more robust, but shared workers can suffer from the noisy neighbor problem when tasks compete for resources on the same hosts. Because the workers stay warm, per-task startup overhead does not accumulate the way a pod-per-task model would.
- Deciding condition: task volume is high, task duration is short, and you can operate a worker fleet and its queue backend.
- First test: replay a peak-hour load and watch queue depth and per-host resource contention.
- Main risk: shared hosts make one heavy task slow down every task on the same worker, and worker patching becomes your work.
- Database fit: the scheduler can run more than one instance at once, using database row-level locks, but the metadata database must be PostgreSQL or MySQL, since SQLite is for testing only and MariaDB is not supported.
Ask your team to confirm the queue backend, worker scaling rules and the on-call owner for worker hosts before you commit to this path.
Choose this path when a small number of jobs need large amounts of memory or CPU, or must not share resources with other tasks. KubernetesExecutor gives each task a dedicated container, which is the isolation the Airflow executor docs describe. A startup delay matters less here because these jobs usually run long. Isolation is not automatic, though: a pod without resource requests can still compete with its neighbors on a shared node.
- Deciding condition: a few jobs are heavy enough that a noisy neighbor would cause real harm, and their run time is long compared with pod startup.
- First test: run the heaviest job with its declared resource requests and confirm it does not evict or starve other workloads.
- Main risk: missing or inaccurate resource requests leave isolation on paper only, and node pool placement must be set per task.
- Security baseline: the Pod Security Standards describe policy levels you can apply to the namespace that runs task pods, which matters if these jobs touch sensitive data.
If these jobs need a different executor from the rest of your estate on Airflow 3.x, the multiple executors feature is the supported route, because the hybrid executors were removed in 3.0.0. Confirm the request sizes with the team that owns the jobs.
Choose this path when your estate contains both latency-sensitive, short Dags and heavy, isolated Dags, and you want each group on the executor that suits it. Since Airflow 2.10.0, one deployment can run a multi-executor configuration. The first configured executor is the default, and tasks or Dags can select the others. On Airflow 3.x, this is the supported way to mix executors, because CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported from 3.0.0.
- Deciding condition: your workloads split cleanly into two or more profiles, and your Dag authors can follow a routing convention.
- First test: deploy one Dag that explicitly selects each executor, then confirm the default executor is the one you intended.
- Main risk: two execution paths double the monitoring, upgrade and security surface, and a wrong executor selection in a Dag can misplace a heavy job.
- Migration note: if you run Airflow 2 today, the hybrid executors belong to a version past end of life, so this decision is also a migration decision.
Before you adopt this path, confirm with your vendor documentation how each executor path handles secrets, logs and access control in your environment.
What KubernetesExecutor asks of your cluster
KubernetesExecutor moves much of the operating burden onto your cluster, so the first decision is whether your platform team already runs Kubernetes control planes well enough to carry Airflow tasks on top. Every task pod depends on the control plane being available, so the kube-apiserver, its etcd cluster and the scheduling path have to be treated as production dependencies with their own backups, upgrades and on-call coverage. The kubeadm high availability guide describes two topologies: a stacked control plane with etcd on the same nodes, and an external etcd cluster. Either way, it calls for at least three control plane nodes and a TCP load balancer in front of kube-apiserver on port 6443. Kubernetes documentation also says etcd should run with an odd number of members, recommends five members in production, and requires a backup plan, using etcdctl snapshot save for backups and etcdutl for restores. Confirm who owns these steps and how often you rehearse a restore.
Version management is the second recurring cost. The Kubernetes version skew policy requires kube-apiserver to be upgraded one minor version at a time and before kube-controller-manager, kube-scheduler and kubelets, and it allows kubelets to be up to three minor versions older than the API server. Kubernetes maintains the three most recent minor releases. As of September 2026 these were 1.37 (end of life October 28, 2027), 1.36 (June 28, 2027) and 1.35 (February 28, 2027). Airflow 3.3.2 lists Kubernetes 1.30 to 1.35 as its prerequisites, so a cluster on 1.36 or 1.37 may fall outside what Airflow documents. Check the current Airflow prerequisites page before planning any upgrade order.
Security needs the same attention. Kubernetes stores Secrets unencrypted in etcd by default, and task pods are where Airflow connection credentials usually land. Enable encryption at rest for etcd, restrict list and watch on Secrets with least-privilege RBAC, and review which service accounts the executor and task pods run under. Where your policy requires a central vault, the Secrets Store CSI Driver is one documented option for pulling secrets from external stores. On certification, DeployOpen does not issue certifications or attestations. CNCF conformance applies to the Kubernetes distribution you choose, not to Airflow or to a DeployOpen implementation, so ask your distribution vendor for its current conformance evidence.
What each executor adds to upgrades, backups, monitoring and skills
This table compares the day-two work each executor adds: upgrades, backups, monitoring and team skills. Read each row as the extra scope you own beyond running Airflow itself. The main takeaway is that KubernetesExecutor shifts work onto a Kubernetes cluster you must also upgrade, back up and monitor, while CeleryExecutor adds a broker and worker fleet to operate.
| Area | CeleryExecutor | KubernetesExecutor |
|---|---|---|
| Airflow version upgrades | Upgrade the scheduler, DAG processor, API server and Celery workers together, keeping worker images on the same Airflow version as the scheduler. | Upgrade Airflow and the worker image that tasks run in. Each task pod uses the image you configure, so image tags must match the DAGs they run. |
| Kubernetes upgrades | Not required by the executor. Only applies if your Airflow components happen to run on Kubernetes. | Plan minor-version upgrades of the cluster. The Kubernetes version skew policy requires the API server to move one minor version at a time, ahead of controller manager, scheduler and kubelets. |
| Backups of the metadata database | Back up the metadata database, which holds DAG run and task state. Airflow's production guide recommends PostgreSQL or MySQL over SQLite, which is for testing only. | Same metadata database requirement. Confirm with your DBA how often and how far back you restore, since task state is needed to recover interrupted runs. |
| Backups of cluster state | Back up any queue or broker you choose to run and the worker configuration. Confirm the broker recovery approach with your team and your Celery broker documentation. | Back up etcd for the cluster. Kubernetes docs say etcd needs a backup plan, with etcdctl snapshot save for backups and etcdutl for restores. Secrets are unencrypted in etcd by default, so confirm encryption at rest. |
| Worker and queue monitoring | Watch worker health and queue depth. Tasks waiting in the central queue show as backlog, and shared worker hosts need resource monitoring for noisy neighbor effects. | Little worker fleet to watch. Monitor the scheduler and the pods it creates, which can be slow to start while the pod deploys. |
| Pod scheduling and image pulls | Not applicable in the same way. Watch host capacity and worker restarts instead. | Watch pods stuck pending for scheduling reasons and image pull failures. Startup latency comes from the pod deploying before the task can begin. |
| Required skills | Celery worker operations, broker operations and host-level sizing, plus Airflow administration. Confirm the depth needed with your own platform team. | Kubernetes operations: cluster upgrades, RBAC, network policy and Pod Security Standards, plus Airflow administration. Confirm which of these your team already runs. |
| Airflow 2 end of life as an upgrade driver | Airflow 2 reached end of life on April 22, 2026 and gets no fixes. Plan the Airflow 3 upgrade regardless of executor, since the upgrade changes how you run both workers and schedulers. | The same end-of-life date applies. Airflow 3 changes apply to pod-based tasks as well, so plan the upgrade and the Kubernetes version together with the Airflow upgrade guide. |
Running both executors and what changed with the hybrids in Airflow 3
Running both executors is now a supported option, but only if the routing is simple enough to test. Since Airflow 2.10.0, a single deployment can run several executors. The first configured executor is the default, and individual tasks or Dags can select another one. In practice, a common split is a CeleryExecutor pool for routine, short tasks where worker reuse keeps latency low, with KubernetesExecutor reserved for tasks that need isolation from noisy neighbors or a heavy, task-specific image. The executor docs describe both behaviors, so the split follows directly from the trade-offs covered above.
The split adds cost in operations, not just configuration. You now run two sets of worker infrastructure: long-lived Celery workers with their broker and queues, and a Kubernetes cluster with its pod templates, image pipeline and RBAC. Each side has its own images to keep patched and its own logs to search when a task fails, so on-call engineers must know which executor ran a given task. Routing rules also need tests. Confirm that every Dag that should land on the Kubernetes side actually selects it, that the default executor is what you intend, and that a misrouted task fails visibly rather than running in the wrong environment. Ask your Airflow administrator to document the selection rule in the Dag authoring guide before teams start relying on it.
If you are migrating from CeleryKubernetesExecutor or LocalKubernetesExecutor, treat that move as an upgrade task, not a configuration tweak. Both hybrid executors are no longer supported from Airflow 3.0.0, and the Airflow docs recommend the multiple executors feature instead. Read the Airflow 3 upgrade guide before you change anything, since it covers the wider set of Airflow 3 changes that will also affect your Dags. Then test Dag-level executor selection in a non-production environment, run representative Celery and Kubernetes tasks end to end, and confirm the Airflow version you are targeting is one that still receives fixes. Plan cutover only after those tests pass.
Questions buyers ask before choosing an Airflow executor
What drives the cost of CeleryExecutor and KubernetesExecutor?
The cost follows where the compute runs and who operates it. CeleryExecutor keeps long-running worker hosts or node capacity busy, so you pay for that capacity whether or not tasks are queued, plus a message broker and the metadata database. KubernetesExecutor pays for node capacity used by each task pod, plus the cluster itself, which someone must upgrade, secure and monitor. Engineering time is the largest variable for both. Ask any vendor or internal team to price worker or node capacity at your peak and average task load, the broker and database sizing, and the hours needed to run the platform after go-live, and keep the Airflow project itself free under the Apache License 2.0.
Does KubernetesExecutor require a Kubernetes cluster my team does not run yet?
Yes. KubernetesExecutor launches each task as a pod, so it needs a working Kubernetes cluster that the Airflow scheduler can reach and that your team can operate. The Airflow 3.3.2 prerequisites list Kubernetes 1.30 to 1.35, and the Kubernetes project maintains only its three most recent minor releases, so an existing cluster must also be kept within a supported version. A production control plane needs at least three control plane nodes behind a TCP load balancer, and etcd needs backups. If your organization has no cluster and no Kubernetes operations skills, CeleryExecutor on existing virtual machines is the smaller first step. Confirm your cluster version, upgrade path and etcd backup ownership before choosing.
How long does switching executors take, and what does it touch?
The duration depends on your DAG count, how many tasks depend on shared worker state, and how much of the platform is already containerized, so ask for a written estimate tied to your environment rather than relying on a generic figure. A switch usually touches several areas: DAG code that assumes shared workers or local files, container images that must include every Python dependency per task, Kubernetes secrets and service accounts for each pod, and logging, because task logs must be collected from ephemeral pods rather than long-running hosts. The Airflow executor documentation is the right starting point for these checks. Plan a pilot on a representative set of DAGs before migrating the rest.
Can one Airflow deployment run both CeleryExecutor and KubernetesExecutor?
Yes, from Airflow 2.10.0 onward. The multi-executor configuration lets several executors run in one deployment, where the first configured executor is the default and individual tasks or Dags can select another. This suits a team that wants shared workers for light, frequent tasks and isolated pods for heavy or untrusted workloads. The older hybrid executors, CeleryKubernetesExecutor and LocalKubernetesExecutor, are no longer supported from Airflow 3.0.0, and the Airflow docs recommend the multiple executors feature instead. Running both means operating both worker types, so the operational cost of each path still applies. Confirm the selection method for your Airflow version in the executor documentation before designing around it.
What skills does each executor need from my team?
CeleryExecutor needs people who can run a message broker, manage shared worker hosts, tune worker concurrency and handle host-level patching and dependency drift on those workers. KubernetesExecutor needs people who can operate a Kubernetes cluster, write and review pod and image configuration, manage RBAC and secrets, and troubleshoot scheduling and pod startup. Both need solid Airflow administration: a production metadata database, PostgreSQL or MySQL rather than SQLite, and scheduler high availability. Skills gaps are often the deciding factor, so map your current staff against these areas before scoping work.
Which security controls change when tasks run in Kubernetes pods?
Several controls shift to Kubernetes. Kubernetes stores Secrets unencrypted in etcd by default, so you need encryption at rest, least-privilege RBAC limiting list and watch access, or an external store through the Secrets Store CSI Driver. Each task pod also needs its own image hardening and pod security settings, which you can check against the Pod Security Standards. Kubernetes authenticates users through certificates, OpenID Connect or other bearer tokens, so the identity model for operators must be designed too. CeleryExecutor shifts the work to host hardening and broker access controls. Ask your security team which model matches your evidence and audit requirements, and confirm the controls with your own assessor.
Can DeployOpen scope a pilot for choosing or implementing an executor?
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and most engagements are fixed-scope with scope, deliverables and timelines agreed in writing before work starts. A pilot could cover a representative set of DAGs, a comparison of worker and node sizing, and a documented runbook for the chosen executor, but the exact scope depends on your environment. DeployOpen does not publish a price list, so request a written scope and quote from the team to see what is included. Ongoing support and upgrades are available as retainers.
How DeployOpen scopes, delivers and supports Airflow executor deployments
DeployOpen plans, implements, integrates, hardens, backs up, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Airflow CeleryExecutor vs KubernetesExecutor, that means turning the executor choice into a written scope covering workers, queues, clusters, databases and handover before any build begins.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope states which executor or executors run which task types, such as Celery workers, Kubernetes pods or both.
- Scope names the metadata database, Kubernetes cluster or worker hosts, and the identity and secrets integrations involved.
How we build
- Build covers the Airflow installation method, executor configuration and any queue or multi-executor routing agreed in scope.
- Testing checks task startup latency, worker isolation and failure recovery against the executor behavior the scope defines.
- Hardening covers database backups, secrets handling and access controls in your own environment, mapped to your internal policy.
How we hand over
- Handover includes documentation of the executor design, configuration keys, upgrade path and operational runbooks.
- Your team receives a record of what was tested, how it was tested, and which items remain open at handover.
- Ask us to confirm that the handover covers the Airflow 3 version your team will operate, not only the initial install.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainer scope should name which Airflow and Kubernetes versions are covered and who owns version upgrades.
- Ask for the support and response terms in writing before signing, since they are set per engagement and not published as a price list.
