Apache Airflow executor selection: Celery vs Kubernetes

Airflow executor selection starts with where your tasks must run

The executor decides where task processes run and who operates that capacity. If you are planning a self-hosted Apache Airflow 3 deployment, you must choose an executor before any infrastructure is provisioned, because the choice determines whether you will run a fixed pool of worker hosts, a Kubernetes cluster that starts a pod per task, or a combination of both. Airflow ships LocalExecutor, CeleryExecutor, KubernetesExecutor and EdgeExecutor, along with AWS Batch and ECS executors, and the executor documentation confirms that since Airflow 2.10.0 several executors can run in one deployment.

The decision is hard to reverse. The executor shapes your worker infrastructure, the load your metadata database carries, how task logs are collected, and how your container image pipeline is built and patched. A team that starts with a worker fleet and later moves to pod-per-task execution will rebuild most of those pieces. Before you commit, write down the workloads you expect to run: their typical duration, their resource needs, whether they need GPUs or specialized images, and how many run at the same time. Those answers matter more than any general recommendation.

The common mistake is copying a pattern from a blog post or conference talk without matching it to your situation. A Celery setup that works for one organization may fail another that has no team to operate a message broker, or that runs its tasks inside a network segment where the workers cannot reach the database. A Kubernetes setup that suits a platform team with cluster experience may be a poor fit for a data team that has never managed pod scheduling. Check the pattern against three things: your workloads, your network boundaries, and the skills your team will actually have after go-live.

If your Airflow 2 plan included the CeleryKubernetesExecutor or LocalKubernetesExecutor, you must re-plan. The executor documentation states that both hybrid executors are no longer supported as of Airflow 3.0.0. The question they answered, which tasks go to Celery workers and which go to Kubernetes pods, is still valid, but you now answer it by configuring more than one executor in the same deployment and assigning tasks to them explicitly. Confirm the supported combinations and the assignment method against the current executor documentation before you design the queues and images.

Which Airflow components the executor choice touches

The executor sits between the scheduler and the place where tasks actually run, so its choice shapes several components at once. The cards below separate what the executor decides for each component from what stays a separate decision, so you can plan infrastructure, access and upgrades without assuming one setting controls everything.

Scheduler and its executor handoff

The scheduler decides which task instances are ready and passes them to the executor it is configured with. Airflow supports running more than one scheduler at the same time, coordinated through database row-level locks, so the executor choice does not remove the need to plan scheduler count and database capacity. Confirm the scheduler documentation for the locking and concurrency behavior of your target release before sizing the database.

Worker or pod that runs each task

The executor decides where a task process starts: a long-lived Celery worker that pulls from a broker, or a fresh Kubernetes pod created for each task. It does not decide the task's resources, image or network reach, which you set separately through worker sizing, container images and cluster policy. Check which runtime your DAGs need, such as system libraries or GPU access, before choosing the execution environment.

Message broker and queues

CeleryExecutor depends on a message broker to carry task messages between the scheduler and workers, and that broker becomes a component you must operate and secure. KubernetesExecutor does not use a broker for task dispatch, so this component appears only in Celery designs. Confirm the broker product, its high availability model and its access controls with your platform team as part of the executor decision.

DAG processor as a separate component

In Airflow 3 the DAG processor must run as its own component, so it is no longer a side process of the scheduler. The executor choice does not change this requirement, but it affects how DAG files reach workers, because Celery workers and Kubernetes pods each need access to the same DAG code. Confirm how your DAG delivery method, such as git-sync or baked images, reaches every execution environment.

API server replacing the webserver

Airflow 3 replaces the webserver with an API server started with airflow api-server. Task code in Airflow 3 reaches Airflow through the REST API or Task SDK rather than direct database sessions, so the executor's workers depend on API server reachability. Plan network paths from workers or pods to the API server and confirm them in a test run on your chosen executor.

Metadata database

Every executor writes task state to the metadata database, so its sizing, backend and access are shared across the executor choice. Airflow's prerequisites list PostgreSQL 14 to 18 and MySQL versions, and state that MariaDB is not supported, while SQLite is for testing only. Confirm the exact supported versions for your target release, and plan connection limits for the number of workers or concurrent pods you expect.

Choosing LocalExecutor, CeleryExecutor, KubernetesExecutor or EdgeExecutor

This table compares the four Airflow executors and a multi-executor deployment by workload fit and by what each one asks your team to run. Read each row against your team's skills and existing platform, then confirm the behavior that matters for your DAGs with a test run before you commit.

Executor	When it fits	What it costs operationally
LocalExecutor	Small installations where tasks run on the scheduler host, a single team owns the deployment, and task volume stays modest. Teams without a message broker or cluster to operate often start here.	Little extra infrastructure, but tasks share the scheduler machine, so capacity and isolation are limited. Scaling usually means moving to a distributed executor.
CeleryExecutor	Steady, high task volume that a fixed fleet of workers can absorb, and a team comfortable running a message broker and long-lived worker hosts.	A worker fleet to patch and scale, a broker (such as Redis or RabbitMQ, to be confirmed for your setup) to run, and worker dependencies kept in sync across hosts. Secrets and logs must reach every worker.
KubernetesExecutor	Bursty or heterogeneous workloads that benefit from a pod per task, and an existing Kubernetes platform with a team that runs it. Supported Kubernetes versions for Airflow 3.3.2 are 1.30 to 1.35.	A Kubernetes cluster to run, container image builds for task environments, and pod startup time added to each task. Workers need network reachability to the metadata database, and logs and secrets must be wired through the cluster.
EdgeExecutor	Tasks that must run in locations separate from the central Airflow components, such as remote sites, where workers connect back to the core deployment.	Edge workers to deploy and patch at each location, connectivity and authentication between sites and the core, and monitoring of remote workers. Confirm current maturity and provider requirements in the executor documentation before planning.
Multiple executors in one deployment	Mixed workloads, for example light tasks on one executor and isolated or heavy tasks on another, since several executors can run together from Airflow 2.10.0 onward. Hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are not supported in Airflow 3.	Two execution platforms to operate, two sets of images or dependencies, and routing rules that DAG authors must follow. Confirm the executor-selection method for your DAG patterns in a test run, since the choice of executor per task affects logging and failure handling.

Running more than one executor after hybrid executors were removed

Since Airflow 2.10.0, a single Airflow deployment can run more than one executor at the same time, but the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0. If your team planned to use one of those hybrids, treat that plan as withdrawn. The supported route is to run several executors side by side and assign each DAG or task to a named executor through configuration. Confirm the current assignment syntax in the Airflow executor documentation before you write any DAG code, because the mechanism is the part most likely to change between releases.

The practical alternative is not a single hybrid class with automatic routing. Instead, your platform has two execution paths: for example, tasks that need a long-running worker pool on Celery and tasks that need an isolated pod per task on Kubernetes. Each path brings its own set of container images, its own Python dependencies and provider packages, its own secrets paths for connections and variables, and its own monitoring and log collection. Before you choose this design, ask who will maintain both images through upgrades, and whether your secrets backend can serve workers in both environments with the same access rules.

Decide the routing rule before you split anything. Common candidates are per-DAG assignment for workloads with clearly different resource profiles, or per-task assignment inside a DAG when only a few steps need special isolation. Per-task routing gives more control but makes DAG reviews harder, because a reader must check each task's executor to understand where code runs. Write the rule down, add it to your DAG authoring standards, and make the executor for each task visible in code review so that a task does not silently land on an executor without the libraries it needs.

Test each executor path separately before you rely on it. Run a representative DAG on every executor with the same image tags, connections and sample data that production will use, and confirm that task logs, retries, timeouts and cleanup behave as expected on each one. Then run a mixed DAG that moves data between the two paths. Failures often appear only at the boundary, such as a missing environment variable on a Kubernetes pod or a worker that cannot reach a database that the scheduler can reach. Record these results so that an upgrade of either path can be repeated against the same checks.

Ask any implementation partner to show how each executor path is built, tested, patched and monitored, and to quote that work as part of the written scope. Running two paths roughly doubles the surface that needs image patching, dependency review and alert tuning, so the effort and the recurring cost of maintenance should be explicit in the quote. If the workload does not clearly need two execution models, a single supported executor with clear resource settings is usually easier to operate and to explain to auditors.

How to build the executor setup in order

This sequence takes an Airflow 3 deployment from version and database checks through a stable single-scheduler setup on each executor path, and only then adds a second scheduler. Each step closes with a check so the team knows it can move on.

Confirm the release, platform versions and prerequisites
Before any installation, the platform team pins the Airflow 3 patch release it will run, currently 3.3.2 according to the version support table, and checks it against the prerequisites page for supported Python, Kubernetes, PostgreSQL and MySQL versions. Confirm the exact versions in your own image and environment, and confirm the Kubernetes version if you plan to use KubernetesExecutor. Check that MariaDB is not your chosen metadata database. The step is done when a written version record lists the pinned release and every supported dependency, approved by the owners of the database and the cluster.
Move the metadata database off SQLite
The metadata database must move from the default SQLite backend to PostgreSQL or MySQL before any executor is configured. The production deployment guide says SQLite is for testing only and can lose data in production. Your database team provisions the instance, sizes it for your DAG count and run history, and confirms backups and restore for it. Confirm RAM for the scheduler and other components against the prerequisites recommendation of at least 4 GB, and size higher for your workload. The check is a successful migration on the new database and a test run that writes task state there.
Deploy scheduler, DAG processor and API server separately
Deploy the scheduler, DAG processor and API server as separate components, since Airflow 3 replaces the webserver with an API server started by the airflow api-server command and requires the DAG processor to run on its own. Each component gets its own process or container, configuration and health probe. Keep all three pointed at the same metadata database and configuration. The check is that each component reports healthy, the UI loads through the API server, and the DAG processor parses at least one DAG without errors.
Pick and configure the executor
Choose the executor from the workload profile: LocalExecutor for a small single-host setup, CeleryExecutor for a worker pool that needs a broker and shared queues, and KubernetesExecutor for per-task pods. Since Airflow 2.10.0 several executors can run in one deployment, but the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported in Airflow 3.0.0 and later, so multi-executor setups must use the supported routing. Record the broker or cluster dependency for the chosen path. The check is a configuration review signed off by the platform owner, with the executor name written in your deployment values.
Set up DAG delivery and the image pipeline
Every executor needs the same DAG code and dependencies on each worker or pod. Choose how DAGs arrive (a git sync or a baked image), then build the image pipeline that produces the Airflow image with your providers and pinned Python packages. The Airflow production guide recommends container images for Kubernetes deployments, and the community Helm chart uses the official Docker image. Pin the image tag in configuration. The check is that a DAG change lands on the scheduler and a worker or pod in the same release, and that the image tag matches the one recorded in step one.
Configure logs and secrets for workers
Workers and pods must read task logs and connection secrets, or tasks will fail even when the executor is correct. Choose a remote log location both the scheduler and workers can reach, and a secrets backend or Fernet key handling that every executor path uses. Confirm the worker identity has read access only to the secrets it needs. The check is a task that writes a log line visible in the API server UI and reads one test connection from the secrets backend, run on the executor you configured.
Run a test DAG on each executor path
Run the same small test DAG on every executor path you enabled, one at a time, and watch the full path: queueing, worker pickup, execution, log output and state update in the metadata database. For Celery, confirm the task reaches a worker through the broker. For Kubernetes, confirm a pod is created, runs and is cleaned up. The check is a task reaching success on each chosen executor, with the run recorded in the UI and the logs readable. Repeat after any change to images or secrets.
Add a second scheduler only after the first path is stable
Airflow supports more than one scheduler at the same time, using database row-level locks, but add the second scheduler only after the single-scheduler setup has passed the test runs on every executor path. Run the second scheduler alongside the first, then stop the first during a scheduled task run to confirm the surviving scheduler keeps dispatching work. The check is a failover test with no lost task state and no duplicated runs, documented with the observed results. Confirm the locking and database settings with your DBA before you start this step.

What drives the effort of the executor decision

This table lists the factors that most change how much work an Airflow executor choice creates, with one way to keep each one manageable. Read each row as a question to answer about your own environment before choosing between Celery, Kubernetes or a mix. The main takeaway is that a single executor usually costs less to operate, and a second executor should be added only when a driver clearly demands it.

Driver	Why it changes effort	How to keep it down
Concurrent tasks and peaks	Higher peak concurrency means more worker capacity to size, scale and monitor. Bursty peaks push effort toward autoscaling and queue tuning.	Measure real peak concurrency from current schedulers and pools before sizing, and cap concurrency with pools so one executor can absorb the load.
Task duration and resource profile	Long-running or memory-heavy tasks need stable workers or per-task resource requests, which adds tuning and capacity planning.	Group tasks by profile and route only the heavy ones to a separate executor or queue, leaving short tasks on the simpler setup.
Number of teams and environments	Each team, dev, test and production environment can need its own workers, images, queues or namespaces, multiplying configuration.	Use one shared executor configuration per environment and separate teams with queues or namespaces instead of separate Airflow deployments.
Existing Kubernetes or VM platform	Teams without Kubernetes skills take on cluster operations for KubernetesExecutor, while teams with a mature cluster can reuse its tooling.	Reuse an existing cluster and its logging, monitoring and policy tooling, or stay on Celery workers if the platform team is VM-based.
Network zones and air-gap constraints	Workers in restricted zones need reachable brokers, databases and image registries, and each path needs approval and testing.	Place the scheduler and metadata database in one zone, keep workers near the data sources they use, and mirror images and packages inside the boundary.
Image and dependency complexity	Per-task dependencies, native libraries and provider packages increase build, test and patch work for every image.	Standardize on one task image with pinned provider versions, and split out a separate image only for tasks that truly need different dependencies.
Logging and secrets integration	Remote log storage, secrets backends and connection handling must work on every worker type, so each executor needs its own verification.	Pick one logging target and one secrets backend, then confirm both behave the same on each executor before go-live.
Compliance evidence needs	Assessors may ask for records of worker isolation, image provenance, access logs and change history, which grow with each executor added.	Keep one documented execution model, and export configuration, image digests and access logs from one place to show the design to an assessor.

How the executor choice changes by deployment model

A single-node deployment runs the scheduler, the API server, the DAG processor and task execution on one host, usually with LocalExecutor. It suits development, proofs of concept and small pipeline sets owned by one team. The setup is simple, but the host becomes the whole failure domain, so teams often move to a distributed executor once uptime or task volume matters.

Typical fit: LocalExecutor, with tasks running as local processes on the scheduler host.
Components reachable: The metadata database, the API server and the DAG processor, all on local or private links. Every source and target system the DAGs call must also be reachable from this host.
Image and registry: Usually none. Packages come from the Python index or an internal mirror and are pinned to the Python and Airflow versions listed in the prerequisites.
Main risk: A single host failure stops scheduling and running tasks. Adding a second scheduler requires a shared database and a second host, which changes the design.

Confirm the Python and PostgreSQL or MySQL versions against the Airflow prerequisites, and confirm that the recommended 4 GB of RAM covers your expected task load.

Securing workers and secrets for each executor

Each executor places workers in a different trust zone, so the security work follows the executor you choose. The cards below cover the network paths, secret delivery, encryption, Kubernetes identity and broker access that differ between Celery and Kubernetes, and what to check before go-live.

Network paths to the metadata database

Celery workers and KubernetesExecutor task pods both connect to the metadata database and other systems, so each path needs an explicit allow rule. Confirm which components must reach the database, which can reach external systems, and which should have no route to the scheduler's network at all. Document the rules per executor, because a Celery worker pool and a pod network usually have different egress paths.

Secret delivery to workers

Connection credentials can reach a worker through a secrets backend that resolves them at task runtime, or through environment variables injected at deployment. A secrets backend keeps values out of container specs and process environments, while injection is simpler to run but exposes values to anyone who can read the pod or worker configuration. Decide the approach per executor and confirm which backend your team can operate.

Fernet key and stored connection encryption

Airflow encrypts stored connection passwords and variables with a Fernet key held in its configuration. Every scheduler, API server, worker and task pod that reads encrypted values needs the same key, so a key mismatch breaks tasks only on some executors. Confirm where the key is generated, how it is stored and rotated, and that each execution environment receives it through the same controlled channel.

Least-privilege Kubernetes service accounts

KubernetesExecutor creates a pod per task, and those pods run under a Kubernetes service account. Give the task pods their own namespace and a service account that can read only the secrets and config maps they need, with no rights to create or delete other workloads. Review the role bindings with the cluster team, since a broad binding lets any DAG author reach cluster resources.

Broker access controls for Celery

CeleryExecutor sends task messages through a broker, and anyone who can publish to it can queue work for workers. Restrict broker credentials to the scheduler and workers, use separate credentials per environment, and confirm that the broker is not reachable from outside the Airflow network. Ask the broker vendor which access control and TLS options your version supports before you rely on them.

Worker image patching cadence

Workers and task pods run their own container image, which can drift from the scheduler image if teams patch them separately. Rebuild and redeploy the worker image on the same schedule as the scheduler image, using the official Airflow image as the base. Track the image version in each environment so a security fix on the scheduler is not missing from the workers that run the tasks.

How to test that each executor fails safely

The point of this testing is to prove that each executor path in your chosen setup recovers without losing or duplicating work. Run the tests against the exact executor configuration you plan to operate, not a single-machine stand-in, because the failure behavior of a Celery worker is different from a Kubernetes pod and both differ from a LocalExecutor process. Use a staging environment that mirrors your production metadata database, broker or Kubernetes namespace, worker images and network policies. Record the Airflow version, provider versions and executor settings before the first run so the results can be tied to a specific configuration.

Start with representative DAGs. Build a small set that succeeds, fails on purpose and retries on purpose, and schedule each one on every executor path you run, including the Kubernetes path if you run both. Confirm that the task instance states in the UI match what the DAG should produce, that retries happen the number of times the DAG defines, and that a failed task does not leave the run stuck in a running state. Include at least one DAG that uses the queue or pod template settings you rely on, since routing errors often show up only under those settings. The exact states and retry rules should be checked against the executor documentation for your release, not assumed from older Airflow 2 behavior.

Next, interrupt work deliberately. Stop a Celery worker process or delete a Kubernetes pod while a task is running, then confirm what Airflow records for that task instance and whether it retries as configured. Watch for tasks that remain marked as running long after the worker is gone, which points to missing heartbeat or orphan handling in your setup. If you run more than one scheduler, restart one of them while DAG runs are active and check the scheduler logs and task history for duplicate task instances. A passing result means each interrupted task ends in a state you can explain, is retried at most as often as the DAG allows, and no run contains two successful copies of the same task instance.

Verify the surrounding services as well. Restart the scheduler alone and confirm that scheduling resumes without manual cleanup. Where more than one scheduler runs, confirm the second one takes over scheduling work and that no task instance is created twice. After a task completes on a remote worker or pod, open its log from the web UI and confirm the log is readable, which depends on the remote logging configuration your team chooses. Finally, raise concurrency to the peak you expect and watch metadata database connections against the limits you set in both Airflow and PostgreSQL or MySQL. Ask your database administrator to confirm the connection ceiling, and record the peak observed value.

Keep the evidence for change records: the configuration files and versions tested, the DAG list with expected outcomes, timestamps and task states for each failure test, scheduler and worker log excerpts, screenshots or exports of the UI log views, and the database connection measurements with the limits they were compared against. Store the results with the change request so an auditor or on-call engineer can see what was tested and what was accepted. Any failed test should be logged with its fix and a retest. Confirm with your internal change owner which evidence format they require, because the record your organization needs may differ from this list.

What to monitor, back up and document after go-live

Once the executor is running, a small set of recurring checks decides whether it stays healthy. Each item depends on the executor path you chose, so review them together with your metadata database, image pipeline and DAG delivery plan. Confirm an owner and a review interval for each item before go-live.

Queue and task-state metrics

Track queued, running and failed task counts for each executor path, along with scheduler heartbeat and time spent waiting to start. For Celery, watch broker queue depth and the number of tasks waiting for a free worker slot. For Kubernetes, watch pending task pods and how long each task waits for a pod to start. Set alert thresholds from your own baseline after a load test, and confirm which metrics your Airflow version exports before you build dashboards.

Worker and pod restart counts

Celery workers and Kubernetes task pods fail in different ways, so monitor each on its own terms. Alert on Celery worker restarts, heartbeat gaps and tasks stuck in a running state after a worker disappears. For Kubernetes, track pod restart counts, out-of-memory kills, image pull failures and pods that stay pending. Review retry counts as well, so a failing node does not hide behind tasks that succeed on retry.

Metadata database backups and restore tests

The metadata database holds DAG run history and task state, so it is the first component to back up and the first to restore in a drill. Run PostgreSQL 14 to 18 or MySQL 8.0 or 8.4, using a backup method your database team already operates. The production guide warns that the default SQLite backend can lead to data loss, and MariaDB is not supported as a metadata database. Restore into a separate environment on a schedule, and record how long each restore takes, because that measurement tells you more about recovery than a successful backup job.

Image rebuild and patch cadence

Celery workers and Kubernetes task pods both run your Airflow image, so the rebuild process must cover both paths. Rebuild when a new Airflow patch release appears, such as 3.3.2, and when the base image receives a security update, then promote the image through staging before production. Pin Airflow and provider versions in the image so every worker runs identical code. Ask your security team how quickly a critical base image fix must reach workers, and write that interval into the runbook.

Provider and dependency updates

Executor behavior depends on provider packages and Python libraries, so an update can change how tasks are queued or launched. Read the release notes for each executor-related provider before upgrading it. Airflow 2 entered limited maintenance on October 22, 2025, and reached end of life on April 22, 2026. Limited-maintenance releases receive only security and critical fixes, and end-of-life releases receive none. If you still run Airflow 2, plan the move to Airflow 3 now, and use the upgrades and support topic for the full process.

Runbook for adding an executor path

Adding an executor later, such as moving a set of tasks from Celery to Kubernetes, changes scheduling, images and monitoring together. Document which DAGs or queues route to which executor, the steps to enable the new path, the rollback steps and the checks that prove each path works. Airflow 2.10.0 introduced running several executors in one deployment. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so the runbook must state how tasks are assigned to each executor. Test the runbook in a non-production environment first.

Questions teams ask before choosing or building an executor

Is CeleryExecutor or KubernetesExecutor better for a small team?

For most small teams, the deciding factor is the infrastructure they already run and staff. CeleryExecutor sends tasks to long-running worker processes that pull work from a message broker, so you operate the broker, the workers and their scaling. KubernetesExecutor starts a separate pod for each task, which suits teams that already operate a Kubernetes cluster and want per-task isolation. If you have no cluster and no one who runs one, a Celery setup on a small set of virtual machines is often easier to reason about, but confirm broker and worker sizing against your own workload. Either choice needs a production metadata database. The Airflow prerequisites recommend PostgreSQL or MySQL and advise against SQLite in production.

Do hybrid executors still work in Airflow 3?

No. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, according to the Airflow executor documentation. Teams that ran a hybrid setup on Airflow 2 need a different pattern on Airflow 3. Since Airflow 2.10.0, several executors can run in one deployment, which is the usual replacement. Check your existing configuration for hybrid executor class names before any upgrade, because they will not run as configured. Plan this change as part of the Airflow 2 to 3 migration so it does not surface during cutover.

Can one Airflow 3 deployment run more than one executor?

Yes. Since Airflow 2.10.0, several executors can run in one deployment, according to the executor documentation. Airflow ships LocalExecutor, CeleryExecutor, KubernetesExecutor, EdgeExecutor, and AWS Batch and ECS executors, and you can combine them within one deployment. This lets you keep short, lightweight tasks on one executor and heavier or more isolated tasks on another. The cost is a second execution path to build, monitor and debug. Confirm in the executor documentation which combinations your Airflow 3 release supports, then test failure, retry and log collection on each path before you rely on the split in production.

How do I split DAGs between two executors in one deployment?

Assign executors at the task or DAG level, and treat that assignment as part of DAG authoring. Confirm the exact parameter name and the default executor for your release in the executor documentation before writing standards around it. Base the split on three things: resource profile, isolation needs and dependencies. Tasks that need heavy libraries, specific system packages or data locality often belong on the executor whose workers can run them. Make sure each executor's worker image contains the libraries its tasks import. Record the assignment rules in your DAG standards and review them in code review, so a new DAG cannot land on the wrong executor by accident.

What does a self-hosted executor setup need from our internal team?

The work needs people who can operate Python services, a production relational database, and either a message broker or a Kubernetes cluster, depending on the executor. On Airflow 3, the team must run the API server with the airflow api-server command, run the DAG processor as its own component, and move task code to the REST API or Task SDK, because task code cannot use Airflow database sessions or models directly. Plan within the supported ranges in the prerequisites: Python 3.10 to 3.14, PostgreSQL 14 to 18, and Kubernetes 1.30 to 1.35 if you use it. Ongoing work includes patching, provider updates and monitoring.

Can our internal team deploy Airflow executors without a partner?

Yes, if the team has the skills listed above and time to design, build, test and operate the setup. Airflow is licensed under Apache-2.0 and has no paid edition from the project, so the software itself carries no license fee. The effort sits in architecture, deployment, testing and day-two operations. Many teams start with a non-production proof of concept, test scheduler and worker restarts and database failover, and then plan cutover. A partner is useful when the team lacks capacity, needs a fixed delivery scope, or wants an independent review of the design. DeployOpen builds and documents self-hosted platforms inside your own infrastructure, so ownership stays with your team.

What drives the cost and timeline of an executor rollout?

The main drivers are the number of executors, the target infrastructure, the state of existing DAGs and the operating requirements. Each additional executor adds a second execution path to design, test and monitor. An upgrade from Airflow 2 adds DAG review, since the Airflow 3 upgrade guide recommends checking DAG code with Ruff AIR301 and AIR302 rules and replacing SubDAGs with TaskGroups. Air-gapped networks, strict secrets handling, SSO and high availability each add design and verification work. Recurring costs include the compute and storage you pay for directly, plus upgrades and support after go-live. Ask for each driver to be itemized in the quote so you can see what changes the price.

What should we ask for in a fixed-scope executor quote?

A fixed-scope quote should state in writing, before work starts, the executors in scope and the reason for each, the target environments and cloud accounts, and the Airflow and component versions. It should also list the DAG migration volume, the integrations such as secrets, SSO and logging, and the acceptance tests that close each phase. Ask what is excluded, who operates each component after go-live, and whether ongoing support is a separate agreement. Finally, ask which costs are one-time and which recur, including the infrastructure you pay for directly, so the total cost of ownership is visible from the start.

How DeployOpen scopes, builds and hands over executor work

DeployOpen treats executor choice as a decision made against your workloads, your Airflow 3 version and the infrastructure you already run, then records it in a written scope before any build starts. Every step, from the executor comparison to the handover runbook, is documented so your team can run and change the deployment after go-live.

How we scope

We review your task profiles, runtime needs and peak concurrency to shortlist LocalExecutor, CeleryExecutor, KubernetesExecutor or EdgeExecutor.
We confirm which executors your Airflow 3 release supports, since hybrid executors were removed in Airflow 3.0.0.
Engagements are fixed-scope, so the written scope names the executors, queues, infrastructure and deliverables covered before work starts.

How we build

We provision the metadata database, workers or Kubernetes namespaces inside your own cloud accounts, data centers or air-gapped network.
We configure queues or pod templates so each task type routes to the executor it was assigned, and we test that routing end to end.
We check worker isolation, network paths to the database and scheduler behavior when more than one scheduler runs against the same metadata database.

How we hand over

We deliver the architecture record, configuration files, routing rules and a runbook explaining which executor runs which workloads.
We document the failure modes we tested, such as worker loss, queue backlog and pod scheduling delays, with the recovery steps for each.
We walk your engineers through scaling, adding a queue or changing executors so the team can make those changes without us.

How we support

Ongoing support, upgrades and continuous testing are available as retainers once the deployment is live.
Support covers keeping Airflow on a supported release, since Airflow 2 reached end of life on April 22, 2026 and receives no fixes.
Ask any provider to confirm in writing what a retainer covers, including executor changes, provider package updates and incident response hours.