Airflow orchestrates the work your other systems do
Apache Airflow schedules and monitors workflows written as Python DAGs. It is licensed under Apache-2.0 and suits batch and scheduled pipelines: loading a warehouse, running dbt or Spark jobs, calling APIs and coordinating dependent steps. It works best when tasks hand heavy processing to systems built for it while Airflow tracks order, retries and state.
Airflow 3.0 was released in April 2025 with significant architectural changes, and Airflow 2 reached end of life on April 22, 2026, so it no longer receives fixes. New installations should start on Airflow 3, and existing Airflow 2 installations need an upgrade plan.
Airflow 3 components
Each component scales and fails on its own. Give each a resource budget, health checks and findable logs.
API server
Serves the user interface and REST API, and is the path through which running tasks reach Airflow's state. It replaces the Airflow 2 webserver.
Scheduler
Decides which task instances are ready and hands them to the executor. More than one scheduler can run at once for resilience.
DAG processor
Parses DAG files and stores their structure. In Airflow 3 it runs as its own process, so slow top-level DAG code shows up here first.
Triggerer
Runs deferrable tasks that wait on external events without holding a worker slot.
Executor and workers
Run the task code, in local processes, a Celery worker pool or one Kubernetes pod per task.
Metadata database
Holds runs, task state, connections and variables in PostgreSQL or MySQL. Production uses an external database with a pooler such as PgBouncer; the chart's bundled PostgreSQL is not meant for production.
Choose an executor
The executor shapes isolation, startup time and what the platform team runs. Airflow 3 can also run several executors and assign tasks to each.
| Executor | How tasks run | Fits | Plan for |
|---|---|---|---|
| LocalExecutor | Subprocesses on the scheduler host. | Small installations. | One host's capacity, shared with the scheduler. |
| CeleryExecutor | Standing workers pull tasks from a broker such as Redis or RabbitMQ. | Steady volumes of short tasks. | Broker availability, autoscaling (the Helm chart supports KEDA) and shared worker dependencies. |
| KubernetesExecutor | Each task runs in its own pod. | Tasks with different dependencies, resources or isolation needs. | Pod startup per task, image management and quotas. |
| Multiple executors | Tasks are assigned to a named executor. | Mixed workloads that used the hybrid executors removed in Airflow 3. | Clear rules for which DAGs use which executor. |
Deliver DAG code like application code
Copy DAGs into the image and deploy a uniquely tagged image per release. Rollback means redeploying the previous tag, and every component runs the same code and dependencies.
A sidecar pulls DAGs from one repository branch. The Helm chart documentation recommends git-sync with local volumes. Set known hosts when using SSH keys so the sync cannot be redirected.
Airflow reads DAGs from a POSIX-compliant volume that another process populates. You own the sync, versioning and rollback.
Run a DAG import test, linting and unit tests in CI. Ruff's AIR rules flag code that breaks in Airflow 3. Anyone who can merge DAG code runs code as the workers, so protect the repository.
Upgrade from Airflow 2 to Airflow 3
Move to the latest Airflow 2 release first; Airflow 3 requires at least 2.7. Back up the metadata database, consider airflow db clean to shrink it, review configuration with airflow config update, and upgrade the schema with airflow db migrate. Helm chart users rename webserver values to apiServer.
Code changes are the larger effort. Task code can no longer open database sessions or import Airflow models; it uses the Task SDK (airflow.sdk imports) or the REST API through its Python client. SubDAGs and SLAs are removed, the REST API moves from v1 to v2, context variables such as execution_date are gone, and catchup defaults to off. Authentication routes move under /auth, so OAuth redirect URLs registered with your identity provider need updating.
Inventory DAGs, custom operators, plugins and anything reading the metadata database. Run the upgraded stack in parallel against test connections, then pause each DAG in the old environment before enabling it in the new one to avoid duplicate runs.
Identity, secrets and task access
Airflow 3 delegates sign-in and permissions to an auth manager. The default Simple auth manager is intended for development and testing. Production deployments typically use a provider auth manager: FAB, which supports OAuth and LDAP sign-in through Flask AppBuilder configuration, or Keycloak, which hands authentication and authorization to a Keycloak realm. Map identity provider groups to Airflow roles so access follows joiner and leaver processes.
Airflow encrypts connections and variables in the metadata database with a Fernet key, and a secrets backend such as HashiCorp Vault or Azure Key Vault can supply them from an existing secrets service. Set static API and JWT secrets per instance from Kubernetes Secrets or an equivalent, outside committed values files.
Workers run whatever DAG code says, with the network access and credentials they hold. Use separate worker identities per environment, restrict egress to the systems pipelines need, and limit who can trigger DAGs that touch sensitive data.
Availability and recovery
Recovering Airflow means recovering its state, its code and the keys that decrypt its connections.
Metadata database
Back it up with point-in-time recovery, test restores, and run the cleanup job so run history stays bounded.
Fernet key and secrets
Store the Fernet key with the backup plan. Without it, encrypted connections in a restored database cannot be read.
Schedulers and API servers
Run more than one of each where downtime matters. Multiple schedulers coordinate through row locking in the database.
Task logs
Configure remote logging. Depending on the executor, task logs may be lost once a pod or worker is gone.
Airflow operating ownership
The platform team runs Airflow; DAG authors own their pipelines and the data they produce.
Platform
- Monitor scheduler heartbeat, DAG parse time, queued tasks and database connections.
- Upgrade Airflow, providers and the base image on a schedule.
- Manage executor capacity and worker images.
DAG authors
- Keep top-level DAG code light so parsing stays fast.
- Set retries, timeouts and alerts per pipeline.
- Pin provider versions and test in a lower environment.
Security
- Review role mappings, connections and secrets access.
- Control who can merge DAG code and trigger sensitive DAGs.
- Rotate the Fernet key and service credentials by procedure.
Data owners
- Define freshness expectations for pipeline outputs.
- Approve backfills that rewrite history.
- Serve as the escalation point when a pipeline fails.
Airflow deployment questions
Can we keep running Airflow 2?
Airflow 2 reached end of life on April 22, 2026 and receives no further fixes, including security fixes. Running it means accepting unpatched issues in Airflow and its dependencies.
Do we need Kubernetes?
No. Airflow runs on virtual machines with LocalExecutor or CeleryExecutor, and the official Helm chart makes Kubernetes deployments repeatable. The Docker Compose quick start is meant for learning and exploration.
Our custom operators query the metadata database. What changes?
Airflow 3 task code cannot use Airflow's database sessions or models. Rework those operators to use the Task SDK or the REST API. A database hook against the metadata database is documented only as a temporary workaround.
Can DAGs move between self-hosted and managed Airflow?
DAGs built on standard operators are portable, but Airflow and provider versions, plugins, secrets backends and executor behavior can differ. Test representative DAGs in the target first.
Scope and handover
DeployOpen plans the architecture with you, deploys Airflow 3 with the official Helm chart or on virtual machines, sets up the metadata database, executor, DAG delivery, secrets backend and SSO through a production auth manager, configures logging and monitoring, upgrades Airflow 2 installations, documents runbooks and provides ongoing support if you want it. Scope depends on environments, DAG count, custom operators and plugins, executor choice, identity integration and how much Airflow 2 code needs rework.
Handover includes the component layout, executor and capacity settings, DAG delivery pipeline, auth manager and role mapping, connection and secrets inventory, metadata database restore result, Fernet key custody, alert routes and the version plan for Airflow and providers.
