Is the Airflow Docker Compose setup fit for production? Usually not, and here is why
For most teams, the official Docker Compose setup is not fit for production. It is a good way to learn Airflow and try DAGs on a laptop or a test host, but the Airflow documentation describes it as meant for learning and exploration. It does not provide the security guarantees a production deployment needs, and the docs recommend Kubernetes with the official Helm chart for production. Before you decide anything, read the Running Airflow in Docker guide to see what the file sets up and what it assumes about your environment.
Airflow is an Apache Software Foundation project released under the Apache License 2.0, and the project offers no paid edition. The gap between the Compose file and production is therefore not a missing product tier you can buy your way past. It is operational work: choosing a supported metadata database, sizing memory (the prerequisites recommend at least 4 GB of RAM, and the Compose file needs at least 4 GB for Docker itself), separating the API server, scheduler, DAG processor and workers into components you can restart and monitor, and deciding how secrets, logs and DAG code reach each of them. Confirm which of these the Compose file covers in your version before you assume any of them.
The useful question is what your team will own once the file moves into production. Someone must patch Airflow and its providers on a schedule, since Airflow 2 reached end of life on April 22, 2026 and Airflow 3 is the line to plan on. Someone must also back up the metadata database, restore it in a tested way, and handle authentication and access for DAG authors and operators. If your team cannot staff those tasks, a fixed-scope deployment on Kubernetes or a virtual machine layout is the realistic next step, and a single-host Compose install should be limited to development or a short-lived proof of concept.
Which deployment fits which stage
This table compares four ways to run Airflow 3: the official Compose file, a hardened single-host Compose install, separate services on virtual machines, and the official Helm chart on Kubernetes. Read each row to see where the metadata database sits, who carries patching and backups, and when the option is the wrong choice.
| Deployment | Fit and typical stage | Metadata database state | Patching and backup owner | When it is the wrong choice |
|---|---|---|---|---|
| Official Compose file | Learning and exploring Airflow on a laptop or test host, with Docker needing at least 4 GB of memory | Bundled for testing; not a production-grade database setup | Your team, with no guarantees from the file itself | Any production schedule, shared team use or regulated data |
| Hardened single-host Compose | Small internal pipelines on one server where a short outage is tolerable | External PostgreSQL or MySQL is required; SQLite is out for production | Your team owns host, image, backup and restore runbooks | Multiple teams, high availability needs or growing run history |
| Virtual machines with separate services | Stable pipelines on fixed hosts, with API server, scheduler, DAG processor and triggerer as separate services | External PostgreSQL or MySQL, sized and backed up separately | Your operations team owns OS, service units, upgrades and database backups | Frequent scaling needs or when you want declarative rollback of the whole stack |
| Official Helm chart on Kubernetes | Production platforms with several teams, scheduled upgrades and cluster-level controls | External database recommended, with planned connection limits and growth | Your platform team owns cluster, chart release, images and recovery tests | Organizations without Kubernetes skills or capacity to run a cluster |
| Minimum memory for Docker | At least 4 GB of memory for Docker, per the Airflow Docker guide | Not applicable | Not applicable | Not applicable |
| Minimum host RAM in prerequisites | At least 4 GB of RAM recommended for Airflow, per the prerequisites | Not applicable | Not applicable | Not applicable |
What the Compose file leaves out for production
The official Compose file is built to let a team start Airflow quickly and explore it, and it does not supply the security guarantees a production system needs. Treat it as a working reference for how the components fit together, not as a deployment you can operate for scheduled business pipelines. Everything the file leaves open becomes your responsibility: secrets management, network exposure, TLS termination, backups, upgrades and monitoring. Each of these needs a named owner and a decision before any production DAG runs.
The most immediate gap is the metadata database. The default SQLite backend is meant for testing only, and running it in production can lead to data loss. The production choice is PostgreSQL or MySQL, with a version from the supported list in the Airflow prerequisites. Plan the database as its own asset with its own backups and restore test, since the scheduler, API server and workers all depend on it. Also confirm the Fernet key and other encryption keys are stored outside the file and are backed up, because losing them can make stored connections unreadable.
Before relying on the file, compare its current contents against the Compose how-to page in the Airflow documentation, since images, default credentials and service layout can change between releases. Ask your platform team to list every service, port, volume and environment variable it sets, then mark each one as keep, replace or remove. Confirm the memory your host provides against the stated minimum for Docker, and plan for the host to be patched, restarted and monitored like any other server. If the list of replacements grows long, that is the signal that a production container or Kubernetes design is the better starting point.
What a production container deployment adds
A production Airflow container deployment splits the single Compose stack into separately managed components, each with its own scaling, restart and access rules. Together they replace a demo database and a single host with infrastructure that can be patched, backed up and recovered without taking the whole pipeline platform down.
External metadata database
Compose runs PostgreSQL in a container on the same host, which is fine for exploration but ties your run history and connections to one disk. Production needs a PostgreSQL or MySQL database you can back up, restore and upgrade on its own schedule. SQLite is not an option, and MariaDB is not supported. Confirm the version against the prerequisites before you choose a database.
Separate API server
Airflow 3 replaces the webserver with an API server started by airflow api-server. It serves the UI and the REST API, and it needs its own process, network exposure and TLS termination. Decide how users reach it, which ingress or load balancer fronts it, and how its replicas are restarted when a deployment rolls.
Scheduler and DAG processor
The scheduler decides which tasks run, and the DAG processor parses DAG files into the metadata database. In Airflow 3 the DAG processor must run as its own component, so a slow or broken DAG file no longer hides behind the scheduler process. Plan replicas and resource limits for both, and confirm how many schedulers your database and version support at the same time.
Triggerer service
The triggerer runs deferrable operators that wait on external events without holding a worker slot. Compose often starts it, but a production setup needs it supervised, sized and restarted like the other services. Without it, deferred tasks stall quietly, so add a check on its health to your monitoring plan.
Chosen executor and workers
The executor decides where tasks run: LocalExecutor, CeleryExecutor, KubernetesExecutor and others each bring different worker, queue and isolation needs. Compose hides this choice behind defaults. Production requires you to pick one, size its workers, and document how tasks reach them and how failed tasks are retried.
Pinned image and release
A production deployment runs a specific Airflow image or Helm chart version, not a floating tag, so upgrades happen on purpose. Record the Airflow version, provider versions and Python version together. Compose images are a starting point, and the production guide recommends container images for Kubernetes deployments. Check the version life cycle before you pin.
How to move from Compose to a production-ready setup
This sequence takes a learning-grade Compose install to a production layout by checking version support first, then replacing SQLite, file-based DAGs and inline secrets before any cutover.
Confirm the Airflow version and support status
Check the Airflow version your Compose file pulls against the version life cycle table. Airflow 3 is the current major line, with 3.3.2 as the current patch release. Airflow 2 reached end of life on April 22, 2026, so an install still on 2.x gets no fixes and needs an upgrade plan before anything else. If you are on Airflow 2.7 or later, the upgrade guide recommends moving to the latest 2.x first and running Ruff AIR301 and AIR302 checks on your DAG code. Done when you have a written version target and its support end date. Ask your team to confirm the exact image tag currently in use.
Choose a supported metadata database
Pick PostgreSQL or MySQL for the metadata database and a version from the prerequisites. The Airflow 3.3.2 prerequisites list PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases. MariaDB is not supported, and SQLite should not be used in production. Confirm whether the database will be self-run or managed, and plan how it is backed up and sized for run history. Check: connect with the production connection string and confirm the reported backend is not SQLite and the version is on the supported list.
Select the deployment target
Decide where Airflow will run: Kubernetes with the official Helm chart, which the production guide recommends, or virtual machines for a single-host or small footprint. Your platform team should confirm cluster availability, image registry access and the network path to the metadata database. Record the choice along with its supported versions, for example Kubernetes 1.30 to 1.35 for Airflow 3.3.2. Done when the target, its owners and its prerequisites are written down and approved.
Move DAG delivery out of the Compose file
Stop mounting DAG folders from the developer's host. Choose a delivery method that suits the target, such as baked images or git-sync from an internal Git repository, and define how a release is promoted and rolled back. Confirm that a DAG change in version control reaches the scheduler and workers without manual copying. Check: trigger a test DAG change in a non-production environment and confirm the new code is picked up and the previous version can be restored.
Replace inline secrets with managed secrets
Remove passwords, the Fernet key and the API secret key from environment entries in the Compose file. Load them from your secrets store or Kubernetes Secrets, and use the supported secret variables so values stay out of images and version control. Confirm who can read each secret and how rotation will work. Check: search the repository and image history for the old values and confirm none remain, and that a component starts only when its secret is supplied.
Separate the Airflow components
In Airflow 3, the webserver is replaced by the API server, and the DAG processor runs as its own component. Deploy the API server, scheduler, DAG processor, triggerer and workers as separate services or pods, with the chosen executor set in configuration. Several schedulers can run at once against the database using row-level locks, so decide the scheduler count with your database team. Check: restart the scheduler and API server and confirm both return to healthy state and that the DAG processor keeps parsing DAGs.
Verify health, backups and restore before cutover
Run a test that proves recovery, not only startup. Take a backup of the metadata database, restore it into a separate environment and confirm the restored Airflow reads its DAG history and connections. Confirm logs persist beyond container lifetime and that health checks reflect each component's state. Your team should sign off on the restore result and the rollback path before any scheduled pipeline moves. Done when a restore has been tested and the cutover plan lists who decides to proceed.
When a single-host Compose install is still acceptable
A single-host Compose install is acceptable only in a few narrow cases: a developer sandbox on a personal machine, a proof of concept that uses no regulated or production data, or a short evaluation where the team is learning how Airflow 3 components fit together. In these cases the goal is exploration, not service delivery, so the lack of high availability, hardened secrets handling and tested recovery is an accepted trade-off rather than an oversight. The team should still confirm that the host is not reachable from networks it does not need to reach and that any credentials used are throwaway.
The acceptability ends as soon as the environment changes. Shared access by more than one engineer, production schedules that feed downstream systems, sensitive or regulated data in DAGs, connections or task logs, or any requirement to restore pipeline history all move the install out of scope for Compose. Each of these brings controls the single-host file does not provide, such as an external metadata database following the prerequisites, access control, backups and a reviewed upgrade path. A useful check is to ask whether anyone would be disrupted or exposed if the host failed or was compromised tomorrow.
Your organization must decide its own risk tolerance and the controls it requires before accepting any Compose install, and that decision should be written down with an owner and a review date. Confirm with your security and compliance leads which data classes are permitted, how long the sandbox may run, and when it must be retired or rebuilt on the production path. If the answer to any of these questions is unclear, treat the install as production and plan the move early, because retrofitting a schedule, its data and its credentials later is far more costly.
Security and operational gaps to close first
The Compose file is built to get Airflow running quickly on one machine, so most of the gaps below are defaults that a production team must replace on purpose. Each card names one gap, why it matters for this decision, and what to check in your own environment before you put pipelines or regulated data on the stack.
Default credentials and secret handling
The Compose file ships with sample account and key values that are meant only for exploration. Production needs its own admin account, a unique Fernet key for encrypting connection passwords, and a secret key for the API, all held in a secrets store rather than in the Compose file or the .env file on disk. Confirm who can read those files on the host and how the keys are rotated. The Simple auth manager is the default in Airflow 3, and you choose a different auth manager through the auth_manager setting in configuration, so plan that choice before go-live.
API server network exposure
In Airflow 3 the user interface and REST API are served by the API server, which the Compose file publishes on a host port. Publishing it directly to a corporate network or the internet without a reverse proxy, TLS termination and an access policy exposes sign-in and the API to every host that can reach it. Check which ports are bound to which interfaces, whether traffic is encrypted end to end, and whether the database and Redis ports are reachable from anywhere other than the Airflow containers.
Database access and backups
The metadata database holds DAG run history, connections and variables, so its loss or exposure affects every pipeline. The Compose file starts a PostgreSQL container with a volume on the host, which has no replication, no tested point-in-time recovery and no separate backup job. Confirm where the volume lives, who holds the database credentials, how often backups run, and whether a restore has ever been tested on a fresh host. The production guide recommends PostgreSQL or MySQL, and SQLite is for testing only because it can lose data.
Log access and secret exposure
Task logs can include values that a DAG printed, including tokens, customer identifiers or query results, and the Compose setup keeps them on the local filesystem of the host. Anyone with read access to that volume, or to the log endpoint on the API server, can read them. Decide which roles may view logs, whether logs are shipped to a store with its own access controls, and how long they are kept. Treat log access as a data access decision, not a convenience setting.
Upgrade ownership against end-of-life dates
Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it receives no fixes. A Compose install pinned to a 2.x image therefore runs unpatched unless someone owns the upgrade. Name the owner, the review cadence for new patch releases, and the path to Airflow 3, which requires being on Airflow 2.7 or later first. Also confirm the DAG code has been checked with the Ruff AIR301 and AIR302 rules, since the upgrade guide lists changes that break DAGs.
Scheduler health monitoring
Compose gives you container status, but not a signal that the scheduler is stuck, parsing DAGs slowly or no longer creating runs. A stopped or hung scheduler can leave pipelines silently late. Set up an alert on scheduler heartbeat and on DAG parse health, and confirm that someone is paged when it fails. If you later run more than one scheduler, Airflow uses database row-level locks to coordinate them, so the database load and lock behavior should be part of the monitoring plan.
Questions teams ask before moving off Compose
What does it cost to make an Airflow Compose setup production ready?
Cost depends on the scope you agree in writing, and no fixed figure applies to every team. The main cost components are the people who design the target architecture, build and harden the deployment, connect identity and secrets, test recovery, and hand over documentation. Software is free, since Apache Airflow is licensed under the Apache License 2.0 and the project has no paid edition. Recurring costs sit in your infrastructure: compute, the metadata database, storage for logs, and staff time for upgrades. Cost rises with the number of environments, the number of integrations, the identity provider you connect, and how sensitive the data flowing through DAGs is. Ask any provider for a quote that lists each of these items separately, with a clear split between one-time work and ongoing support.
How long does a move from Compose to production take?
Duration depends on how much of the target platform is already in place. Teams that already run Kubernetes, PostgreSQL and an identity provider usually move faster than teams building those layers from scratch. The factors that set the timeline are the number of environments, whether DAG code needs a delivery pipeline, how many secrets backends and data sources must be reached, change approval cycles, and how many Airflow versions must be checked before deployment. Ask for a schedule with milestones tied to working environments and signed-off test results, not calendar guesses. Duration figures should come from the written scope, not from general estimates.
What does our team need to provide for a production Airflow project?
The work needs access and decisions from your side before the build starts. Typical inputs include:
- Access to the cloud account, data center or virtual machines where Airflow will run, with network paths to data sources.
- A metadata database decision. Airflow supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, while SQLite should not be used in production and MariaDB is not supported.
- Identity provider details, such as the protocol, client configuration and group mappings for role assignment.
- A named owner for DAG code, secrets, and the change approval path that applies to production.
Agree on these early, because missing inputs are the most common reason a fixed schedule slips.
Can our internal platform team set up production Airflow without a partner?
Yes, many teams do this in-house, provided they have people who already run Linux, containers or Kubernetes, a relational database, and secrets management. The hard parts are rarely the first install. They are the scheduler high availability design, log retention, backups, the Airflow 3 architecture with a separate API server and DAG processor, and upgrade discipline as Airflow 2 reaches end of life on April 22, 2026. Ask whether your team can cover those areas, who is on call, and how many hours per month can go to upkeep. A partner makes sense when those skills or hours are missing, or when you want the work fixed to a written scope.
Does Apache Airflow have a paid enterprise edition?
No. Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0, and the project itself offers no paid edition. Any cost you see comes from infrastructure, staff time, or services such as implementation, support and training from a company you choose to hire. When comparing quotes, check whether a vendor is charging for software, which the project does not require, or for the work and operations around it. Confirm license terms for any provider packages or add-ons your team plans to use.
Is the official Helm chart the right next step after Compose?
For most teams that already run Kubernetes, yes. The Airflow community maintains an official Helm chart that uses the official Docker image, and the production guide recommends container images for Kubernetes deployments. The chart still needs decisions on how DAG code is delivered, how secrets are supplied, which database is external, and how migrations run during upgrades. If you do not run Kubernetes, a virtual machine deployment with separate API server, scheduler, DAG processor and triggerer services is an alternative. Confirm the Kubernetes version you run against the supported range, which for Airflow 3.3.2 is 1.30 to 1.35.
Which Airflow version should a new production deployment start on?
Start on Airflow 3, since Airflow 3 was first released on April 22, 2025 and its current patch release is 3.3.2. Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. If you have existing DAGs written for Airflow 2, plan the migration before cutover. The upgrade guide requires Airflow 2.7 or later before upgrading, recommends moving to the latest 2.x first, and suggests checking DAG code with Ruff AIR301 and AIR302 rules. Confirm your current version and DAG compatibility with the upgrade guide before fixing the scope.
How DeployOpen scopes a move from Compose to production
A move from the Compose file to a production Airflow starts with a written scope, not a build. DeployOpen plans and implements self-hosted Airflow inside your own cloud accounts, data centers or air-gapped networks, and the scope document names the target components before any work begins: the API server, scheduler, DAG processor and triggerer as separate services, the executor choice (such as LocalExecutor, CeleryExecutor or KubernetesExecutor), the external metadata database, the log and DAG delivery paths, and the secrets backend. Most engagements are fixed-scope, so the deliverables, the acceptance checks and the timeline are agreed in writing before work starts. Ask for the same list in any quote you receive, whether from DeployOpen or another firm.
Once the scope is agreed, the work covers the controls that the Compose file leaves out. That typically means a PostgreSQL or MySQL metadata database instead of SQLite, secrets moved out of the compose file and into your secrets store, authentication configured for your identity provider, backups and restore tests for the metadata database, and a DAG delivery method your team can roll back. DeployOpen hardens the result, documents how it was built and how each component is run, and hands over the runbooks your operators will use. Audit evidence is produced as part of that documentation, but DeployOpen prepares systems, controls and evidence for compliance assessments only. It does not issue audit reports, attestations or certifications, and independent auditors and certification bodies make those determinations. DeployOpen is also not an official Apache Airflow partner, and the Airflow project itself has no paid edition.
After go-live, ongoing support, upgrades and continuous testing are available as retainers, which is useful because Airflow changes on a fixed cadence. Confirm the Airflow version you plan to run against the version life cycle before you sign, since limited-maintenance and end-of-life versions receive fewer or no fixes. Ask whether the retainer covers patch releases, provider updates and the Helm chart or image rebuilds, and what happens when a major upgrade is needed.
