Why teams run Apache Airflow on virtual machines without Kubernetes
A team that controls its virtual machines but has no Kubernetes platform yet can still run scheduled pipelines in production. Airflow 3 runs on VMs as several separate services, each started by its own command: the API server (airflow api-server), the scheduler, the DAG processor (airflow dag-processor) and the triggerer, with an external PostgreSQL database holding the metadata. The setup is a set of cooperating services, not one process, and that is the first thing to plan for.
Apache Airflow is licensed under the Apache License 2.0 and has no paid edition from the project itself. The software costs nothing to obtain, so the real cost is the team's own infrastructure: VM capacity, the PostgreSQL service, backups, monitoring, patching and the engineering time to operate all of it. Ask whether the organization has people who can own those services around the clock, because the platform will need restarts, upgrades and capacity changes long after the first pipeline runs.
VMs are a sound fit when the team already runs and patches Linux hosts, wants few moving parts, and has a modest number of DAGs that do not need elastic scaling per task. They become a poor fit when the workload needs burst capacity, isolated per-task environments or many teams sharing one platform. Confirm the Python and PostgreSQL versions you intend to use against the Airflow prerequisites before you choose hosts, and plan the move to Kubernetes now if you expect those growth pressures.
LocalExecutor or CeleryExecutor: which fits a VM fleet
When the workload fits on one host, LocalExecutor is usually the simpler choice. With LocalExecutor, the scheduler starts each task as a local process on the same machine that runs the scheduler. There is no message broker to install, no worker processes to register, and no second host to keep in step with the DAG code. The trade-off is that task capacity is capped by that one VM's CPU, memory and disk, and a host failure stops both scheduling and execution. Size the VM against the peak number of tasks you expect to run at once, not the average, and remember that each task process and the components around it draw on the same memory. Confirm the current LocalExecutor behavior and its concurrency settings in the executor documentation before you fix the VM size.
- Process model: Tasks run as child processes of the scheduler, so a runaway task can starve the scheduler itself.
- Single point of failure: Losing the host pauses scheduling and execution until it is restored.
- Fewer moving parts: No broker or worker registration to monitor or secure.
- Vertical scaling: Capacity grows by resizing the VM, which is simpler but has a ceiling.
Before you commit, check that a single host can meet your recovery expectations and that the peak task count still leaves headroom for the scheduler and API server.
When one host cannot carry the load, or when you want task execution separated from scheduling, CeleryExecutor distributes tasks to worker hosts through a message broker. The scheduler queues task instructions on the broker, and Celery worker processes on each worker VM pick them up and run them. Adding workers adds capacity, and a single worker failure does not stop the rest of the fleet. The cost is operational: you run a broker, the workers need the same DAG code and dependencies, and every worker must reach the metadata database and the log serving ports. Ask your team which broker it already operates and how it will be patched, since the broker becomes a critical dependency. Confirm the supported broker choices and worker configuration in the executor documentation.
- Broker dependency: The broker must be highly available and sized for queued task volume.
- Uniform workers: Every worker needs matching Python versions, providers and DAG code, or tasks fail unpredictably.
- Network paths: Workers need access to the metadata database, the broker and the log serving ports used by the API server.
- Worker sizing: Size hosts by task memory and CPU, then add workers for concurrency rather than enlarging a single host.
Plan a rollout order for worker upgrades so that a version mismatch never runs tasks against an older dependency set.
Airflow ships several other executors, including KubernetesExecutor, EdgeExecutor, and the AWS Batch and ECS executors. Since Airflow 2.10.0, more than one executor can run in the same deployment, so a VM fleet can route some tasks to a local or Celery pool and others elsewhere. Be careful with the hybrid options: CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so do not design a new VM deployment around them or plan an upgrade that depends on them. Mixing executors adds routing rules that your DAG authors and operators must understand, and each executor has its own dependencies and failure modes. Treat a mixed setup as a deliberate design that needs its own test plan, not a convenience added later.
- Multiple executors: Supported since Airflow 2.10.0, but each one adds routing and monitoring work.
- Removed hybrids: CeleryKubernetesExecutor and LocalKubernetesExecutor are not supported from Airflow 3.0.0.
- Executor-specific dependencies: Each option brings its own infrastructure and credentials.
- Routing clarity: Document which queues or pools map to which executor so authors can predict where a task runs.
If you expect to add Kubernetes later, confirm the migration path in the executor documentation before choosing the VM executor.
The executor choice should be validated against a few questions that depend on your environment rather than on general guidance. Start with the expected number of concurrent tasks at peak, the memory each task uses, and how long tasks run. Next, decide which broker your organization can operate and support under its change process, because a CeleryExecutor deployment cannot run without one. Then confirm how many hosts your operations team can patch, monitor and restore within your recovery expectations. Finally, check the executor documentation for the current settings that govern parallelism on each executor, since those names and defaults change between releases.
- Peak concurrency: Measure the highest number of simultaneous tasks, not the typical number.
- Broker ownership: Name the team that runs the broker, its patching cycle and its failover design.
- Host operations: Confirm the number of hosts your team can maintain with the same image and dependency set.
- Release currency: Check the executor documentation for the Airflow 3 release you plan to run.
Write the answers into the design record before sizing hosts, because the executor decision determines the rest of the VM architecture.
Configuration options for a VM deployment and when each fits
This table compares the executor, metadata database and log storage choices for Airflow on virtual machines. Read each row against your expected task volume, how many hosts you can run, and how much operational work your team will accept. The main takeaway is that LocalExecutor keeps the moving parts few, while CeleryExecutor adds a broker and workers that you must size, secure and monitor.
| Option | What it does | When it fits | Trade-off |
|---|---|---|---|
| LocalExecutor | Runs task processes on the same host as the scheduler, using local subprocesses. | A single scheduler host with moderate concurrency and no need to spread tasks across machines. | Capacity is bounded by one host, and a host failure stops task execution until recovery. |
| CeleryExecutor | Sends tasks through a message broker to separate worker hosts that pull and run them. | Workloads that outgrow one host and need horizontal worker capacity on VMs you control. | Adds a broker, worker fleet, shared DAG delivery and log serving to secure and monitor. |
| Mixed executors | Lets more than one executor run in one deployment, supported since Airflow 2.10.0 per the executor documentation. | Teams that want one executor for most tasks and a separate one for specific workloads. | Confirm the exact configuration for your Airflow version before relying on it, as routing adds complexity. |
| Hybrid Kubernetes executors | Not available. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor were removed as of Airflow 3.0.0. | Not applicable to a new Airflow 3 VM deployment. | Plan any migration from Airflow 2 hybrids separately, using the Airflow 3 upgrade guidance. |
| PostgreSQL metadata database | Stores DAG runs, task instances and scheduler state in an external database. | Production VM installs, with supported versions 14 to 18 for Airflow 3.3.2. | You must run backups, high availability, connection limits and cleanup of run history. |
| MySQL metadata database | Offers an alternative supported backend in the prerequisites for Airflow 3.3.2 (MySQL 8.0, 8.4 and Innovation releases). | Organizations with an existing, supported MySQL platform and standards for it. | Confirm the supported version list for your release. MariaDB is not supported, and SQLite is for testing only. |
| Local log files with serving ports | Writes task logs on each worker host and serves them live on port 8793 for workers and 8794 for the triggerer. | Small installs where log access from the API server to workers is simple to allow. | Requires firewall rules, a shared secret key and clear rules on who may read logs that can hold secrets. |
| Remote task logging | Ships task logs to external storage, configured through remote logging settings in your Airflow version. | Environments that need central retention or want logs to survive host replacement. | Requires storage access, credentials and confirmation of the handler settings for your version. |
Setup sequence for Airflow 3 on VMs
This sequence takes a set of virtual machines from an empty state to a working Airflow 3 installation with an external PostgreSQL metadata database, separate services and a test DAG that has run end to end. Each step closes with a check so the next one starts from a known state.
Provision the metadata database outside the Airflow hosts
Your database or platform team creates a PostgreSQL instance on a version within the supported range for Airflow 3.3.2, which is PostgreSQL 14 to 18, on a host or managed service separate from the Airflow machines. Create a dedicated database and role for Airflow, restrict network access to the Airflow hosts only, and confirm that backups are enabled. Do not use SQLite, which the production guide says is for testing only, and do not use MariaDB, which is not supported. Done means a login from each Airflow host succeeds with the new role and the database accepts connections over the network path you intend to use.
Prepare hosts and install Python with Airflow
Provision each Airflow host with a supported Python version from 3.10 to 3.14, and confirm at least 4 GB of RAM is available, which the prerequisites recommend. Create a virtual environment owned by a non-root service account, then install Airflow 3.3.2 together with the providers you need, using the constraint file published for that Airflow version and your Python version. Keep the constraint URL in version control so every host installs the same set. Done means
airflow versionreturns 3.3.2 and the import of your providers succeeds on every host.Configure the metadata connection and shared secrets
Set
sql_alchemy_connin the Airflow configuration to point to the PostgreSQL role created earlier, and place the value in a file readable only by the service account. Generate a[api] secret_keyand set the same value on the API server, scheduler, DAG processor, triggerer and every worker, because the API server and workers use it to authenticate log requests. Store the Fernet key and the secret key in your secrets tool rather than in shared configuration files. Done meansairflow db checkconnects to the metadata database from each host.Initialize the metadata database and create an admin user
Run the database migration once from a single host, using
airflow db migrate, so the schema is created before any service starts. Create the first administrative user through the auth manager configured for your environment. Airflow 3 uses the Simple auth manager by default, so confirm whether your organization needs the FAB auth manager or another provider before you create accounts that must later move to SSO. Done means the migration completes without errors and a query on the metadata database shows the Airflow tables.Start each service under its own process supervisor
Run the API server with
airflow api-server, the scheduler withairflow scheduler, the DAG processor withairflow dag-processorand the triggerer withairflow triggerer, each as a separate unit managed by systemd or an equivalent supervisor. Airflow 3 requires the DAG processor to run as its own component, and the webserver is no longer used. Run the scheduler on more than one host only if you have confirmed the database capacity for it, since Airflow supports multiple schedulers through row-level locks. Done means each unit reports active, and the API server answers its health endpoint on the host it runs on.Open the log serving ports with firewall rules
Set
worker_log_server_portto 8793 on worker hosts and confirm the triggerer serves logs on 8794, then allow inbound traffic only from the API server and other Airflow hosts on those ports. Keep the ports closed to user networks, since log content can include sensitive values. Confirm the API server can fetch a task log from a worker through the open port. Done means a log request from the API server host to a worker returns content, and a request from an unrelated network segment is refused.Deploy and run a test DAG end to end
Place a small test DAG with one or two tasks in the DAG folder that the DAG processor reads, and confirm it appears in the API server. Trigger a run manually, then watch the task move through queued, running and success states on the executor you selected, LocalExecutor or CeleryExecutor. Open the task log from the UI to confirm the log is served through the worker port. Done means the run completes successfully, the log displays in the UI, and the scheduler has recorded the run in the metadata database.
Security and operations to plan before go-live
A VM deployment has no platform layer doing these jobs for you, so access, secrets, database choice, scheduler redundancy, service supervision and backups all become your responsibility. The cards below cover the decisions to make and the checks to run before production traffic arrives.
Restrict access to log ports
Workers serve live task logs on port 8793 and the triggerer on port 8794. Only the API server and the components that fetch those logs should reach these ports, so allow them at the host firewall or security group and deny everything else. Logs can contain secrets and business data, so treat this network path as sensitive. Confirm the exact traffic flows in your own topology before opening anything.
Protect the shared secret key
The API server and workers use a shared [api] secret_key to authenticate log requests. Every component that serves or reads logs must hold the same value, so store it in a secrets file with restricted permissions rather than in a shared configuration file. Plan how you will rotate it, since changing it requires restarting every affected component. Confirm the rotation steps against the version you deploy.
Keep SQLite out of production
The default SQLite backend is intended for testing and can lose data in production. Use PostgreSQL or MySQL as the metadata database from the first install, since moving run history later is harder. MariaDB is not supported as the metadata database. Check the supported database versions in the prerequisites for your Airflow release before provisioning.
Run more than one scheduler
Airflow supports several schedulers at once, coordinated through database row-level locks, which improves performance and resiliency. On VMs this means running a second scheduler host so scheduling continues if one fails. Test failover in staging, confirm that both hosts share the same DAG files and configuration, and watch for lock contention in the metadata database.
Supervise services with systemd
The API server, scheduler, DAG processor, triggerer and workers are separate processes, and none of them restarts itself reliably after a host reboot or crash. Define a systemd unit for each component with restart policies, dedicated service accounts, and environment files that hold configuration. Confirm the unit definitions and restart behavior with your operations team before go-live.
Back up the metadata database
The metadata database holds DAG run history, task state, connections and variables, so losing it means losing operational state. Schedule regular backups of the PostgreSQL or MySQL instance, store copies outside the Airflow hosts, and test restores on a separate system. Also confirm how the Fernet key is backed up, since encrypted connection values cannot be read without it. Agree recovery time and recovery point targets with the business owners.
When a VM setup should move to Kubernetes
A VM setup usually reaches its limits when the platform stops being one team's tool and becomes a shared service. Watch for many teams deploying DAGs into the same scheduler and metadata database, requests for isolated environments per team, and worker capacity that changes with load instead of being sized by hand. Another signal is upgrade friction: when the API server, scheduler, DAG processor, triggerer and workers each need coordinated restarts on every Airflow patch, and the Python environments on each host drift apart, the operations cost of staying on VMs grows faster than the workload. If two or three of these signals are present, the VM model is no longer the cheapest way to run Airflow, and a move deserves its own evaluation.
Kubernetes is the path the project itself points to for scale. The production deployment guide recommends container images for Kubernetes deployments, and the community maintains an official Helm chart built on the official Docker image. Before choosing that route, confirm that your target clusters run a Kubernetes version the Airflow 3.3.2 prerequisites cover, which is 1.30 to 1.35, and that your platform team can operate the cluster itself, including upgrades, node pools and ingress for the API server. Ask whether your organization already runs a supported cluster with an identity integration and storage classes you can reuse, because a new cluster adds a separate security and patching scope to the Airflow project.
A move from VMs to Kubernetes is a re-platform with its own plan, not a lift and shift. Log serving changes, since task logs on worker ports such as 8793 no longer map to fixed hosts, so you need a remote logging decision before cutover. DAG delivery changes from copying files to a host into baking images or using git-sync, and the executor choice may change as well. Database connections need review, because more pods means more connections to PostgreSQL. Plan a parallel run, migrate one set of DAGs first, and define rollback for the metadata database, which is the hardest part to reverse. The Airflow 3 upgrade guide should be read against your existing DAG code before any of this work begins.
Limits and what to confirm in your environment
A VM-based Airflow build is only as current as the versions underneath it. The cards below cover the constraints that most often change a design late, from database support to the auth manager and the release calendar. Confirm each one against the current documentation before you fix versions and architecture.
MariaDB is not a supported metadata database
Airflow's prerequisites rule out MariaDB for the metadata database, and SQLite should not be used in production. If your standard database platform is MariaDB, decide early whether to run PostgreSQL or MySQL for Airflow only. Confirm the current list of supported database versions against the prerequisites page before you provision the database host.
Python and PostgreSQL versions must match the release
The Python interpreter, PostgreSQL server and Airflow release must be chosen together, because each Airflow line lists its own supported matrix. Pick the Airflow release first, then pin Python and the database to versions it lists. Check the matrix again before any minor upgrade, since the supported ranges move between releases.
Executor broker and worker settings
LocalExecutor needs no broker, while CeleryExecutor adds a message broker, result backend and worker processes, each with its own availability and security settings. Several executors can run in one deployment, but hybrid Celery and Kubernetes executors are gone in Airflow 3. Confirm the executor choice and its broker configuration in the executor documentation before you size the hosts.
Auth manager configured for SSO
Airflow 3 uses the Simple auth manager by default, and only one auth manager can be active at a time, set by the auth_manager option. The Simple manager is not an identity integration, so SSO needs a deliberate choice of FAB, Keycloak or AWS auth manager and its own configuration. Confirm which provider your organization will use and how roles map to Airflow permissions before go-live.
Support dates set the build lifetime
Airflow 2 reached end of life on April 22, 2026, and a release in limited maintenance gets only security and critical fixes. A VM build is long-lived, so the chosen Airflow line determines how long patches are available. Check the version life cycle table at the start of the project and schedule upgrades before a line leaves maintenance.
Questions teams ask before running Airflow on VMs
What does a VM-based Airflow build cost, and which parts are one-time versus recurring?
The build is a one-time cost for design, installation, configuration, hardening, documentation and handover, and it is priced from the written scope. Recurring costs sit mostly outside any consulting fee: virtual machine or hardware capacity, PostgreSQL licensing or managed database fees if you use one, backups, monitoring, and staff time for patching. Each Airflow patch release also needs regression testing of your DAGs. Costs rise with the number of executor hosts, the number of environments (development, staging, production), the identity integration, and the amount of DAG code that must be migrated or reviewed. Ask any provider for a quote that separates one-time build work from an optional support retainer, and for a list of the systems and accounts you must provide.
How long does an Airflow build on virtual machines take?
Duration depends on scope and on how ready your environment is. The main drivers are the number of Airflow services you run (API server, scheduler, DAG processor, triggerer, and workers if you use CeleryExecutor), whether you need a separate staging environment, how many providers and secrets backends you connect, the identity model, and the readiness of your network, DNS and certificates. A single-host LocalExecutor setup with an existing PostgreSQL server is much smaller than a multi-host CeleryExecutor cluster with a broker, remote logging and SSO. Ask for a timeline in the written scope, and confirm which items depend on your change approvals, which can slow a build more than the technical work.
What must our internal team provide before the build starts?
Most delays come from missing inputs, so prepare these before work begins:
- A supported PostgreSQL instance (Airflow 3.3.2 supports PostgreSQL 14 to 18) with a dedicated database, role, backups and connection limits agreed with your DBA.
- Virtual machines with the operating system, Python version (3.10 to 3.14), at least 4 GB of RAM recommended, and time sync.
- Network rules between the API server, scheduler, DAG processor, triggerer, workers and the database, plus the ports Airflow uses for log serving (8793 for workers and 8794 for the triggerer).
- A Git repository or internal artifact location for DAG code, with access for the service account.
- Identity provider details, secrets store access, and DNS or load balancer names for the API server.
Confirm each item against your own standards.
Can our team run Airflow on VMs without outside help?
Yes, if your engineers can operate Linux services, PostgreSQL, Python packaging and network security, and if someone owns upgrades. Airflow is open source under the Apache License 2.0, so there is no license fee to pay. The harder part is the operating model: installing each component as a separate service, keeping constraint-based installs consistent across hosts, handling the Airflow 3 changes (for example, the separate DAG processor and the API server replacing the webserver), and rehearsing upgrades. A small internal team can do this, but it should write runbooks for restarts, backups and log access before go-live. An outside partner is useful when you need the design reviewed, a fixed-scope build with written deliverables, or knowledge transfer on a deadline.
Which Airflow 3 changes should our engineers check on VMs?
Engineers should check the service layout first. Airflow 3 replaces the webserver with an API server started with airflow api-server, and the DAG processor must run as its own component started with airflow dag-processor. Task code can no longer use database sessions or models directly and must use the REST API or Task SDK. Before upgrading from Airflow 2, the upgrade guide requires being on Airflow 2.7 or later and recommends checking DAG code with the Ruff AIR301 and AIR302 rules. Also note that the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are not supported in Airflow 3.0.0 and later. Review the upgrade guide before you choose a layout.
When should we move Airflow from VMs to Kubernetes instead?
Consider Kubernetes when you already run a platform team, need elastic worker capacity, or want per-task isolation that KubernetesExecutor provides. VMs remain a sound choice for a stable workload on a small number of hosts, for teams without container operations, or in networks where a cluster is not approved. Airflow supports running several executors in one deployment since 2.10.0, so a mixed model is possible, but it adds design work. Kubernetes also adds costs: cluster operations, image pipelines, and Helm chart upkeep. Pick based on your scaling pattern, your operations skills and your change controls. Note that the official Helm chart and container images are the production path on Kubernetes, and that the production guide advises against SQLite outside testing on any platform.
What operational work continues after a VM build is handed over?
Once the build is live, recurring work includes patching the operating system and Airflow, testing DAGs against each new patch, monitoring the scheduler and workers, rotating secrets and keys, and cleaning up metadata database growth from run history. Airflow 2 reached end of life on April 22, 2026 and gets no fixes, so keep your deployment on a supported release line and plan upgrades on a schedule. The Airflow support policy says limited-maintenance releases receive only security and critical bug fixes. Ask any provider whether ongoing support, upgrades and testing are offered as a retainer and what the response expectations are in writing.
How scope and delivery are agreed for a VM build
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, whether that is your cloud accounts, your data centers or an air-gapped network. For a virtual machine build of Airflow, that means the work happens on hosts you own or control: the API server, scheduler, DAG processor and triggerer services, the workers (for LocalExecutor or CeleryExecutor), and the connection to an external PostgreSQL database. Most engagements are fixed-scope, so the component list, the executor choice, the operating system baseline and the handover items are written down before any build starts. Ask for that document to name each host role explicitly, since a VM build has no Kubernetes control plane to absorb ambiguity later.
Before a quote is issued, expect the written scope to settle the questions that drive effort: which Airflow 3 patch line is targeted, which Python and PostgreSQL versions your hosts will run, whether you need one scheduler or several, how DAG code reaches the hosts, how logs are served and who may reach the log ports, and which secrets backend holds the Fernet and API secret keys. Your side should confirm network rules between hosts, the identity provider used for sign-in, backup retention and the monitoring stack that will watch these services. DeployOpen does not publish a price list. The quote follows from the written scope, so a vague scope produces a vague quote, and a change to any of these items after signature should be handled as a documented change to the deliverables and timeline.
Ask every bidder, DeployOpen included, to state in writing what is one-time and what recurs. One-time work usually covers design, the build, hardening, testing and handover. Recurring work covers patching to new Airflow releases, host and database maintenance, backup restore tests and any retained support. Ask for deliverables that your team can run without the implementer, such as runbooks, configuration in version control, a restore test record and a list of open risks. Confirm the support model, response expectations and the upgrade path in the contract before work starts, and check Airflow's version life cycle so the agreed patch line still receives fixes when the platform goes live.
How DeployOpen delivers Airflow on your infrastructure
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Airflow on virtual machines, that means fixing the executor choice, service layout, database and log ports in a written scope before any host is configured, then handing over a system your team can run.
How we scope
- Written scope that names the executor, either LocalExecutor or CeleryExecutor, and the reasons for the choice
- Service map covering the API server, scheduler, DAG processor and triggerer on each host
- External PostgreSQL plan with version, backup and connection requirements confirmed against the Airflow prerequisites
- Confirmed log serving ports, firewall rules and the criteria that would trigger a move to Kubernetes
How we build
- Hosts configured with the Airflow 3 components started as separate services, with the DAG processor run on its own
- Shared secret key and connection settings handled through your secrets process rather than plain configuration files
- DAG delivery path set up from your internal source control, with a staging step before production runs
- Test runs verifying scheduler behavior, task execution, log access and recovery after a service restart
How we hand over
- Runbook covering start, stop, upgrade and rollback steps for each service on each virtual machine
- Architecture record listing hosts, ports, database settings, executor configuration and access boundaries
- Walkthrough with your platform and data engineering staff so they can operate the system without us
- Documented checks for database cleanup and backup restore so recovery steps are tested, not assumed
How we support
- Support and maintenance available as a retainer, covering patch reviews and Airflow version planning
- Upgrade plans scoped separately, including the Airflow 2 to 3 path and DAG code checks
- Periodic review of executor, database and log settings as run volume and team size change
- Security testing of the deployment available as a separate written engagement when you need it
