What an Airflow 3 implementation involves
A planned Airflow 3 rollout covers the platform build on infrastructure you control: the metadata database, the executor, DAG delivery, secrets and the first production pipelines. Each component needs a decision before the build starts. The metadata database must be PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, since MariaDB is not supported and SQLite is for testing only. Airflow 3 also runs the API server and DAG processor as separate components alongside the scheduler, so the topology has more parts to plan than Airflow 2.
Teams usually start this work for one of three reasons: a new scheduler for a growing data platform, a move off an Airflow 2 release that no longer receives fixes, or a plan to consolidate cron jobs and ETL scripts into one orchestrator. Airflow 2.11.2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, so an Airflow 2 estate now gets no fixes. Confirm your exact release and its support status in the version life cycle table before you set a date.
Apache Airflow is licensed under Apache-2.0, and the project offers no paid edition. The cost of a rollout is therefore the work and the infrastructure: engineering time for design, build and testing, the database, compute for schedulers, workers or executor pods, storage, and ongoing upgrades. Ask any quote to itemize these components and to separate one-time build effort from recurring operations.
Build in-house when your team already runs PostgreSQL and Kubernetes or Celery well, can own DAG standards and upgrades, and has a first pipeline set small enough to learn on. Bring in a partner when the deadline is fixed, when existing DAGs depend on Airflow 2 patterns such as direct database access, or when the platform must pass security review at first release. A useful test is whether the people who will run the platform after go-live are involved from the start of the build.
What the work includes and what you receive
A typical fixed-scope Airflow 3 engagement is split into work packages, each ending in a deliverable your team keeps. The scope and deliverables are agreed in writing before work starts, so you can compare them directly with an internal build.
Architecture and component layout
Airflow 3 runs as separate components: the API server (started with airflow api-server), the scheduler, the DAG processor, the triggerer and workers. The work package maps each component to hosts or pods, sets resource sizing assumptions and documents how they connect. You keep a reference architecture diagram and a component inventory to check against your capacity plan.
Metadata database setup
The metadata database must be PostgreSQL or MySQL in production, because SQLite is for testing only and MariaDB is not supported. Confirm the exact supported versions against the Airflow 3.3.2 prerequisites before you select a database. This package covers instance sizing, backups, connection limits and the migration path, and you receive the database configuration and a provisioning runbook.
Executor configuration
The executor decides how tasks reach workers, so this package compares LocalExecutor, CeleryExecutor and KubernetesExecutor against your task volume and isolation needs. Hybrid Celery and Kubernetes executors were removed in Airflow 3.0.0, so multi-executor designs must use supported options. You receive the chosen executor configuration, worker sizing notes and the reasoning behind the choice.
DAG delivery pipeline
This package sets how DAG code moves from your repository to the DAG processor, whether through git sync, a versioned image or another method you approve. It also adds linting and checks, including Ruff AIR301 and AIR302 rules for Airflow 3 compatibility. You receive the pipeline definition, branch and review rules, and a short guide for DAG authors.
Secrets backend and connections
Connections and variables should be read from a secrets backend your security team already runs, not typed into the UI. This package maps each connection to its backend path, sets naming rules and defines who may create or rotate credentials. You receive a connection inventory, the backend configuration and the rotation procedure to confirm with your security team.
Identity integration at planning level
Airflow 3 uses the Simple auth manager by default and allows one auth manager at a time, so the identity choice affects every user and API client. At planning level, this package records the identity provider, the group-to-role model and the API access approach. Full SSO configuration is covered in its own topic. You receive an access model document for review.
Logging, monitoring and alerting
Scheduler lag, task failures, database health and worker capacity are the signals that show whether Airflow is keeping up. This package defines which logs go to your central platform, which metrics are collected and which alerts reach on-call staff. You receive a monitoring configuration and an alert catalog that your operations team can tune.
Backups and restore testing
Backups protect DAG history, connections and run state in the metadata database, while DAG code should live in version control. This package sets backup frequency and retention, and it includes at least one restore test to confirm the procedure works. You receive backup configuration, a restore runbook and the results of the test.
Handover and runbooks
The final package transfers operation to your team through runbooks for routine tasks, upgrades, incident response and access reviews. Walkthroughs with the people who will run the platform are included in scope, so questions are answered before go-live. You receive the runbook set, the handover checklist and a list of open items that remain with your team.
What drives cost and duration
Each row names one factor that moves Airflow implementation effort, why it does so, and one practical way to keep it contained.
| Driver | Why it moves effort | How to keep it down |
|---|---|---|
| DAG count and task volume | Every DAG needs authoring standards, tests, review and monitoring. Effort grows with the number of pipelines and tasks per pipeline, not only with the platform install. | Start with a defined first wave of pipelines, and apply shared templates and a DAG authoring standard so later DAGs follow the same pattern. |
| Source and target systems | Each connection needs credentials, network paths, provider packages, error handling and test data. Unfamiliar or legacy systems add discovery time. | List every source and target in the scope document and confirm owners, access methods and network rules before the quote is final. |
| Custom operators and hooks | Custom code must be designed, tested, packaged and kept compatible across Airflow upgrades. Each custom component adds ongoing maintenance, not only build time. | Prefer existing provider packages where they meet the need, and limit custom code to gaps that are documented and owned. |
| Executor model and executor count | LocalExecutor, CeleryExecutor and KubernetesExecutor each bring different queues, worker images and scaling behavior. Running more than one executor type in a deployment adds routing rules to test. | Choose one executor model for the first release based on workload shape, and document when a second executor type would be justified. |
| Kubernetes or VM hosting | Kubernetes adds cluster access, image pipelines, Helm values and network policy work. VM hosting shifts effort to process supervision, patching and manual scaling. | Use the hosting platform your team already operates, and confirm the supported Kubernetes versions for the Airflow release you are deploying. |
| Metadata database and high availability | The metadata database must be PostgreSQL or MySQL in production, with backups, failover testing and planned schema migrations. Multiple schedulers and API servers add configuration and failover tests. | Reuse a managed or already supported database service where your policy allows, and define the recovery test before build starts. |
| Secrets backend | Connections and variables must be stored in a secrets backend your organization already governs. Integrating it with workers and the API server requires access policies and rotation tests. | Use the secrets service your security team already operates, and confirm how rotation will reach running tasks before development begins. |
| Environments (dev, test, prod) | Each environment needs its own database, connections, deployment pipeline and data handling. Keeping them consistent is ongoing work. | Generate all environments from one versioned configuration, and limit production-like data in lower environments through an agreed masking approach. |
| Compliance evidence needs | Assessors may ask for access records, change history, backup restore results and configuration baselines. Collecting these after the fact takes more work than designing for them. | Agree the evidence list with your assessor early, and set up logging, change records and restore tests as part of the build, not afterward. |
| Team count and identity model | More teams mean more roles, DAG ownership rules and SSO group mappings to design and test. Airflow 3 requires a chosen auth manager, which affects this design. | Define a small set of roles and group mappings first, and confirm which auth manager and identity provider will be used before the scope is signed. |
Implementation phases and outputs
The sequence moves from an inventory of workloads to a hardened, handed-over Airflow 3 platform, with a written output at the end of each phase.
Inventory workloads and dependencies
Your team and DeployOpen list existing schedules, DAGs, data sources, connections, owners and service-level expectations, and record which pipelines are candidates for Airflow 3 at all. The output is a workload inventory with owners, dependencies and an agreed first pipeline set. The phase closes when the client has signed off on that list.
Decide architecture and sizing
DeployOpen documents the decisions that shape the build: executor choice (LocalExecutor, CeleryExecutor, KubernetesExecutor or EdgeExecutor), metadata database, DAG delivery method, secrets backend, auth manager and sizing. Confirm the database against the Airflow 3.3.2 prerequisites, which list PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, and note that MariaDB is not supported and SQLite should not be used in production. The output is an architecture decision record that your security and platform teams approve.
Build a non-production environment
DeployOpen installs Airflow in a non-production environment inside your infrastructure, with the chosen database, executor, secrets backend and DAG delivery path. Confirm the Python version against the prerequisites (3.10 to 3.14) before build. SQLite may be used only to test a local, throwaway setup, never as a shared or production metadata store. The output is a working environment, installation notes and a deployment runbook.
Migrate the first DAG set
DeployOpen moves the agreed first pipelines onto the new platform and adapts them to Airflow 3 conventions, including the Task SDK, the removal of direct database access from task code and TaskGroups in place of SubDAGs. DAG code is checked with the Ruff AIR301 and AIR302 rules before it is accepted. The output is a versioned DAG repository and a list of pipelines migrated, adapted or deferred.
Test and cut over
Test each migrated pipeline for correct scheduling, retries, connections, output and failure alerts, comparing results with the existing system where one runs in parallel. Agree a cutover window and rollback path for each pipeline with its owner. The output is a signed test record and a cutover plan with a documented rollback.
Harden for production
DeployOpen configures production controls: a shared production database with backups, one or more schedulers as your design requires, Fernet key handling, secrets backend integration, role mapping, network isolation for workers and audit logging. Your security team reviews the configuration against your baseline. The output is a hardened production environment and a configuration baseline with the reasoning for each control.
Hand over and document
DeployOpen hands the platform to your operations team with runbooks for upgrades, backup restore, scheduler and worker recovery, DAG release and on-call escalation. Your team runs a supervised operation period before ownership transfers. The output is a handover pack, a trained operations team and a list of open items with owners. Ongoing support and upgrades can be scoped separately as a retainer.
What stays in-house and what to outsource
This table separates the Airflow work your data and platform teams should keep from the platform work many organizations hand to a partner, with the skills each part needs so you can match work to your own team.
| Work area | Usually kept in-house | Often outsourced |
|---|---|---|
| Ownership and skills | DAG ownership and pipeline logic in Python, business rules, schedules and alerts owned by the teams that understand the data. Needs Python, SQL and data engineering. | Platform architecture, deployment and sizing decisions. Needs Kubernetes or VM administration, PostgreSQL administration, networking and identity engineering. |
| Business logic and data contracts | Transformation logic, source and target definitions, data quality expectations and schema change agreements with upstream and downstream owners. Needs data engineering and domain knowledge. | Advisory review of how contracts map onto Airflow connections, sensors and retries, if your team lacks that pattern. Needs data engineering and Airflow operator experience. |
| Platform architecture | Final approval of executor, database and high availability choices, since these tie to your risk and cost model. Needs an architecture owner who can sign off. | Drafting the target design, comparing executor and database options and documenting the decision record. Needs solution architecture and Airflow 3 familiarity. |
| Kubernetes or VM deployment | Operating the cluster or VM fleet after handover, including node patching and capacity reviews. Needs Kubernetes or Linux operations. | Installing the Helm chart or VM packages, setting values, probes and resource limits, and documenting the install. Needs Kubernetes, Helm and Linux administration. |
| Executor and database setup | Choosing capacity targets and reviewing database growth. Needs capacity planning and a database owner. | Configuring the metadata database on supported PostgreSQL or MySQL versions, executor settings and multiple scheduler setup. Needs PostgreSQL administration and Airflow internals. |
| DAG delivery pipeline | Writing DAGs, reviewing merge requests and owning the release cadence for pipeline code. Needs Python and the team's own review standards. | Building the CI/CD path that tests, packages and syncs DAGs (git-sync or image-based). Needs CI/CD, Git, container and Python tooling. |
| Secrets and identity integration | Defining which connections exist, who may use them and the group-to-role mapping for your users. Needs security and access governance. | Wiring a secrets backend, connection encryption and SSO through the auth manager. Needs identity engineering, secrets management and Kubernetes secrets handling. |
| Cutover and go-live | Business sign-off, parallel run results and the decision to retire the old scheduler. Needs data owners and operations leads. | Runbooks, dry runs, rollback steps and cutover execution support. Needs CI/CD, Airflow operations and change management experience. |
| On-call for pipelines | Responding to failed DAGs, data delays and upstream issues in your pipelines. Needs data engineering and on-call rotation. | Platform-level on-call for scheduler, database and worker health, if covered by a support agreement. Confirm response terms in the written scope. |
What usually goes wrong
Most stalled Airflow rollouts trace back to a small set of early decisions. Each card names one of them and the step that prevents it.
Metadata database run as a test setup
Airflow stores scheduler state, DAG runs and task history in its metadata database. The production guide says the default SQLite backend is for testing only and can lose data, and MariaDB is not a supported metadata database. Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, then plan point-in-time backups and a tested restore before go-live.
Executor chosen without worker capacity
The executor decides where tasks run, and each choice needs its own capacity plan. LocalExecutor shares the scheduler host, CeleryExecutor needs worker nodes and a message broker, and KubernetesExecutor needs cluster quota for task pods. Size capacity from peak concurrent tasks and memory per task, and confirm which hybrid executors are still supported in Airflow 3.
DAG processor and API server left out
In Airflow 3 the webserver is replaced by an API server started with airflow api-server, and the DAG processor must run as a separate component. Teams that plan only a scheduler and a web container often find this late, when DAGs stop updating or the UI has no backend. Design both as monitored, restartable services with their own resource and replica plans.
DAG delivery that breaks on each deploy
DAG files must reach the scheduler, DAG processor and workers in step. Manual copies or unpinned branches can leave components running different code after a release. Pick one delivery path, such as git-sync or DAGs baked into a versioned image, and test that a change reaches every component. Run the Ruff AIR301 and AIR302 checks in CI before each release.
Secrets stored in DAG code or variables
Passwords, API tokens and keys written into DAG files or plain Airflow Variables can end up in source control, task logs and backups. Use a secrets backend your security team already operates, restrict who can read connections and variables, and protect the Fernet key that encrypts stored connections. Confirm the backend options for your Airflow version in the official documentation.
Build started on Airflow 2
Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. A build started on it now carries a later migration, and the Airflow 3 upgrade guide requires being on 2.7 or later before upgrading. Start on Airflow 3, using the patch release current at the time of your build, and plan the DAG changes Airflow 3 requires.
How the work changes by starting point
A greenfield platform has no existing DAGs, scheduler or database to carry over, so the work centers on design choices made once and then built. The decisions come first: which metadata database to run, how many schedulers and API servers to plan for, which executor fits the expected task mix, and how DAGs reach the environment.
The client typically supplies:
- A database platform decision, with PostgreSQL or MySQL on infrastructure the client operates, since SQLite is for testing only
- Target Python and Kubernetes or VM versions that match the Airflow 3 prerequisites
- A sample of the first pipelines, so the DAG delivery and secrets design can be tested early
- Named owners for the platform and for pipeline authors
Because nothing is inherited, the quote drivers are mostly the number of environments, the executor and delivery design, and how much of the secrets and identity setup is in scope. Ask the partner to separate design, build and handover in the written scope.
Teams replacing cron jobs usually have scripts that run on fixed schedules with no dependency model, retries or central logging. The work is less about the platform and more about turning each job into a task with a defined schedule, inputs, outputs and failure behavior. Decide early which jobs move, which are retired, and which stay on the host until a later phase.
The client typically supplies:
- An inventory of existing cron jobs with owners, schedules and the systems they touch
- Credentials and service accounts those scripts use today, so secrets can move to a managed backend
- Decisions on alerting, retry policy and who responds to failures
- A review of which scripts depend on local files or host state
Effort rises with the number of scripts that rely on hidden host state or undocumented side effects. Ask the quote to state how many jobs are in scope and whether rewriting them as DAGs is included or only wrapping them.
Moving from Airflow 2 or another scheduler adds a migration track on top of the platform build. Airflow 2 reached end of life on April 22, 2026, according to the Airflow version support table, so existing Airflow 2 estates need a planned path regardless of this project. DAG rewrites, Ruff AIR301 and AIR302 checks, removed direct database access and SubDAG changes are covered in the sibling migration topic, Apache Airflow 2 to 3 migration.
The client typically supplies:
- The current Airflow version, provider packages and a list of DAGs in use
- Access to run existing DAGs in a non-production environment for comparison
- A cutover window and rollback expectations agreed with pipeline owners
- Decisions on whether run history is migrated or kept in the old system
The quote drivers shift toward DAG count, the number of custom operators and the amount of database access in task code. Confirm whether DAG rewriting and parallel running are priced in the same scope.
When Airflow runs on an existing Kubernetes platform, the platform itself is largely already built, but the Airflow deployment adds its own decisions. The official community Helm chart and container images are the usual route, and the Helm chart deployment topic covers values, external databases, DAG delivery by git-sync or image, and logging in detail.
The client typically supplies:
- The Kubernetes version and distribution, checked against the supported range for Airflow 3.3.2
- Cluster namespaces, network policies, storage classes and ingress rules
- A container registry the cluster can pull from, and the image build pipeline owner
- Access to an external database, since the chart is not a substitute for a managed metadata store
Effort depends on how much of the Kubernetes platform the client already runs and how much the partner must configure. Ask whether the quote includes cluster-level changes or only the Airflow release on top of an existing cluster.
In an air-gapped or restricted network, the installation depends on what can be brought in. Container images, Python provider packages and the Helm chart must be mirrored through an approved transfer process before the build starts, and updates follow the same path. Every dependency that reaches the internet during install or DAG runs becomes a design item.
The client typically supplies:
- An internal artifact repository for Python packages and container images, with a documented import process
- Approved sources for provider packages and a list of providers the pipelines need
- Egress rules that define which external systems DAG tasks may reach
- A security review path for each imported artifact
The quote drivers include the size of the mirrored set, the number of providers, and the transfer and review cycle. Ask the partner to list every artifact in scope and who performs each import and verification step.
What you need to provide
Send these inputs before scope is agreed, because each one changes the quote. Start with a workload inventory: the number of DAGs, their schedules, how many are expected to run at the same time, which providers and custom operators they import, and which ones depend on other systems. DAG count and schedule frequency drive scheduler, worker and database sizing, and the custom code list shows how much of the effort is migration or rewriting. If any DAGs have already been checked against the Airflow 3 upgrade rules, note which ones.
Next, provide hosting and access. Name the cloud accounts, data center or cluster where Airflow will run, the network paths needed to each target system, and the change process that opens them. Confirm the approved runtime versions. Airflow 3.3.2 supports Python 3.10 to 3.14 and PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release. MariaDB is not supported as a metadata database, so if it is your standard, that needs a separate decision. If containers are planned, also confirm whether Kubernetes 1.30 to 1.35 is available.
Provide Git hosting for DAG code, with access for the team that will commit to it, branch protection rules and a named approver for merges. Share the secrets platform Airflow should read from, along with the service identity and the paths it may access. Also send your credentials policy for each target system, covering rotation intervals, whether shared accounts are allowed and who may issue them. Connections and secrets handling shape much of the security design, so settle them before design begins.
Decide three things in writing before work starts: a named platform owner, a data owner for each pipeline domain, and a security owner who approves access. Choose the identity model now. Airflow 3 uses the Simple auth manager by default and allows only one auth manager at a time, so connecting an identity provider through the FAB auth manager with OAuth or LDAP changes the build. You also set the retention, access and change approval policies. DeployOpen can prepare the controls and evidence for those policies, but it does not issue audit reports. Independent auditors do that work.
Questions buyers ask before hiring help
Does Apache Airflow have a license fee?
No. Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0, and the project does not offer a paid edition. Your costs are the infrastructure you run it on, the people who build and operate it, and any support agreement you choose to buy. The cost of ownership topic compares self-hosting with managed Airflow services over several years, including the trade-offs in each direction.
What drives the cost of an Airflow implementation?
The main drivers are scope and complexity. Scope covers how many environments you need, how many DAGs exist and how hard they are, and which integrations and identity controls must be built. Infrastructure choices matter too: a supported PostgreSQL or MySQL metadata database (SQLite is not for production), the executor you pick, worker capacity, a secrets backend, and whether you add redundant schedulers and API servers. A one-time build cost is followed by recurring costs for hosting, upgrades and support. Ask any quote to separate the two.
How long does an Airflow rollout take?
The duration depends on scope rather than on Airflow alone. The biggest factors are the number and complexity of DAGs to deliver, whether you are migrating from Airflow 2 (Airflow 2 entered limited maintenance on October 22, 2025 and reaches end of life on April 22, 2026), how many environments and approval gates your change process requires, and how quickly identity, secrets and network access can be provided. Your team's familiarity with Python-based pipelines also affects the schedule. Confirm the timeline in the written scope, where each phase and dependency is listed.
Which Airflow executor should we choose first?
Choose based on where tasks will run and how much capacity you need. LocalExecutor suits a small installation on a single host. CeleryExecutor distributes tasks to a pool of worker machines you manage. KubernetesExecutor starts a pod per task and fits teams that already run Kubernetes. Since Airflow 2.10.0, several executors can run in one deployment, but the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0. The executor selection topic covers the decision in more depth.
Does an Airflow implementation include migrating DAGs from Airflow 2?
Only if the written scope says so, so confirm it before you compare quotes. Moving to Airflow 3 requires Airflow 2.7 or later, and the upgrade guide recommends moving to the latest 2.x release first and checking DAG code with the Ruff AIR301 and AIR302 rules. Airflow 3 also blocks direct database access from task code, requires the DAG processor as a separate component, and removes SubDAGs in favor of TaskGroups. The Airflow 2 to 3 migration topic describes the work and how cutover is planned.
What should a fixed-scope Airflow quote include?
A fixed-scope quote should name the deliverables and the exclusions in the same document. Look for the environments, the Airflow components to be deployed (API server, scheduler, DAG processor, triggerers and workers), the metadata database, executor, DAG delivery method, secrets backend and identity approach. It should also list backups, testing, handover documentation and acceptance criteria. Ask for a clear list of what you must provide, such as cloud accounts, network access, certificates, identity provider details and sample DAGs, because late inputs are a common cause of schedule slips.
How DeployOpen scopes and delivers Airflow work
DeployOpen plans, builds and supports self-hosted Apache Airflow 3 inside your own cloud accounts, data centers or air-gapped networks. Each engagement starts from a written scope that sets the metadata database, executor, DAG delivery and secrets approach before any installation begins.
How we scope
- Engagements are fixed-scope, with the Airflow version, executor, metadata database and DAG sources written down before work starts.
- The written scope lists which Airflow 2 pipelines, if any, must move to Airflow 3 and which DAG code needs checking first.
- Scope names the secrets backend, identity approach and network zones your team has agreed to, so the quote reflects real boundaries.
How we build
- Builds run in your environment, with the metadata database deployed on PostgreSQL or MySQL rather than SQLite for production use.
- Executor configuration follows the scoped workload, and DAG delivery is set up through the method your team chooses and can audit.
- Each build includes tests of DAG code, scheduler and API server behavior, and restart or failover behavior agreed in the scope.
How we hand over
- Handover includes deployment documentation covering components, configuration values, versions pinned and the upgrade path agreed in scope.
- Your operators receive runbooks for common tasks such as adding DAG sources, rotating connections and restarting components.
- Acceptance tests and their results are recorded against the scope so your team can confirm each agreed deliverable before sign-off.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover, sized to the scope you agree.
- Retainer work can track Airflow's version life cycle, including planning moves off Airflow 2 once it reaches end of life.
- Confirm with DeployOpen in writing what the retainer covers, including response expectations and which components it includes.
