Why SaaS teams run Apache Airflow in their own infrastructure
Apache Airflow is licensed under the Apache License 2.0 and the project offers no paid edition, so the decision for a SaaS data team is not about a license fee. It is about who operates the scheduler, workers, metadata database and DAG delivery pipeline, and who carries the upgrade work. The current line is Airflow 3, first released on April 22, 2025, with 3.3.2 as the current patch release. Teams that already run product analytics on their own warehouse often find that keeping the orchestrator in the same environment simplifies access control and network paths.
The picture changes once pipelines carry customer data. Product event feeds, billing extracts and customer-facing data deliveries pass through the same DAGs, so the scheduler, its connections and its logs become part of the system a SOC 2 examination describes. Auditors will ask who can change DAG code, how those changes are reviewed, who can read connection secrets and how runs are traced to a change. Your team should confirm early which Trust Services Criteria apply to the scope, since the security common criteria apply to every SOC 2 and the other categories only when included. Airflow does not produce that evidence on its own, so plan where approvals, access reviews and run logs will be stored before the first audit period starts.
Self-hosting also means owning the support calendar. Confirm with your platform team which Airflow line and Python, database and Kubernetes versions your environment will run, and check them against the version life cycle and prerequisites before committing. Ask whether your Airflow 3 deployment will use the separate DAG processor and API server components that the upgrade guide requires. The effort to keep that stack current is a recurring cost that belongs in any comparison with a managed service.
Which SaaS requirements shape the Airflow build
This table maps five requirements that SaaS companies commonly face to the source that sets them and the build decision each one affects. Read each row as a question the build must answer before go-live, such as how long run history is kept or who can change a DAG. The main takeaway is that evidence, access and supportability requirements are settled in the platform design, not added after an audit window opens.
| Requirement | Source | What it means for the build |
|---|---|---|
| SOC 2 common criteria CC1 to CC9 | AICPA 2017 Trust Services Criteria; security common criteria apply to every SOC 2 | CC6 logical access and CC8 change management shape DAG promotion through reviewed code, role-based access in the Airflow UI and API, and separate production credentials. CC7 system operations shapes alerting on failed runs and scheduler health. |
| Type 1 versus Type 2 evidence windows | AICPA SOC 2 report definitions; the auditor's operating effectiveness testing covers a period | A Type 1 report looks at control design on a date, while a Type 2 report tests whether controls operated over a period. Retain run history, deployment records and access review exports for the full observation window, and confirm retention periods with your auditor before setting Airflow database cleanup rules. |
| GDPR Article 28(3) processor instructions | Regulation (EU) 2016/679, Article 28(3) | Where pipelines process customer personal data on behalf of customers, documented instructions bound the processing. Keep pipeline purposes, data categories and transfer destinations in DAG metadata or a register, and limit task access to the datasets each DAG is documented to use. |
| EU Data Act switching rights | Regulation (EU) 2023/2854, Articles 23, 25 and 50; Article 29 on switching charges | If your company provides data processing services to customers, customers may need to switch to another provider or to on-premises infrastructure. Keep DAGs, connections and exports portable and documented. From 12 January 2027, switching charges may not be imposed, so price the export path accordingly. |
| Airflow support policy | Airflow version life cycle; Airflow 2 reached end of life on April 22, 2026 | Airflow 2 no longer receives fixes, so any remaining 2.x deployment is a supportability gap. Plan the Airflow 3 upgrade path, including the Ruff AIR301 and AIR302 checks and the separate DAG processor, and confirm the target patch release with your team before cutover. |
Which pipelines a SaaS company puts on Airflow first
A SaaS company usually starts Airflow with the pipelines that already break most often when run by hand or by cron: product analytics event loads into a warehouse, customer usage metering that feeds invoices, billing and CRM syncs, and per-tenant data exports for customers who ask for their data. Product event loads are the usual first candidate because they are high volume, depend on upstream application releases, and need backfills when a schema changes. Usage metering and billing syncs come next, since a missed or duplicated run becomes a customer-facing invoice error. Per-tenant exports are often last, because they touch the most sensitive data and need the strictest access rules. As an illustrative example, a team might start with one event load and one CRM sync, then add metering once the retry and alerting patterns are proven. Confirm with your own data owners which of these pipelines carry contractual or revenue reporting obligations, since those set the order of work.
Tenant isolation and DAG ownership matter more here than in a single-department deployment. In a single-department Airflow, one team can reasonably share connections, variables and a metadata database. In a SaaS company, a DAG that exports one customer's data can read the connections that serve every other tenant, so the design question is which DAGs can read which secrets, which tenant identifiers are passed as parameters, and whether exports run in separate worker pools or namespaces. Assign each DAG a named owning team in code review and tag it so that changes to customer-facing pipelines follow the same change control as the product. Review the Airflow 3 access model and the connection and variable permissions for each team before the first tenant export runs, and confirm with your security team how those permissions map to your SOC 2 access controls.
Which Airflow components and settings matter for a SaaS data platform
A SaaS data platform rests on a handful of Airflow components and settings that are hard to change later. The cards below cover the metadata database, executors, scheduler resilience, the Airflow 3 process layout, authentication and the Kubernetes packaging, with what each one means for a product analytics and customer data team.
Metadata database choice
Airflow stores DAG runs, task states and connections in a metadata database, and every scheduler, worker and API component depends on it. Use PostgreSQL or MySQL in production, where the supported versions are listed in the prerequisites. SQLite is for testing only, and MariaDB is not supported as the metadata database. Ask your platform team who owns backups, major version upgrades and connection limits before the first customer pipeline goes live.
Executor selection and scaling
The executor decides how task work reaches compute. KubernetesExecutor runs each task in its own pod, which suits bursty batch loads from many product teams. CeleryExecutor keeps long-lived workers and can suit steady, high-volume ingestion. Since Airflow 2.10.0 several executors can run in one deployment, but the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor were removed in Airflow 3.0.0, so plan the choice with the upgrade path in mind.
Multiple schedulers for resilience
Airflow can run more than one scheduler at the same time, coordinating through database row-level locks. For a SaaS platform this means a scheduler restart does not stop customer-facing data refreshes, and throughput can grow as the number of pipelines grows. Confirm that your database connection limits and scheduler replica counts are sized together, and test a scheduler failover in staging before relying on it for reporting deadlines.
Separate DAG processor and API server
In Airflow 3 the webserver is replaced by an API server, and the DAG processor must run as its own component. Engineering teams that deploy DAGs from Git need to size the DAG processor for parse load, since a large number of tenant pipelines can slow scheduling if it is undersized. Your deployment templates, health checks and monitoring need one entry per component, not one for the old webserver.
Auth manager and identity integration
Airflow 3 uses the Simple auth manager by default, and only one auth manager can be active at a time, set by the auth_manager option. Teams that need role-based access for analysts, data engineers and on-call staff usually plan for the FAB auth manager or a Keycloak integration. Decide this before onboarding users, because switching later changes how roles and permissions are mapped.
Official Helm chart and images
The Airflow community maintains an official Helm chart that deploys the official Docker image, and the production guide recommends container images for Kubernetes. Using this pairing keeps your platform on a supported packaging path, with image builds pinned to a tested Airflow release. Ask who builds and scans your custom images, and how chart values for executors, secrets and persistence are reviewed in change management.
Self-hosting versus managed Airflow: what the cost comparison includes
Airflow is open source under the Apache License 2.0, so there is no license fee to compare. The cost sits in the platform your team already runs and in the people who keep Airflow current. Your company pays for the cluster capacity, the managed or self-run PostgreSQL metadata database, object storage for logs, and the engineers who build the Helm chart or manifests, configure executors and maintain the deployment. Airflow's official Helm chart and container images reduce the build effort, but they do not remove the upgrade work.
- Infrastructure: Cluster nodes, database instances, storage and network are billed by your cloud or data center team and recur monthly or yearly.
- Upgrades: Major version moves, such as the move from Airflow 2 to Airflow 3, are one-time projects. Routine patch releases recur.
- On-call and patching: Your platform or data team carries the pager for scheduler, worker and database incidents, and applies security patches to images and the cluster.
- Backups and evidence: Your team sets metadata database backup schedules, restore tests and change records that an auditor will sample.
- Scaling drivers: Scheduler count, Kubernetes worker pod scaling, database size and the number of teams sharing one deployment each change the recurring bill.
Ask any partner for a quote that separates one-time build work from monthly operating work, names who holds the on-call rota, and states which Airflow versions and upgrade windows it covers. Confirm your Kubernetes version against the Airflow prerequisites before you price the build.
Running Airflow on virtual machines removes the Kubernetes layer but moves more of the work onto hosts. Your company pays for the virtual machines, the database and the staff who manage package installs, service units, worker processes and host patching. The schedule of host maintenance is usually the largest recurring item, because each machine needs its own patch cycle, configuration drift control and monitoring.
- Infrastructure: Virtual machines for the scheduler, API server, DAG processor and workers are billed continuously, and you size them for peak concurrency rather than scaling pods on demand.
- Upgrades: Each upgrade touches every host and the Python environment, so rehearsal in staging is a recurring cost tied to each release.
- On-call and patching: Operating system and Python patches land on your team's calendar. Host failures and disk pressure become on-call events.
- Backups and evidence: Backup jobs and restore tests cover the database and the DAG and log storage. Evidence comes from host configuration records and change tickets.
- Scaling drivers: Adding scheduler or worker hosts is a manual change, so the number of teams and DAGs drives headcount time as much as hardware.
When you request a quote, ask how the partner handles host configuration management, how it tests each Airflow upgrade on the same host image, and what happens to the schedule during a failed patch window. Confirm that the database engine is PostgreSQL or MySQL, since SQLite is not suitable for production.
A managed Airflow service shifts infrastructure operation and part of the upgrade work to the provider. Your company still pays for the subscription, the cloud resources the service consumes, and the internal staff who write DAGs, own data quality and answer to auditors. The subscription price varies by provider and configuration, so compare the quoted structure rather than the headline rate. Confirm first which parts of the service are the provider's responsibility and which remain yours.
- Infrastructure: The provider runs the scheduler, workers and database, but usage-based charges for workers, storage and data transfer usually recur and can grow with DAG count.
- Upgrades: Check whether the provider upgrades Airflow on a published schedule, whether you can pin a version, and whether you must move DAG code to new APIs yourself.
- On-call and patching: The provider typically carries infrastructure incidents, but DAG failures, data quality issues and downstream impacts stay with your team.
- Backups and evidence: Ask for the provider's backup retention, restore testing record and its own audit report. Your company still needs evidence for its own controls over DAG changes and access.
- Scaling drivers: Worker concurrency limits, environment count and the number of teams with their own access drive the subscription tier and usage charges.
Request a quote that lists every usage meter, the support tier and the upgrade commitments in writing. Confirm where data is processed and stored, and what the exit terms are for moving schedules back to a self-hosted deployment.
How a SaaS Airflow rollout is phased
The sequence moves a SaaS team from planning to a production Airflow platform that runs its first customer-data pipeline under version control, with identity, secrets and runbooks in place before handover. Each step closes with a check the team can verify before the next one starts.
Confirm the upgrade path if migrating from Airflow 2
If you already run Airflow 2, confirm your current version is 2.7 or later, move to the latest 2.x release first, and run the Ruff AIR301 and AIR302 rules against your DAG code. Your engineers should also remove direct database session and model use from task code, which Airflow 3 no longer allows, and replace SubDAGs with TaskGroups. The step is done when the DAG code passes these checks and a test run succeeds on the 2.x release. Confirm the exact upgrade steps against the upgrade guide for your version, because the breaking changes depend on which features your pipelines use.
Deploy the metadata database and core components
Provision a production metadata database, using PostgreSQL or MySQL in a supported version, and start the scheduler, the DAG processor and the API server as separate components. Airflow 3 replaces the webserver with the API server and requires the DAG processor to run on its own. Choose an executor that matches your workload and confirm that your Kubernetes or virtual machine base meets the prerequisites for the release you chose. The step is done when all components start cleanly, the scheduler runs a test DAG end to end, and backups of the metadata database are being taken.
Set up DAG delivery through Git and CI
Your platform team sets up a Git repository for DAG code, with CI that lints DAGs, runs unit tests and deploys to a staging environment on merge. Decide how DAG files reach the scheduler and workers, and document who can approve changes to production. Customer-data pipelines usually need code review, so the pipeline should record who changed what and when. The step is done when a change merged to the main branch reaches staging automatically and a failing test blocks the deployment.
Connect identity and secrets
Configure the auth manager that matches your identity provider and map your engineering roles to Airflow permissions, so that access to customer-data pipelines is limited to the teams that own them. Move connection credentials and variables for warehouses and SaaS sources into a secrets backend that your organization already operates, rather than storing them in DAG code. Confirm that secrets are masked in logs and that administrative access is logged. The step is done when a test user sees only the intended DAGs and a test connection resolves its credentials from the secrets backend.
Onboard the first customer-data pipeline in staging
Pick one pipeline that moves customer or product event data, such as a daily load from a product database into a warehouse, and build it in staging with its tests, data quality checks and alerting. Agree in advance who receives failure alerts and what the rollback looks like if a load writes bad data. Run the pipeline through several scheduled cycles and review the task logs, retries and resource use. The step is done when the pipeline completes on schedule in staging, its failures alert the named owners, and the team has signed off on the output.
Cut over to production and hand over runbooks
Promote the verified pipeline and its configuration to production through the same CI path used for staging, and run it in parallel with any existing job for a period agreed with your data owners before retiring the old job. Hand over runbooks for restarts, backfills, credential rotation, database cleanup and upgrades, and record the access and change evidence your SOC 2 program will need. The step is done when production runs match staging results, the previous job is switched off under a recorded change, and the operations team has walked through each runbook.
What SOC 2 evidence Airflow can produce for the auditor
A SOC 2 examination tests whether controls exist and, for a Type 2 report, whether they worked over a period. The cards below map the Airflow evidence a SaaS team can plan for to the security common criteria themes it will be asked about. DeployOpen prepares the systems, controls and evidence; an independent CPA firm decides what the report says.
Logical access and roles
Airflow 3 uses the Simple auth manager by default, and only one auth manager can be active at a time, set by the auth_manager option. For a SaaS platform, the design question is whether production DAG editing, connection and variable access, and API tokens map to named roles tied to your identity provider. Ask the auditor which access reviews and joiner-mover-leaver records they will sample, and confirm that the chosen auth manager can produce them.
Change management through reviewed DAGs
If DAG files are delivered only from a Git repository through merge requests, each pipeline change has an author, a reviewer, a commit and a deployment record. Promotion from staging to production should be a documented step, not a manual copy to the scheduler. Agree with your auditor how emergency changes are recorded and how sample changes will be pulled over the examination period.
System operations and monitoring
Scheduler health, failed task alerts, worker capacity and API server availability are the operational signals an auditor will expect you to watch and respond to. Logs for task runs and administrative actions need a defined retention period and a route to a central log store that the team cannot silently edit. Confirm with your auditor which alert records and incident tickets they will request as proof that monitoring operated.
Metadata database and backup recovery
The metadata database holds run history and task state, and the production guidance recommends PostgreSQL or MySQL rather than the SQLite default, which is for testing only. Availability and recovery evidence depends on backups that are taken on a schedule and restored in tests, with the results recorded. Ask your auditor which restore test records they expect to see and how often you must produce them.
Vendor and open-source dependency risk
Airflow is released under the Apache License 2.0 with no paid edition from the project, so the vendor risk sits in your own dependency list, provider packages, Helm chart and base images. Keep a maintained inventory of versions and track the support policy, because Airflow 2 reached end of life on April 22, 2026 and gets no fixes. Record your review of each upgrade and of third-party provider packages as your risk-assessment evidence.
What SaaS teams ask before hiring an Airflow partner
What drives the implementation effort and cost of self-hosted Airflow?
The main drivers are the number of pipelines and data sources, how many environments you need, the executor and deployment target, and how much identity, secrets and logging integration the platform must have. A team running a few product analytics loads on one Kubernetes cluster has a very different build from one running customer data pipelines across several regions with separate staging and production. Cost is set by the written scope, so a quote follows from the list of pipelines, integrations, environments and deliverables you agree before work starts. Ask the partner to show which items are one-time build work and which are recurring operations or support.
Can our engineering team run Airflow in-house instead of hiring a partner?
Often yes, if you already operate Kubernetes or similar infrastructure, have engineers who know Python and DAG testing, and can own upgrades. The harder parts are usually the metadata database, scheduler high availability, DAG delivery, secrets and the Airflow 3 changes, which include a separate DAG processor, an API server in place of the webserver, and no direct database access from task code. Weigh those skills against your roadmap. A partner is most useful for planning the build, documenting it and handing over runbooks, while your team keeps the pipelines and data ownership.
How long does an Airflow rollout take, and what sets the duration?
Duration depends on the number of integrations, the deployment target, how many teams will write DAGs, and how quickly your identity, secrets and network approvals move. Migrations from older Airflow versions add time, because Airflow 2 reached end of life on April 22, 2026, and the Airflow 3 upgrade guide requires being on Airflow 2.7 or later and checking DAG code with Ruff AIR301 and AIR302 rules. Ask for a timeline in the written scope that lists the approval steps and the client-side dependencies, so you can see which waits sit with your organization.
What does the partner need from our team to start?
Plan for access and decisions before the build starts. Typical needs include a cloud account or cluster where Airflow will run, network paths to your warehouse and SaaS data sources, an identity provider for sign-in, a secrets store for connections, and named data owners who can confirm which datasets are in scope and who approves changes. Airflow 3 allows one auth manager at a time, set by the auth_manager option, so decide early whether you will use the default Simple auth manager, FAB, or an identity-specific manager. Confirm each of these items with your platform and security teams during scoping.
Does DeployOpen prepare SOC 2 evidence, or does an auditor issue the report?
DeployOpen prepares systems, controls and evidence for a compliance assessment, and independent auditors issue the SOC 2 report. For an Airflow platform, evidence usually covers access reviews, change management for DAG code, logging of runs and configuration changes, backup and restore tests, and documented job ownership against the common criteria in the 2017 Trust Services Criteria. A SOC 2 Type 1 report covers control design as of a date, while a Type 2 report adds an opinion on operating effectiveness over a period, so plan evidence collection for the period your auditor will test. Ask your auditor which criteria and points of focus they will examine before scoping the controls.
How does self-hosting Airflow compare with managed Airflow on recurring cost?
Self-hosting moves recurring costs onto your own infrastructure and staff time: compute for the scheduler, workers, API server and DAG processor, the metadata database, logging, backups, upgrades and on-call effort. Managed Airflow bundles most of those into a subscription, but you pay the vendor's rates and accept its configuration limits. Airflow itself is open source under the Apache License 2.0 with no paid edition from the project, so the comparison is about operating labor and infrastructure against a service fee. Ask both options for a written three-year cost model that shows one-time and recurring items, and include the cost of upgrades and support retainers if you choose them.
What operating work continues after the Airflow platform goes live?
Plan for recurring work in four areas: version upgrades, database maintenance, capacity changes as your customer base grows, and evidence collection for audits. Airflow 2.11.2 is the last Airflow 2 patch and is past end of life, so upgrades on a supported line should be scheduled rather than left to crisis. Multiple schedulers can run together using database row-level locks, which helps with resilience as load grows, and the metadata database needs regular cleanup and backups. Ongoing support, upgrades and continuous testing are available as DeployOpen retainers, and the scope of any retainer is agreed in writing.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports Apache Airflow inside infrastructure you control, including your cloud accounts, data centers or air-gapped networks. For SaaS and technology teams, the work starts with a written scope that separates customer-facing pipelines from internal analytics, so the platform fits how your data is produced and used.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- We map which product analytics and customer data pipelines run on Airflow and which data sources and sinks they touch.
- We confirm the Airflow 3 version, Python and metadata database targets, and the executor model your load requires.
How we build
- We deploy the API server, scheduler, DAG processor and workers as separate components in your own infrastructure.
- We set up DAG delivery, secrets handling, SSO and role mapping, and tenant boundaries between teams and customer data.
- We test DAG code, failure paths and recovery steps before the platform carries production pipelines.
How we hand over
- You receive architecture records, configuration documentation, runbooks and the access model in writing.
- We document evidence sources for SOC 2 control areas, such as change records, access reviews and job logs, for your auditor to assess.
- We walk your engineers through deployment, upgrades, backups and troubleshooting so your team can operate the platform.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the build.
- Retainers can cover planned Airflow patch releases, dependency updates and reviews of DAG changes before promotion.
- DeployOpen prepares systems and evidence for compliance assessments but does not issue audit reports or attestations.
