Apache Airflow or Dagster: when each one fits
If your Airflow 2 estate is past its end of life on April 22, 2026, the choice is no longer whether to change, but whether the change is an upgrade to Airflow 3 or a move to Dagster. Apache Airflow fits best when you have a large, working library of task-centric DAGs, a team fluent in Airflow operators and the scheduler, and a need to keep existing pipelines running with the smallest change to how they are written. Dagster fits best when you are rebuilding the pipeline layer around data assets, want lineage and testability built into the authoring model, and can accept a rewrite of the DAGs as part of the move.
The decision turns on four questions. First, how large is the existing DAG estate, and how many DAGs would need rewriting rather than just upgrading? Second, what skills does the team have today, and is it comfortable declaring software-defined assets in Python? Third, does the platform need asset-level lineage as a core capability or only task-level run history? Fourth, where must RBAC, SSO and audit logs live. In open-source Dagster these are not part of the free edition, so you either build or operate those controls yourself or buy them in Dagster+. Confirm the Airflow 3 controls you need through the Apache Airflow auth manager documentation before you compare.
Ask DeployOpen or your internal team to inventory every Airflow 2 DAG, its operators, its connections and its dependencies on the metadata database before anyone commits to a target. Request a written scope for each option that separates the upgrade work from the rewrite work, and confirm who will operate the scheduler, database and identity integration after go-live. Treat the end of life date as a fixed deadline, since Airflow 2 receives no fixes after it, and plan the migration path with that date in mind.
Which platform fits which scenario
This table matches common team situations to the platform that usually fits better, with the condition that would reverse each recommendation. Read it as a starting point, then check the reversal condition against your own estate, skills and identity requirements. The main takeaway is that Airflow 2 end of life makes the migration cost the deciding factor for most existing estates, while Dagster's asset model suits teams building new around warehouse tables.
| Scenario | Better fit and reason | What would reverse it |
|---|---|---|
| Large existing Airflow 2 estate with many DAGs | Airflow 3 usually fits better, because Airflow 2 reached end of life on April 22, 2026 and a move to Dagster means rewriting task-based DAGs as assets. Upgrading in place keeps DAG authoring largely familiar. | Dagster fits if the estate is small enough to rewrite and the team wants asset lineage as a long-term goal. Confirm DAG compatibility with the Airflow 3 upgrade guide first. |
| Greenfield data platform built around dbt and warehouse tables | Dagster usually fits better, because its software-defined assets map directly onto tables and models, and lineage follows from that model. | Airflow fits if the platform must orchestrate many non-data tasks such as API calls, file transfers and infrastructure jobs alongside dbt. |
| Team with strong Airflow operators and maintainers already in place | Airflow 3 usually fits better, because the team's operational knowledge of the scheduler, metadata database and executors carries over directly. | Dagster fits if the team is also rebuilding its platform and can absorb a new deployment model with its code location servers and daemon. |
| Team that wants asset-level lineage and testable Python | Dagster usually fits better, because assets and Python functions are the unit of testing and lineage in its documented model. | Airflow fits if lineage needs are modest and the team prefers task-level control with the Airflow operator ecosystem. Confirm which lineage features your use case needs in open-source Dagster. |
| Organization that must self-host SSO and audit logging | Neither is a clear fit without a paid tier. Dagster's pricing page lists SAML SSO, RBAC and audit logs as Dagster+ features, not open-source Dagster features. | Airflow fits if your own deployment supplies SSO and audit through its auth manager and logging stack. Confirm these controls with each vendor before choosing. |
| Team that must run on a single self-hosted stack with minimal moving parts | Airflow fits better for a small team that wants a scheduler and metadata database it already knows, with PostgreSQL as the recommended production backend. | Dagster fits if the team is comfortable running its webserver, daemon and code locations, and wants swappable run storage and launchers. |
| Air-gapped or regulated site with strict change control | Either can run in your own environment. Choose the platform whose upgrade path, release cadence and Python support your change board can approve. | Reverse the choice if one platform's supported Python or database versions conflict with your approved baseline. Confirm both against current release notes. |
How Airflow 3 and open-source Dagster are built
Both platforms split orchestration into a few long-running processes and a metadata store. The differences that matter for on-call, scaling and upgrades sit in how many replicas each process allows and where state is kept, so check these before comparing features.
Airflow scheduler and multiple schedulers
The scheduler decides which tasks run and when, reading DAG state from the metadata database. Airflow supports running more than one scheduler at the same time for performance and resiliency, coordinating through database row-level locks. For on-call work, this means a scheduler failure is less likely to stop all scheduling, but your metadata database becomes the coordination point. Confirm lock behavior and scheduler counts against the Airflow scheduler documentation for your database version.
Airflow DAG processor, API server and executors
In Airflow 3, DAG parsing, the API server that serves the UI and API, and task execution are separate components. Executors determine where tasks run, and since Airflow 2.10.0 several executors can run in one deployment. The hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so existing hybrid configurations need redesign during migration. Ask your team which executor each workload type needs before you size workers.
Airflow metadata database and SQLite limits
Every Airflow component reads and writes state in the metadata database, so its availability and backups govern the whole platform. The production deployment guide warns that the default SQLite backend is for testing only and can lead to data loss, with PostgreSQL or MySQL recommended. Airflow 3.3.2 supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, and MariaDB is not supported. Confirm your database version and backup plan before a pilot.
Dagster webserver and replicas
The open-source Dagster webserver serves the UI and GraphQL API and can run with multiple replicas. This keeps the interface available during a single pod restart, but it does not change where run state lives. Plan the webserver as a stateless front end behind your load balancer and confirm that your storage backend is shared across replicas. Ask DeployOpen or your internal team to test a rolling upgrade of the webserver before go-live.
Dagster daemon and code location servers
A single Dagster daemon operates schedules, sensors and run queuing, so there is one place where scheduled work is triggered. Code location servers are limited to one replica each, which means a crashing user code server stalls only the assets it hosts but cannot be scaled out horizontally. For on-call work, alert on daemon health and code location status. Confirm how your team will restart the daemon during an upgrade window, since scheduling pauses while it is down.
Dagster swappable storage and run components
Dagster's run, event log and schedule storage defaults to SQLite and can use PostgreSQL. The run launcher, run coordinator and executor are swappable components, so you can change where runs execute without rewriting asset definitions. The SQLite default is suitable for trying the product, not for a shared production environment. Confirm PostgreSQL as the production storage choice, and document which launcher and executor your Kubernetes, Docker or Amazon ECS deployment uses.
Capabilities that decide the choice
This table compares the capabilities that most often settle an Airflow versus Dagster decision, from licensing and programming model to the features Dagster+ keeps behind a paid tier. Read each row against your team's skills and the access controls your auditors expect, then check the paid-feature rows against the current Dagster+ pricing page before you rely on them.
| Capability | Apache Airflow 3 | Dagster (open source) |
|---|---|---|
| License | Apache-2.0. The project offers no paid edition of its own. | Apache-2.0. Dagster+ is a separate paid managed service. |
| Programming model | Task-centric DAGs. Pipelines are defined as ordered tasks and dependencies in Python. | Asset-centric. Tables, models and other data products are declared as Python functions, and dependencies follow from those assets. |
| Authoring and local testing | DAG files in Python. Test locally against your own database and executor setup, which your team must provide. | Assets and resources in Python are designed to be testable in isolation. Confirm the local dev workflow against your own project layout. |
| Executors or run launchers | Several executors ship, including LocalExecutor, CeleryExecutor and KubernetesExecutor. Multiple executors can run in one deployment since 2.10.0. | Run launcher, run coordinator and executor are swappable components. Kubernetes with Helm, Docker and Amazon ECS are documented open-source options. |
| Metadata database | PostgreSQL or MySQL recommended for production. SQLite is for testing only, and MariaDB is not supported. | Run, event log and schedule storage default to SQLite and can use PostgreSQL. Confirm the production storage layout with your own deployment. |
| UI and API access | Web UI and REST API are part of the core platform. Confirm the exact API surface for your Airflow version in the documentation. | Webserver serves the UI and GraphQL and can run multiple replicas. Code location servers are limited to one replica each. |
| Role-based access control | Controlled by the configured auth manager. Confirm the options available for your identity provider in the auth manager documentation. | Included in Dagster+ only, per the Dagster pricing page. Not part of the free open-source Dagster. |
| SSO (SAML) | Depends on the auth manager you configure. Confirm SAML or OIDC support with your identity provider before committing. | SAML SSO is a Dagster+ feature, per the Dagster pricing page. |
| Audit logs | Not listed as a built-in Dagster-style feature. Confirm what your deployment logs and where, using your own logging stack. | Audit logs are a Dagster+ feature, per the Dagster pricing page. |
| Catalog search, column lineage and branch deployments | Not a built-in catalog or column-level lineage feature. Plan a separate metadata catalog if your auditors need it. | Catalog search, column lineage and branch deployments are Dagster+ features, per the Dagster pricing page. Lineage is also tied to the asset model. |
| Alerts | Alerting is set up through Airflow configuration and integrations. Confirm the channels your on-call process needs. | Alerts are a Dagster+ feature, per the Dagster pricing page. |
What is free and what is paid
Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0). The project itself does not offer a paid edition, so the software you download carries no license fee. The current release is Airflow 3.3.2, and Airflow 2 reached end of life on April 22, 2026, which means it no longer receives fixes. Any managed Airflow service is a separate decision with a separate vendor, and that vendor's pricing is not part of the open-source project.
- No project-issued paid tier: The Apache-2.0 license covers the code, and the Airflow project does not sell an enterprise version with additional features.
- Cost sits in operation: Your costs are the infrastructure, the metadata database, the staff who run upgrades and the time spent on security patching.
- Managed services are separate: A hosted Airflow offering from a third party is a vendor contract, priced by that vendor, and it is not a feature of the open-source project.
- End of life affects planning: Airflow 2 no longer gets fixes, so an Airflow 2 estate carries an upgrade cost whatever orchestrator you pick later.
Before comparing totals, confirm which Airflow version your estate runs and whether any managed service in your shortlist changes the support obligations you carry.
Dagster is licensed under the Apache License 2.0 (SPDX: Apache-2.0), and the current release is Dagster 1.13.26. You can self-host the open-source edition in your own infrastructure at no license fee. The Dagster documentation separates self-hosting open-source Dagster from the managed Dagster+ service and lists Kubernetes with Helm, Docker and Amazon ECS as open-source deployment options. Your costs are the compute, the storage for run and event data, and the people who operate the deployment.
- Apache-2.0 licensing: You can use, modify and redistribute the open-source code under the license terms.
- Kubernetes with Helm: A documented option for teams already operating clusters.
- Docker: A documented deployment option for teams running containers on their own hosts.
- Amazon ECS: A documented option for teams that run containers on AWS.
- Your operations, your responsibility: Upgrades, backups, identity integration and the storage backend (SQLite by default, PostgreSQL as an alternative) are your work to plan and run.
Confirm with Dagster which of the features you need are available in open source before you assume a capability, especially for access control and audit, which the pricing page places in Dagster+.
Dagster+ is the paid, managed service from Dagster. According to the Dagster pricing page, the following features are Dagster+ features rather than part of the free open-source Dagster: role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs. Pricing is quoted by the vendor, not by DeployOpen, so the figures you use for planning should come from Dagster's own quote process.
- Access control: Role-based access control is listed as a Dagster+ feature.
- Identity and audit: SAML SSO and audit logs are listed as Dagster+ features.
- Data visibility: Catalog search and column lineage are listed as Dagster+ features.
- Delivery workflow: Branch deployments and alerts are listed as Dagster+ features.
- Vendor-quoted pricing: Ask Dagster for a written quote that states which features, deployment model and support level it covers.
If an audit or identity requirement applies to your platform, check the pricing page against that requirement before deciding whether open-source Dagster can meet it.
What drives the running cost of each platform
The running cost of Airflow and Dagster is set less by the software license, which is Apache-2.0 for both, than by the components you host and the people who keep them current. For self-hosted Airflow 3.3.2, the recurring infrastructure is the scheduler, DAG processor, API server, workers or pods chosen by your executor, and a metadata database. The Airflow prerequisites list PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, and the production guide recommends PostgreSQL or MySQL over SQLite. For open-source Dagster, the recurring pieces are the webserver, a single daemon that runs schedules, sensors and run queuing, one code location server per location, and run storage, which defaults to SQLite and can move to PostgreSQL. Ask the vendor or partner to price each component against your environment, including the number of schedulers you will run, since Airflow supports several at once using database row-level locks.
Engineering time is usually the largest recurring line. Someone must patch the metadata database, track Airflow and Dagster release lines, test upgrades against your DAGs or assets, and answer for failed runs at night. Airflow 2 is already past end of life (April 22, 2026), so any estate still on it carries an unpatched-platform risk that the migration budget must cover. If Dagster+ is not bought, you also own identity and audit work: Dagster's pricing page lists role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs as Dagster+ features. Each of those you build or operate yourself is a recurring engineering cost. Dagster+ itself is a subscription whose price you must obtain from Dagster directly, and its cost offsets the identity and audit work rather than adding to it.
One-time costs are the DAG rewrite, test suites, parallel running, cutover and rollback rehearsal, and any partner fees for those tasks. Airflow's own docs note that Airflow 2 to 3 needs upgrade work, and the Airflow 3 executor documentation confirms the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported, so those estates carry extra rework. In a written quote, ask for: the number of DAGs in scope and how many are simple versus dependent on custom operators; the executor model (CeleryExecutor, KubernetesExecutor or both); the metadata database and hosting pattern; the identity integration and audit requirements; whether Dagster+ is assumed; the test and cutover plan; and who carries on-call and upgrade work after handover. A quote that does not state these assumptions cannot be compared with another.
Moving an Airflow 2 estate to Airflow 3 or Dagster
This sequence turns an Airflow 2 estate that is past end of life into a tested target platform, with a documented inventory, a chosen destination, parallel runs and a retirement record. Each phase ends with a check you can confirm before the next one starts.
Inventory DAGs, connections, variables and operators
The platform team, with the owners of each pipeline, lists every DAG, its schedule, its upstream and downstream dependencies, and its sensors. The same inventory covers connections, variables, pools, custom operators and hooks, plus any plugins or custom executors. Record which DAGs run on a schedule, which are triggered by other systems, and which depend on Airflow's UI or REST API for their operation. The step is done when the inventory is reviewed by each pipeline owner and every item is marked as keep, rebuild, or retire. Illustrative: a team might find that a third of its DAGs have not run in months, which narrows the scope before any code moves.
Decide the target platform and version
Leadership, the data platform lead and the security owner choose between Airflow 3 and Dagster for each group of pipelines, using the inventory and the team's skills. Airflow 3 is the current major line, with Airflow 3.3.2 as the current release, and Airflow 2 reached end of life on April 22, 2026, so it receives no fixes. Confirm the Python, database and Kubernetes versions that the chosen target supports in your environment before procurement starts. The decision is recorded in writing with its reasons, the owners of each pipeline group, and the list of Dagster+ features you will or will not require.
Assess the upgrade path and the effort
For estates that stay on Airflow, the upgrade path is documented in the Airflow 3 upgrade guide, which lists breaking changes to imports, context variables, and removed features such as the hybrid executors. Run the upgrade checks against a copy of the DAG code and note every DAG that fails them. For estates that move to Dagster, map each DAG to software-defined assets and note where scheduling, sensors and the dependency model differ. The effort depends on the estate, so the assessment should produce a per-DAG estimate that the partner or internal team confirms. The step closes with a ranked backlog of DAGs by complexity and business criticality.
Rebuild or port pipelines and carry over tests
Engineers port the Python logic of each pipeline group, either into Airflow 3 DAGs with the documented upgrade changes applied, or into Dagster assets, jobs and schedules. Python transformations, business rules, and unit tests usually carry over with changes to imports and runtime context. Connection secrets move into the new platform's secrets backend rather than being copied in plain form, and each credential is rotated where the security policy requires it. Scheduling semantics, sensors, and the dependency model are rebuilt and tested against the inventory. The step is done when every ported pipeline passes its tests in a non-production environment and a reviewer has signed off the mapping.
Run old and new pipelines in parallel
Operate the Airflow 2 estate and the new platform side by side for the pipelines being migrated, writing outputs to separate staging locations or comparing results against the same source data. The team compares row counts, freshness, and downstream report values on a agreed set of runs before any production traffic moves. Monitor both schedulers and their metadata databases during this period, since each consumes compute and needs its own backups. The step closes when the owners of each downstream dataset confirm that the outputs match or that differences are explained and accepted.
Cut over schedules and alerts
Move one pipeline group at a time by disabling its schedule in Airflow 2 and enabling the equivalent in the new platform, with the on-call rota and alert routes changed in the same window. Confirm that failure notifications reach the same channels and that someone receives them during the first runs. Keep a documented rollback that re-enables the Airflow 2 schedule for each group. The cutover is complete for a group when its scheduled runs succeed on the new platform for the agreed observation period and no alert is left routed to the old system.
Retire Airflow 2 and keep the evidence
After every pipeline group is cut over, disable the remaining Airflow 2 schedules, export the metadata database and logs you are required to keep, and store them under your retention policy. Remove the Airflow 2 deployment, its service accounts, network rules and stored secrets, and confirm the removal in your configuration management records. Update the architecture diagrams, runbooks and access reviews. The step is done when the decommissioning checklist is signed by the platform owner and security, and the evidence bundle is filed where your auditors can find it.
Where implementation projects usually run long
Most of the effort in an Airflow to Dagster project sits in your own infrastructure, not in writing asset definitions. On the Airflow side, the metadata database, scheduler, DAG processor, API server and executor are already running somewhere, and their sizing, backups and upgrade path are known to your team. On the Dagster side, you decide the storage layout first: run, event log and schedule storage default to SQLite and can use PostgreSQL, so the database to size and back up is a new one. You also choose the run launcher, run coordinator and executor, which are swappable components, and that choice drives how workers are provisioned. Then plan the rest of the platform work: secrets for source credentials, network paths from the code locations to your warehouses and APIs, observability for the webserver and daemon, and a CI pipeline that builds and deploys code locations. Confirm each of these with DeployOpen and your own platform team before work starts, because the answers change the architecture.
Projects run long for three reasons that are specific to this migration. First, Airflow DAGs hide dependencies: a task that writes a table and a later task that reads it may be linked only by a file path, a variable or an execution order. Mapping those implicit links into asset dependencies takes careful review, and it is where a team should expect the most rework. Second, shared operators and helper modules often have no clear owner. When a single custom operator serves ten pipelines, someone must decide whether it becomes a resource, an asset factory or a library, and that decision needs an accountable team before the port starts. Third, identity and audit requirements are often raised late. Role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs are listed on Dagster's pricing page as Dagster+ features, so ask early which of these your controls require and whether a paid tier, or additional controls you build, will meet them.
Scope and timelines are agreed in writing before any work starts, and this page does not give a duration. Ask for a written scope that names the DAGs in scope, the assets they will produce, the storage and launcher choices, the identity and audit controls, the CI approach and the handover documentation. Ask how the Airflow 2 estate will be run during cutover, since Airflow 2 reached end of life on April 22, 2026 and no longer receives fixes. The Dagster OSS deployment architecture is a useful reference for your platform team to review before the scoping call.
Questions buyers ask before choosing or migrating
Do we have to upgrade from Airflow 2 now?
Yes, you should plan the move now, because Airflow 2 has already reached end of life. The Airflow version support table shows that Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026. In practice, end of life means the project publishes no fixes and offers no support for that line, so new vulnerabilities in Airflow 2 will not be patched upstream. Limited maintenance only covered security and critical bug fixes. Whether you move to Airflow 3 or to Dagster, the first task is an inventory of DAGs, providers, custom operators and plugins, which tells you how much work the port will be.
Can Dagster run our existing Airflow DAGs unchanged?
No. Dagster organizes work around assets, which are logical units of data declared as Python functions, while Airflow organizes work around tasks inside a DAG. Each DAG therefore has to be reviewed and usually rewritten as assets, dependencies and schedules. Operators that call external systems need equivalent code or resources in Dagster. Scope the port by listing every DAG, the operators and hooks it uses, the connections and variables it reads, and how it is triggered. Pipelines that are mostly SQL transformations may move more quickly than pipelines that drive many custom operators.
What does Dagster+ add, and can open source meet our RBAC and SSO needs?
Dagster+ is the paid managed service, and the Dagster pricing page lists role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs as Dagster+ features rather than part of the free open-source Dagster. If your policy requires SAML SSO, role-based access or audit logs in the orchestrator itself, the open-source edition alone may not meet that requirement. You can still place a self-hosted Dagster behind your own identity-aware proxy or access layer, but confirm with your security team whether that satisfies the control. Check current edition details with Dagster before you decide.
What must our team do itself, and what can a partner take on?
Your team keeps ownership of the data, the business logic, access approvals and the decision on which pipelines are retired. A partner such as DeployOpen can plan, implement, customize, integrate, harden, back up, upgrade, document and support a self-hosted platform inside your own infrastructure. For Dagster, that typically covers the webserver, daemon, code locations, the PostgreSQL run and event log storage that replaces the SQLite default, the executor and launcher, and the Kubernetes, Docker or Amazon ECS deployment. Ask in writing which of these items sit in scope and which remain with your staff.
How is a fixed-scope migration agreed?
Most DeployOpen engagements are fixed-scope, which means the scope, deliverables and timelines are agreed in writing before work starts. A written scope usually lists the DAGs or assets in and out of the migration, the target platform and deployment method, the integrations and identity requirements, the testing and cutover approach, and the documentation and handover items. Any change to that list should go through a written change to scope. Ongoing support, upgrades and continuous testing are offered as retainers, so you can separate the one-time migration from the recurring service.
What does a quote depend on?
A quote follows from the written scope, because DeployOpen does not publish a price list. The main drivers are the number and complexity of pipelines, the operators and external systems each pipeline touches, the identity and access model you require, the target deployment and its infrastructure, the data volume and history to migrate, the testing and parallel-run period, and whether you want ongoing support after cutover. Ask for a quote that separates the one-time migration from any recurring retainer, and that states which items are excluded. Your own staff time is a cost too, so include it in your internal estimate.
What should our engineers check before committing to Dagster?
Engineers should run a proof of concept against a representative subset of DAGs before committing. Check how assets map to your tables and models, how your current sensors and schedules translate, and how your team tests assets locally. On the operations side, review the open-source deployment architecture, which runs a single daemon for schedules, sensors and run queuing, and limits code location servers to one replica each. Confirm that the storage backend, run launcher and executor you choose are supported in your environment. Document each result so the decision is traceable.
How DeployOpen helps you choose and migrate
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For an Airflow 2 estate past end of life, we start with a written scope that tests whether Dagster, Dagster+ or another orchestrator fits your data platform team before any migration work begins.
How we scope
- Inventory existing Airflow 2 DAGs, operators, connections and dependencies to size the migration in writing.
- Confirm which Dagster features your team needs in open source and which would sit in Dagster+.
- Agree deliverables, timelines and acceptance criteria in a fixed-scope document before work starts.
How we build
- Stand up open-source Dagster in your cloud account, data center or air-gapped network using your chosen deployment option.
- Configure PostgreSQL run, event log and schedule storage, plus your chosen run launcher and executor.
- Rebuild representative pipelines as assets and test them against your data before cutover.
How we hand over
- Document the target architecture, component inventory, configuration and the Airflow-to-Dagster mapping.
- Run knowledge transfer sessions so your engineers can author, test and deploy assets independently.
- Provide a runbook covering upgrades, backups and the decommissioning plan for the old Airflow environment.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review Dagster release changes and plan upgrades against your environment and written scope.
- Confirm with your team which support window and retainer terms fit your operating model.
