Which open-source workflow orchestrators fit which teams
Choose Apache Airflow if your team already runs DAGs and needs the most established scheduler, Dagster if data assets and lineage should drive how pipelines are built, and Prefect if Python engineers want to wrap ordinary functions in flows with retries and event-based automations. Kestra suits teams that prefer YAML-defined workflows that react to events, but its license and edition terms were not available for this page, so confirm them with its maker before it goes on a shortlist. Any team still on Airflow 2 has a deadline: that line reached end of life on April 22, 2026, and gets no fixes. Plan the move to the 3.x line, where the current release is 3.3.2.
Airflow, Dagster and Prefect are all Apache-2.0 licensed at their core, so the open-source builds carry no license fee. The paid layers differ. Dagster+ adds role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs, all listed as Dagster+ features and absent from the free open-source Dagster. Prefect Cloud is the managed counterpart to self-hosted Prefect server, and Prefect's documentation says it adds RBAC, audit logs and SSO that the self-hosted server lacks. If SSO or audit logs are mandatory, price the managed tier or the internal identity work into your decision.
Self-hosting effort depends mostly on the metadata database and the components around the scheduler. Airflow's production guide recommends PostgreSQL or MySQL and warns that SQLite is for testing only. Dagster's open-source deployment runs one daemon for schedules and sensors, with code location servers limited to one replica each, and its storage defaults to SQLite. Prefect server also defaults to SQLite, and multi-server setups need PostgreSQL 14.9 or later, Redis and a load balancer. Run a proof of concept on your target database and identity setup before you commit.
Which orchestrator matches each team situation
This table maps five common team situations to the orchestrator that fits each one best, with the condition that makes the fit hold. Read the first column to find your situation, then check the condition before you rule out the alternatives. The main takeaway is that existing Airflow investment, Python-first authoring, and network isolation requirements each point to a different choice.
| Team situation | Best fit | Condition and why |
|---|---|---|
| Existing Airflow estate with many DAGs | Apache Airflow (stay on Airflow 3) | Fits when DAGs and operators already run in production. Airflow 2 reached end of life on April 22, 2026, so the migration to Airflow 3 (current patch 3.3.2) is the first task, not an optional upgrade. Airflow is Apache-2.0 with no paid edition from the project. |
| Python-first data team that wants asset-based lineage | Dagster | Fits when pipelines are organized around tables and models rather than task sequences. Dagster declares assets as Python functions. Lineage, catalog search and column lineage are Dagster+ features, so confirm whether you need them in the open-source build or in the paid cloud. |
| Python flows with a self-hosted server and optional managed cloud | Prefect | Fits when you want a self-hosted Prefect server with the option of Prefect Cloud later. The self-hosted server bundles the API, UI and state database, and needs PostgreSQL for production. RBAC, audit logs and SSO are documented as Prefect Cloud features, so plan for them if your audit requires them on-premises. |
| Team that prefers declarative YAML and event triggers | Kestra | Fits when workflows should be written as declarative definitions and started by events rather than Python code. Kestra is covered on this page as the YAML and event-driven option. Confirm its license, edition boundaries and self-hosting components directly with the project before you commit. |
| Platform team that must keep everything inside its own network | Apache Airflow or self-hosted Dagster (open-source build) | Fits when no orchestration control plane may leave your network. Both run self-hosted under open licenses. Dagster OSS documents Kubernetes with Helm, Docker and Amazon ECS deployments. Airflow needs PostgreSQL or MySQL for production, not SQLite. Confirm which paid-edition features your policy requires, since Dagster+ is a managed service. |
Four orchestrators, their licenses and what each is built around
All four options belong to the same question: how to schedule, retry and observe data work in your own infrastructure. Airflow, Dagster and Prefect are Python-first and Apache-2.0 licensed, while Kestra centers on YAML and events. The cards below show what each is built around, where the paid layer sits, and the situation in which each is the stronger choice.
Apache Airflow
Airflow is an Apache Software Foundation project under Apache-2.0, and the project itself offers no paid edition. Its core model is the DAG, a directed graph of tasks defined in Python and run on a scheduler that supports more than one instance at a time. Airflow 2 reached end of life on April 22, 2026, so any inherited Airflow 2 estate needs an upgrade plan. Check that your metadata database is PostgreSQL or MySQL, since SQLite is for testing only. Choose Airflow when your team already runs Airflow 3 or has many existing DAGs to keep.
Dagster
Dagster is Apache-2.0 licensed and organizes work around assets, which are logical units of data such as tables or models declared as Python functions. Its paid layer, Dagster+, adds role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs, none of which are in the free open-source build. Open-source Dagster can run on Kubernetes with Helm, Docker or Amazon ECS. Choose Dagster when your work is organized around data products and lineage matters more than task-level scheduling.
Prefect
Prefect is Apache-2.0 licensed and defines workflows as Python flows and tasks with built-in retries, caching and event-based automations. The self-hosted Prefect server bundles the API, UI and state database, while Prefect Cloud adds RBAC, audit logs and SSO that the self-hosted server does not include. Production self-hosting needs PostgreSQL, and multi-server setups also need Redis and a load balancer. Choose Prefect when your team writes Python-first pipelines and wants a lighter operational footprint for the core engine.
Kestra
Kestra is built around declarative YAML workflows and event-driven triggers rather than Python DAG code. Its license must be confirmed with the vendor before this page is published, and so must the feature split between its open-source build and any paid tier. Confirm the license terms, the event triggers you need and the supported database and storage backends in your own environment. Choose Kestra when platform teams want non-Python authors to build workflows and event-driven triggers are central to the design.
How the open-source workflow orchestrators differ on capability
This table sets Apache Airflow, Dagster and Prefect side by side on license, release line, authoring, scheduling, metadata storage, multi-instance operation, execution model, paid editions and prerequisites. Where the researched facts do not settle a cell, the cell names what to confirm with the vendor. Treat the metadata database, multi-instance and prerequisite rows as the main drivers of operating effort in your own infrastructure.
| Capability | Apache Airflow | Dagster | Prefect |
|---|---|---|---|
| License | Apache-2.0. The project offers no paid edition of its own. | Apache-2.0 for open-source Dagster. Dagster+ is a separate paid cloud service. | Apache-2.0 for the framework and self-hosted server. Prefect Cloud is a managed service. |
| Current release line | Airflow 3.3.2. Airflow 2 reached end of life on April 22, 2026. | Dagster 1.13.26, released October 9, 2026. | Prefect 3.8.8, released October 6, 2026. The 2.x line still received 2.20.26 on September 2, 2026. |
| Authoring model | Code-defined workflows. Confirm the DAG authoring details your team will use against the Airflow 3 upgrade guide. | Asset-based. Data assets such as tables or models are declared as Python functions. | Python workflows with scheduling, caching, retries and event-based automations. Confirm specifics with Prefect. |
| Scheduling and backfills | Scheduler supported. Confirm backfill behavior and limits for your Airflow 3.3.2 setup. | Schedules, sensors and run queuing are run by the daemon. Confirm backfill support in the version you deploy. | Scheduling is a core feature. Confirm backfill behavior with Prefect before relying on it. |
| Metadata database options | PostgreSQL 14 to 18 or MySQL 8.0, 8.4 and Innovation releases. MariaDB and production SQLite are not supported. | Run, event log and schedule storage default to SQLite and can use PostgreSQL. | Defaults to SQLite. PostgreSQL is recommended for production. |
| Multi-instance scheduling | More than one scheduler can run at once, coordinated through database row-level locks. | Webserver can run multiple replicas, but the daemon is single and each code location server runs one replica. | Multi-server deployments need PostgreSQL 14.9 or later, Redis for event messaging and a load balancer. |
| Executor or deployment model | LocalExecutor, CeleryExecutor, KubernetesExecutor, EdgeExecutor and AWS Batch and ECS executors. Several can run together since 2.10.0. | Open-source options include Kubernetes with Helm, Docker and Amazon ECS. Run launcher, coordinator and executor are swappable. | Self-hosted server bundles an API, a UI and a database. Confirm the worker and work pool model for your target platform. |
| What the paid edition adds | Nothing from the Apache project. Any paid support comes from other vendors, so confirm terms with each. | Dagster+ adds RBAC, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs. | Prefect Cloud adds RBAC, audit logs and SSO. Per Prefect's pricing page, Team lists $100 per user per month and SSO, while Enterprise adds object-level RBAC and multiple workspaces. |
| Self-hosting prerequisites | Python 3.10 to 3.14, Kubernetes 1.30 to 1.35 if used, and at least 4 GB of RAM recommended. | Confirm the supported Python and Kubernetes versions for the release you deploy. Plan a PostgreSQL instance for production storage. | Python 3.11 or later, plus PostgreSQL, and Redis and a load balancer for multi-server setups. |
What each option keeps behind a paid cloud or enterprise edition
Apache Airflow is an Apache Software Foundation project released under the Apache License 2.0, and the project itself has no paid edition. Any paid support, managed hosting or packaged distribution comes from third parties who set their own terms, so the purchase question is who you pay and for what. Your recurring costs are the infrastructure, the database service and the engineers who run upgrades, while a support contract is an optional extra.
- Project edition: The Apache-2.0 code is the whole project, so there are no feature gates to compare at the project level.
- Third-party support: Vendors that offer support or hosting define their own scope, response terms and version coverage, and those terms must come from them in writing.
- Lifecycle pressure: Airflow 2 reached end of life on April 22, 2026 and gets no fixes, while Airflow 3.3.2 is the current release, so an upgrade is a cost to price into any contract.
- Production backend: The production guide says the default SQLite backend is for testing only and recommends PostgreSQL or MySQL, so the database service is part of your running cost.
Before you sign, ask each support provider which Airflow versions they cover, whether the Airflow 2 to 3 upgrade is in scope, what their response terms are, and whether they support the executor and Kubernetes versions you run.
Dagster is Apache-2.0 licensed, so the open-source build is free to self-host. The commercial layer is Dagster+, the managed service, and Dagster's pricing page lists a specific set of features as Dagster+ features. Those features matter most for teams with many users, audit obligations or several code locations, because they sit only in Dagster+ and are not part of the free open-source build.
- Access control: Role-based access control and SAML SSO are listed as Dagster+ features, so a self-hosted deployment needs another route to enforce them.
- Catalog and lineage: Catalog search and column lineage are listed as Dagster+ features, which affects how far your self-hosted users can explore assets on their own.
- Branch deployments and alerts: Branch deployments and alerts are listed as Dagster+ features, so review how you test code changes and receive failure notices in each setup.
- Audit logs: Audit logs are on the Dagster+ list, which matters if you must show who changed a pipeline and when.
- Self-hosting path: Open-source Dagster can run through Kubernetes with Helm, Docker or Amazon ECS, and the cost of closing these gaps falls on your own team.
Before you request a quote, confirm which listed features each Dagster+ plan includes, whether the service runs in your cloud or Dagster's, how it is priced at your scope, and what migration support exists if you later move from the open-source build.
Prefect is Apache-2.0 licensed, and the self-hosted Prefect server is the open-source route. The Prefect server documentation states that Prefect Cloud provides features organizations require, such as RBAC, audit logs and SSO, which self-hosted Prefect server does not include. The managed Prefect Cloud plans are where those controls are sold, so the choice between self-hosting and Cloud is partly a choice about identity and audit.
- Hobby: Listed on the pricing page as free.
- Starter: Listed at $100 per month.
- Team: Listed at $100 per user per month, with SSO available from this plan.
- Enterprise: Listed at custom pricing, with object-level RBAC and multiple workspaces.
- Self-hosted gap: RBAC, audit logs and SSO are absent from self-hosted server, so you must build, buy or accept the gap yourself.
The Team rate scales with the number of users, so confirm how your seats are counted before you budget. Check the figures above against the current Prefect pricing page, confirm the currency and billing terms, and ask whether the Enterprise quote covers the object-level RBAC and workspace model you need.
Kestra is the YAML and event-driven option in this comparison. Paid tier details were not confirmed for this page, so what Kestra keeps behind a commercial edition must be confirmed with the vendor before you rely on it. Do not assume parity with Airflow, Dagster or Prefect on access control, audit or support, because each vendor draws its line in a different place.
- Edition boundary: Confirm which features sit in the open-source build and which sit only in any paid tier.
- Authentication and access: Confirm whether single sign-on, role-based access control and audit logging are included in the open-source build or reserved for a paid edition.
- Support terms: Confirm response commitments, the release versions covered and whether upgrade help is included.
- Hosting model: Confirm whether any managed offering runs in your cloud account, in a data center you control or in the vendor's environment.
- Licensing terms: Confirm the license of the open-source build and any conditions attached to commercial use.
Ask Kestra for a written feature matrix by edition, the support terms in writing, and a statement of which features apply to your deployment model. Compare that matrix line by line with the other tabs before you decide.
How scheduling, backfills and upgrades change the operating load
Scheduler design and upgrade policy often drive more day-to-day operating load than authoring style does. Airflow can run more than one scheduler at the same time, using database row-level locks to coordinate, which supports performance and resiliency but means your metadata database sizing and connection limits need to be planned for that. Open-source Dagster takes a different shape: a single daemon operates schedules, sensors and run queuing, and each code location server runs as one replica. Plan for that daemon as a component you must monitor and restart quickly, and confirm with Dagster how it behaves during an outage. Self-hosted Prefect server runs as a single API and UI unless you build a multi-server setup, which needs PostgreSQL 14.9 or later, Redis for event messaging and a load balancer.
Backfills are where scheduler design meets cost. For each option, confirm in a proof of concept how a backfill over a large historical window is launched, throttled and resumed, how it competes with live schedules for workers and database connections, and whether a failed partition can be rerun without reprocessing the rest. Ask each vendor or project for the exact configuration keys and limits in the version you plan to run, since these change between releases and are not covered by a single supplied fact here. Add the backfill test to your maintenance window checklist so its load is measured before production.
Upgrade risk is the largest recurring cost. Airflow 2 reached end of life on April 22, 2026 and gets no fixes, so any remaining Airflow 2 estate needs a dated migration plan, and Airflow 3 is the line to target. Its upgrade guide should be read against your DAG code before scheduling the change. Prefect's server documentation notes that old clients work with new servers, but new clients can be incompatible with old servers as fields are added to the REST API. Upgrade the server first in staging, then roll clients, and pin worker versions during the window so a mixed fleet is never left running unplanned.
How to choose and pilot an orchestrator in your own infrastructure
This sequence moves a team from an inventory of existing pipelines and skills to a tested pilot and a written scope, so the choice between Airflow, Dagner, Prefect and Kestra rests on evidence from your own environment rather than on feature lists alone.
Inventory existing pipelines and team skills
The data platform lead and two or three pipeline owners list every scheduled job, its upstream and downstream dependencies, its run frequency, and the languages and tools each job uses. Alongside that list, the team records which engineers can write Python, which know YAML-driven configuration, and who already operates Kubernetes. This step is complete when the inventory names each pipeline's owner, its failure impact, and the skills gap for each candidate orchestrator, so the shortlist is tested against real workloads rather than generic examples.
Decide whether the paid managed layer is in scope
The CIO or head of platform engineering decides, with the security lead, whether a managed control plane is acceptable for this data. Dagster's pricing page places role-based access control, SAML SSO, catalog search, column lineage, branch deployments, alerts and audit logs in Dagster+ rather than the open-source edition, and Prefect's Server documentation says RBAC, audit logs and SSO are Prefect Cloud features. Record the answer in writing. If the managed layer is out of scope, every access control and audit requirement must be met with the self-hosted build, which needs confirming in the pilot.
Check metadata database and runtime prerequisites
The platform engineer checks each candidate against the versions the vendors publish. Airflow 3.3.2 lists Python 3.10 to 3.14, PostgreSQL 14 to 18, MySQL 8.0, 8.4 and Innovation releases, and Kubernetes 1.30 to 1.35, and it does not support MariaDB as a metadata database. Prefect requires Python 3.11 or later, and its multi-server setup needs PostgreSQL 14.9 or later plus Redis and a load balancer. Confirm that your approved database engine, Python baseline and cluster version fall inside these ranges. The output is a pass or fail table per candidate, with any version gaps raised with the platform team before the pilot starts.
Build the production database and scheduler topology
The platform engineer stands up a non-production environment that matches the intended production shape, not a laptop setup. For Airflow this means PostgreSQL or MySQL in place of the SQLite default, which the production guide says is for testing only, and it means checking whether more than one scheduler is needed, since Airflow supports multiple schedulers through database row-level locks. For Dagster, check the single daemon and the one-replica limit on code location servers. For Prefect, confirm PostgreSQL in place of SQLite. The step closes when the environment starts, survives a restart of each component, and its backup and restore route has been written down.
Pilot one representative pipeline with a backfill and failure path
Two engineers, one who knows the business logic and one who will own the pipeline after handover, port a single representative pipeline to each shortlisted orchestrator. The pilot must include a scheduled run, a historical backfill over a fixed date range, and a deliberately induced failure with a retry and a rerun from the failed point. Compare how each tool expresses the asset or task graph, how it handles the backfill's concurrency and idempotency, and how a failed run is surfaced. The step is complete when each pilot has a short written log of what worked, what needed custom code, and where the team hit a wall.
Compare run history, lineage and access control needs
The security lead and the data governance owner check each pilot against the requirements gathered in step one. They look at how long run history is kept and where it is stored, whether lineage is produced by the orchestrator or must come from a catalog such as OpenMetadata or DataHub, and whether access control can be enforced with the open-source build or only in a paid tier. Mark every requirement as met in the open-source build, met only in a paid edition, or unconfirmed. The step is done when the matrix has no unconfirmed rows, or the unconfirmed rows are assigned to a named owner for follow-up with the vendor.
Write the scope and request a quote
The engagement lead writes a scope that names the chosen orchestrator and version, the number of pipelines in migration, the metadata database and cluster versions from step three, the managed or self-hosted decision from step two, the pilot acceptance criteria from steps five and six, and the items that remain unconfirmed. Ask for a written scope with deliverables and timelines agreed before work starts, since DeployOpen works on fixed scope. The quote should separate one-time work such as installation, hardening, migration and documentation from recurring work such as upgrades, patching and support, and it should state which party pays for any paid vendor licenses. DeployOpen does not publish prices, so the quote is derived from this written scope.
What buyers ask about cost, effort and self-hosting orchestrators
Does running Apache Airflow, Dagster or Prefect cost anything in license fees?
The core projects carry no license fee: Apache Airflow, Dagster and Prefect are all Apache-2.0 licensed, and Airflow has no paid edition from the project itself. Your cost is the infrastructure you run them on, the people who install, upgrade and monitor them, and any paid managed layer you choose. Dagster and Prefect offer paid cloud tiers, and Dagster lists RBAC, SAML SSO, column lineage and audit logs as Dagster+ features rather than part of open-source Dagster. Ask any vendor or partner to split a quote into one-time setup work and recurring support, upgrades and hosting.
What actually drives the effort to self-host an orchestrator?
Effort follows the metadata database, the executor and the number of teams and pipelines. Airflow's production guidance warns against the default SQLite backend and recommends PostgreSQL or MySQL, and its prerequisites list supported Python, PostgreSQL, MySQL and Kubernetes versions that you must track. Dagster's open-source deployment runs a webserver, a single daemon and code location servers, with storage that defaults to SQLite and can move to PostgreSQL. Prefect's self-hosted server needs PostgreSQL for production and, for multi-server setups, Redis and a load balancer. Each added executor, environment or high-availability requirement adds components you must operate.
Can a small platform team run one of these orchestrators on its own?
A small team can run a single orchestrator on its own when the workload is modest, the database and runtime are standard, and someone owns upgrades. The risk sits in the parts that are not optional: backups of the metadata database, patching the runtime, securing the UI and handling version changes. Prefect's documentation notes that new clients can be incompatible with old servers, so upgrade order matters. If your team cannot cover on-call for scheduler or database problems, plan for a retainer or a managed option before go-live rather than after the first incident.
How long does a pilot of a self-hosted orchestrator take?
No duration is fixed by the vendor documentation for a pilot, so treat any timeline as something to agree in writing. The length depends on how many pipelines you migrate, how many data sources need credentials and network access, whether you need SSO and RBAC from the start, and how much of your environment is in place. DeployOpen works on fixed-scope engagements, where scope, deliverables and timelines are agreed before work starts. Ask for a written scope with a named set of pipelines and exit criteria, and ask what changes the schedule.
What does your team need to provide during an orchestrator implementation?
Expect to provide access and decisions, not just budget. Typical inputs include cloud or data center accounts where the platform will run, network paths to data sources, a PostgreSQL or MySQL instance that meets the vendor's version list, identity provider details for SSO, and named owners who can approve pipeline changes and production access. Your security team should also confirm secrets handling, logging destinations and backup targets. Ask the implementation partner to list each input and owner in the written scope so gaps show up before work begins.
When is a paid managed orchestration layer worth pricing?
A paid managed layer is worth pricing when the features it adds solve a problem you already have, such as SSO and RBAC across many teams, audit logs for compliance evidence, or lineage across many assets. Dagster+ bundles RBAC, SAML SSO, catalog search, column lineage, branch deployments and audit logs, and Prefect Cloud adds RBAC, audit logs and SSO that self-hosted Prefect server does not include. Compare those features against what you would have to build and operate yourself, and ask each vendor for current pricing for your user count, since the figures on vendor pricing pages can change.
Do we need a migration plan if we still run Airflow 2 workloads?
Yes, plan the migration now. Airflow 2 reached end of life on April 22, 2026, and after that it receives no fixes or support, according to the Airflow version support table. Airflow 3 was first released on April 22, 2025, and the current patch release is 3.3.2. Airflow 3.0.0 also dropped the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor, so check your executor setup before upgrading. Inventory your DAGs, plugins and connections, test them on Airflow 3, and agree a cutover window with the teams that depend on them.
What does a fixed-scope engagement with DeployOpen cover?
A fixed-scope engagement covers the work named in a written scope: the platform to plan, implement, integrate, harden, back up, document and hand over, along with the deliverables and timelines agreed before work starts. DeployOpen plans, implements, customizes, integrates, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. Ongoing support, upgrades and continuous testing are available as retainers. DeployOpen does not publish a price list, so a quote follows from the written scope. Ask for the list of excluded items too.
How DeployOpen scopes orchestrator selection and implementation
DeployOpen plans and implements self-hosted orchestrators inside your own cloud accounts, data centers or air-gapped networks. Engagements start with a written scope that fixes deliverables and timelines before any work begins, so the orchestrator choice is tested against your actual pipelines, identity rules and hosting constraints.
How we scope
- Inventory existing pipelines, schedules, backfill needs and the teams that author and operate them.
- Confirm hosting target, database backend, identity requirements and which paid features your team actually needs.
- Agree the orchestrator choice, deliverables and timeline in writing before implementation starts.
How we build
- Deploy the chosen orchestrator with the production metadata database, scheduler or daemon layout and executor you have agreed.
- Port representative pipelines, configure connections and secrets, and test scheduling, retries and backfills against your data.
- Harden the deployment and run security testing on the orchestrator, its web interface and its integrations.
How we hand over
- Document the architecture, upgrade path, backup and restore steps, and the operating runbook for your platform team.
- Walk your engineers through authoring patterns and deployment workflows so they can extend pipelines without us.
- Provide the written record of scope, configuration and test results for your own review and any compliance assessment.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer if your team wants DeployOpen to keep operating the platform.
- Plan version upgrades around the vendor's published support windows, which you should confirm for the release you choose.
- Quote follows from the written scope, so ask for the cost of each deliverable, the recurring retainer and any licensing you will pay the vendor directly.
