Which to choose when Step Functions runs your data jobs
If most of your state machines run nightly or hourly data jobs, the first question is how much of that work is scheduled and how much is triggered by events. Apache Airflow fits scheduled, DAG-shaped batch work: extract, transform and load chains with clear dependencies, reruns of past intervals and backfills across date ranges. Step Functions fits event-driven, AWS-native orchestration, where a file landing in Amazon S3, a message on a queue or an API call starts a short workflow and the team wants very little to operate. Airflow is released under the Apache License 2.0 (Apache-2.0), and the project itself offers no paid edition, so the license cost is zero. Your costs will be infrastructure, the people who run it and any commercial support you buy from a vendor.
Three decisions usually settle the choice. First, count your workloads by trigger type. If most pipelines run on a timetable and need backfills, Airflow's scheduler and DAG model match that shape. If most run on events and branch on the outcome of each AWS call, Step Functions' state machines are a closer fit. Second, decide how much you want to own. Airflow means a Python codebase, a metadata database and a scheduler you patch and monitor. Confirm that your team can keep up with the supported versions listed in Airflow's version life cycle, since older lines move to limited maintenance and then end of life. Third, measure how deep your AWS integrations go. Step Functions calls many AWS services directly with little glue code, while Airflow reaches them through provider packages and operators that your team must choose, version and test.
Before you commit, run a short inventory: list each state machine, its trigger, its schedule, its retry and backfill needs and the AWS services it calls. Then ask your team to confirm which Airflow prerequisites apply to your environment, including the supported Python and PostgreSQL versions and the production database requirement, and to estimate the operations time the metadata database and scheduler will need. Pipelines that are mostly event-driven and AWS-native may stay on Step Functions, with Airflow reserved for the scheduled batch estate. A split like that is a valid outcome and often lowers both risk and migration effort.
Which option fits which workload
This table maps common workloads to the option that usually fits better and gives the reason for each choice. Read each row against your own pipelines, and treat the weaker-choice column as the condition that would change the answer.
| Workload | Better fit | Reason and where the other option is weaker |
|---|---|---|
| Nightly ETL across warehouse and lake tables | Apache Airflow | Airflow's Python DAGs, schedule intervals and task dependencies are built for batch data work. Step Functions can run the same jobs, but it is weaker when the logic lives in many Python data libraries and needs local testing. |
| Backfills of historical date partitions | Apache Airflow | Airflow models a DAG run per logical date, so reruns over past partitions are a native operation. Step Functions has no equivalent calendar-based backfill, so teams usually script it and track the state themselves. |
| Event-triggered microservice sagas | AWS Step Functions | Step Functions reacts to events, API calls and callback tokens with short, service-to-service steps. Airflow is a scheduler first, so long-lived event waits and compensating steps are a weaker fit. |
| Pipelines that run only on AWS services | AWS Step Functions | Native service integrations, IAM-scoped permissions and no servers to patch keep the operating load low. Airflow can call the same services, but the team then runs and secures the scheduler, metadata database and workers. |
| Multicloud or on-premises data | Apache Airflow | Airflow is Apache-2.0 licensed and runs on your own Kubernetes, VMs or data center, with no AWS dependency. Step Functions is weaker here because the control plane and most integrations are tied to AWS. |
| Mixed batch and event-driven processing in one estate | Both, scoped by job type | Use Airflow for scheduled data pipelines and Step Functions or a second engine for event flows. Running two orchestrators adds cost, so confirm that the split is worth the extra operating surface. |
| Regulated pipelines needing audit trails of each run | Depends on evidence required | Both keep run histories, but retention, export and access control differ. Confirm log retention, metadata database backups and access review procedures in your own environment before choosing. |
Where the data and control sit with each option
Moving from Step Functions to self-hosted Airflow changes where metadata, logs and access decisions live. The cards below separate what AWS manages from what your team must run, secure and document. Check each one against your own data classification and audit scope before you commit.
Metadata database location
Airflow keeps DAG runs, task instances, connections metadata and scheduler state in a relational metadata database that you host. Step Functions keeps execution history inside the AWS-managed service. For a self-hosted Airflow, the metadata database must be PostgreSQL or MySQL, since the production guide warns that SQLite can lead to data loss. Confirm where that database runs, who holds its backups and how long run history is retained.
Task logs and payload storage
Task logs, XComs and any data passed between tasks are written wherever your workers and log configuration direct them. With Step Functions, execution history and payloads sit in AWS. With Airflow, you decide whether logs go to local disk, object storage or a log platform. Confirm that logs do not capture credentials or personal data, and set retention rules that match your policy.
What runs in your VPC
A self-hosted Airflow deployment runs the scheduler, web server, workers and metadata database in your network, whether on Kubernetes, EC2 or another platform. Step Functions is an AWS-managed control plane that calls AWS services through IAM roles. Map each Airflow component to a subnet, security group and owner, and record which AWS APIs the workers call and from which roles.
Identity through the auth manager
Airflow's auth manager controls who can view, trigger and edit DAGs and connections. Step Functions relies on IAM policies on state machines and executions. Plan how your identity provider maps to Airflow roles, how service accounts are scoped, and how credentials for AWS connections are stored, preferably in a secrets backend rather than in the Airflow UI. Verify the auth manager options supported by the release you select.
Patch and version control evidence
Airflow's support policy means you must track lifecycle dates and apply security fixes yourself. Limited-maintenance versions receive only security and critical fixes, and end-of-life versions receive none. Keep a record of the running version, patch history and upgrade approvals. Step Functions patching is handled by AWS, so that evidence changes from vendor attestations to your own change logs.
Compliance evidence you collect
DeployOpen can help prepare controls, configuration baselines and evidence packs for your compliance assessment, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations. Expect to collect access reviews, backup restore tests, log samples and change records yourself, and confirm with your auditor which evidence they will accept for each control.
What a self-hosted Airflow deployment needs in place
A self-hosted Airflow deployment is more than one process. The scheduler parses DAG files, decides which tasks are ready and hands them to workers, the webserver (the API server and UI in Airflow 3) serves the interface, and the executor determines where tasks actually run. Airflow supports several executors, including LocalExecutor, CeleryExecutor, KubernetesExecutor and the AWS Batch and ECS executors, and since Airflow 2.10.0 more than one can run in the same deployment. Decide the executor before you size anything, because it sets the worker topology, the queueing layer and the on-call burden. Note that CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so teams migrating from older hybrid setups need a new design.
The metadata database is the component to get right first. Airflow's production deployment guide says the default SQLite backend is for testing only and can lose data in production, so plan for PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release from the start. MariaDB is not supported as the metadata database. Check the exact versions against the Airflow prerequisites for the Airflow release you choose, along with the supported Python range (3.10 to 3.14 for Airflow 3.3.2) and, if you run on Kubernetes, the supported cluster versions (1.30 to 1.35). Confirm which managed database service and version your cloud account offers before you commit to a design.
The scheduler can run as several instances for performance and resilience, because Airflow uses database row-level locks to keep them from scheduling the same work twice. Treat that as a design decision to test, not a default: it depends on the database handling the extra lock contention. The prerequisites recommend at least 4 GB of RAM, which is a floor for the scheduler and webserver rather than a sizing estimate for your workload. Ask your team or DeployOpen to validate CPU, memory and connection pool settings with a load test that reflects your DAG count, task concurrency and backfill plans before go-live.
Why the Airflow version you start on matters
Airflow's version lifecycle and executor model decide how long your deployment stays supported and which scheduling patterns carry over from Step Functions. The cards below cover the decisions that are hardest to reverse after a migration starts.
Airflow 3 is the current line
Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. Starting a new deployment on the 3.x line means planning against the version that receives ongoing fixes. Confirm the current patch against the version life cycle table before you pin a version in your build.
Airflow 2 has reached end of life
Airflow 2 entered limited maintenance on October 22, 2025, and reached end of life on April 22, 2026. Limited maintenance covers only security and critical bug fixes, and end-of-life versions get no fixes or support. Any Airflow 2 DAGs you inherit from a Step Functions migration need an upgrade plan, not a copy of the current environment.
Upgrade path from Airflow 2
Moving from Airflow 2 to 3 changes operators, imports and several configuration defaults, so DAG code usually needs review. Read the upgrade guide early and inventory which DAGs use deprecated features. Your team should run the upgrade in a non-production environment and test scheduling, retries and backfills before cutover.
Hybrid executors are gone
The CeleryKubernetesExecutor and LocalKubernetesExecutor hybrids are not supported as of Airflow 3.0.0. If a team copies an older design that splits tasks between Celery and Kubernetes, it must choose a single executor or use the multiple-executor support added in Airflow 2.10.0. Map each Step Functions task type to an executor before writing the platform design.
Check prerequisites before choosing a database
Airflow 3.3.2 supports PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, but does not support MariaDB as a metadata database. SQLite is for testing only, and production runs need an external database. Ask your DBA team to confirm the supported version, backup window and high availability design before you size the metadata store.
Confirm the table before planning
Version support dates and patch numbers change as the project publishes releases. Before you set a migration timeline or a budget line, check the current support table and the release history in the upstream repository. Record the version you approved and the date you checked it, so the decision can be reviewed at the next renewal.
How a move from Step Functions to Airflow usually runs
The sequence turns scheduled state machines into reviewed DAGs with equivalent access, a verified comparison window and a documented cutover, so the team can retire the Step Functions workflows without losing runs or history.
Inventory state machines and service integrations
Export every state machine definition, its schedule or trigger, the IAM role it assumes, and each AWS service it calls. Record which workflows are batch jobs, which wait on callbacks or external events, and which are owned by which team. A team lead and the platform engineer who owns the AWS accounts usually do this together. The step is done when every state machine has a row in one migration inventory with its trigger, integrations, owner and downstream consumers, and the team agrees which ones are in scope for Airflow.
Translate control flow into DAG tasks
Map each Task state to an operator or a Python task, each Choice state to branching logic, each Wait state to a schedule or a sensor, and each Parallel or Map state to parallel task groups or dynamic task mapping. Data engineers usually carry out the translation and record any state that has no direct equivalent. The step produces a reviewed DAG for each workflow and a short note on every pattern that changed shape. Confirm the mapping against the Airflow version your platform runs, since operator and task APIs differ between releases.
Redesign callback and event-driven steps
Step Functions callback patterns, where a task pauses until an external system returns a token, usually need a different design in Airflow. Replace them with sensors that poll for a condition, deferrable operators that release the worker while they wait, or triggers from an external event source. This is the step most likely to need design decisions from the owning team. It is complete when each callback has a named replacement, a timeout behavior and a test that shows the workflow resumes when the external condition is met.
Rebuild access through connections and secrets backends
Step Functions uses the IAM role attached to each state machine. In Airflow, access comes from connections, and credentials are best stored in a secrets backend that your organization controls. Work with the identity and cloud teams to decide how each DAG authenticates to AWS, whether through a role the Airflow workers assume or through stored credentials, and document the least-privilege policy for each connection. The step is done when each DAG runs with only the permissions its tasks need and a review confirms that no long-lived keys sit in DAG code.
Run both systems for a comparison window
Run the Airflow DAGs alongside the existing state machines, with the Airflow versions and worker setup confirmed against the supported prerequisites. Compare outputs, task durations, failure handling and alerts for each workflow. Where a workflow writes to a shared target, keep one side in dry-run or write to a staging location until the results match. The window ends when the team has reviewed agreed comparison criteria for every workflow and signed off on the differences. Ask the implementation partner to write the criteria down before the window starts.
Backfill historical runs and verify results
Use Airflow's backfill and catchup behavior to rebuild historical runs for the data periods the business needs, starting with the workflows whose outputs feed reports or regulated records. Check row counts, checksums or reconciliation totals against the outputs produced by Step Functions for the same periods. Keep the exported execution history from the Step Functions side as evidence. The step is complete when every backfilled period has a reconciliation record that the data owner has accepted.
Cut over and retire the state machines
Switch each workflow's schedule to Airflow in a planned order, disable the matching Step Functions state machine so it cannot start a duplicate run, and watch the first production runs closely. Confirm that alerts reach the on-call channel and that the runbook names the owner for each DAG. Retire the state machines only after the retention and audit requirements for their exported history are met. The step is done when no workflow runs in both systems, the cutover log is filed and the old state machines are removed from the AWS accounts.
Questions buyers ask before switching from Step Functions to Airflow
What does it cost to switch from Step Functions to Airflow?
The switch cost is mostly people time, plus the infrastructure to run Airflow, and the software itself carries no license fee because Apache Airflow is released under the Apache-2.0 license with no paid edition from the project. Costs fall into four groups: rewriting state machines as Python DAGs, building the metadata database, scheduler and workers, setting up monitoring and backups, and ongoing operations. Step Functions charges per state transition or per request, so the saving depends on your current bill and how much you spend on the new platform's servers and staff. Ask any implementation partner for a fixed-scope quote built from your state machine inventory, and confirm the infrastructure line items with your cloud team.
How long does a Step Functions to Airflow migration take?
The timeline depends on how many state machines you have, how many use AWS service integrations and callbacks, and how much history you must keep. Airflow itself needs a production-grade deployment first, which means a PostgreSQL or MySQL metadata database, a scheduler, an executor suited to your workload and a clear upgrade path. Airflow 2 entered limited maintenance on October 22, 2025 and reaches end of life on April 22, 2026, so a new build should target Airflow 3, first released on April 22, 2025. Run both systems side by side during cutover so you can compare outputs before retiring anything. Ask for a schedule in the written scope rather than a generic duration.
What skills and ongoing work does the team need after switching?
Your team needs Python skills for DAG authoring, database and Kubernetes or container operations for the platform, and on-call coverage for scheduler and worker incidents. Airflow runs on a supported matrix of Python 3.10 to 3.14, PostgreSQL 14 to 18 or MySQL, and Kubernetes 1.30 to 1.35 for the current 3.3.2 release, so you must keep these components patched. Recurring work includes Airflow upgrades, dependency updates, backup testing and capacity planning. Airflow's documentation recommends at least 4 GB of RAM and warns against SQLite in production. Confirm staffing numbers with your own platform leads, since they depend on how many DAGs and environments you run.
Can our own engineers do the migration without an outside partner?
Yes, a capable platform team can do the migration in-house, provided it has someone who has run a Python workflow scheduler in production and time to test failure paths. The hard parts are not the DAG syntax but translating retry policies, Choice and Map logic, and AWS callback patterns, along with securing the metadata database and access controls. Airflow's auth manager documentation and production deployment guide cover settings you must decide, and its upgrade guide for Airflow 3 lists changes that affect existing code. Many teams bring in outside help for the design and cutover while keeping operation in-house. Whichever route you choose, put the scope and acceptance tests in writing before work begins.
When is Step Functions still the better choice than Airflow?
Step Functions remains the better fit when your workflows are event-driven, short-lived and tightly coupled to AWS services such as Lambda, EventBridge, SQS, Step Functions Express callbacks or Glue jobs, because AWS manages the state, scaling and retries for you. It also suits teams that do not want to run a scheduler, database or workers. Airflow is strongest for scheduled batch and data pipelines with backfills, dependency graphs and Python-native logic. If most of your state machines respond to events rather than calendar schedules, a move to Airflow adds operations without solving a problem. Consider a split approach, keeping event-driven flows on Step Functions and moving only scheduled data jobs.
What changes for AWS integrations and IAM when moving to Airflow?
Airflow reaches AWS through provider packages and connections rather than Step Functions' built-in service integrations, so each integration must be rebuilt and tested, and IAM roles move from state machine execution roles to the identities your Airflow workers use. Airflow ships AWS Batch and ECS executors, which can run tasks on AWS compute. Any service integration that relied on Step Functions' managed retry, timeout or callback behavior needs an explicit equivalent in your DAG code. Plan for secrets handling, network paths to private resources and audit logging of connections. Confirm each integration against the Airflow provider documentation for your version before cutover.
What are the main risks of running Airflow in our own infrastructure?
The main risks are data loss from an unsuitable metadata database, outages from a single scheduler, and security gaps from default settings. Airflow's production guide warns that the default SQLite backend is for testing only and can lead to data loss, so use PostgreSQL or MySQL with tested backups. Airflow supports running more than one scheduler at once, using database row-level locks, which improves resiliency when configured correctly. Executor choices also matter: the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported from Airflow 3.0.0. Have your security team review authentication, secrets and network exposure before go-live.
When Step Functions still fits better than Airflow
Step Functions still fits better when the workflow is short, triggered by an event and tightly bound to AWS services. A state machine that reacts to an S3 upload, an EventBridge rule or an API call, calls a Lambda function or DynamoDB, and finishes in seconds or minutes has no scheduling problem for Airflow to solve. Airflow's scheduler, DAG parsing and metadata database add fixed moving parts that a short event-driven flow does not need. Before choosing, check the workflow's trigger pattern: if most runs start from events rather than from a cron-style schedule, and the steps are AWS API calls, Step Functions keeps the orchestration in the same account, IAM model and billing as the services it coordinates. Confirm current Step Functions quotas and pricing with AWS, since those drive the cost of high-volume event traffic.
The second case is a team without Python platform skills or without capacity to run a data platform. Airflow DAGs are Python code, and a production installation needs a supported metadata database such as PostgreSQL or MySQL, a scheduler, workers and an executor such as KubernetesExecutor or CeleryExecutor. The Airflow prerequisites list specific supported Python, PostgreSQL, MySQL and Kubernetes versions, so someone must own those upgrades. If the team has no one who can operate that stack, a managed state machine with no servers to patch is the lower-risk choice, and the team should weigh the cost of running Airflow honestly, including on-call time for the scheduler, not only the license, which is Apache-2.0 at no charge. Ask whether the people who would maintain Airflow are already on staff, and whether they can support it through the Airflow 3 upgrade path described in the upgrading guide.
Step Functions also fits when the operating appetite is low and the workflow is a small part of a larger serverless application. In that setting the state machine is versioned with the application in infrastructure-as-code, deployed with the same pipeline, and monitored with CloudWatch and X-Ray. Airflow would add a separate platform with its own access model, backups and upgrade calendar. Note the trade-off: Airflow's auth manager and role model cover data teams well, but they add a second identity surface to review. A practical test is to write a short inventory of the workflows, marking each by trigger type, run length, number of AWS services called and owning team. Workflows that score as event-driven, AWS-bound and small usually stay on Step Functions, while batch and data-pipeline workflows are the ones to move to Airflow.
How DeployOpen helps evaluate, implement or migrate
DeployOpen is a Chainscore Labs company that plans, implements, hardens, backs up and documents self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a move from AWS Step Functions to Apache Airflow, we start with your existing batch and data jobs and the AWS services they depend on, then build the Airflow deployment around them.
How we scope
- Inventory each state machine, its schedule, retry rules, and the AWS services it calls
- Confirm which DAGs need backfill support and which depend on event triggers
- Agree the scope, deliverables and timelines in writing before any work starts
- Confirm the Airflow version, metadata database and executor model against current supported versions
How we build
- Deploy Airflow on your infrastructure with a production metadata database such as PostgreSQL or MySQL
- Translate state machine logic into Python DAGs with tested retry and failure handling
- Configure AWS connections, IAM access and secrets handling to match your existing security controls
- Test scheduling, backfills and failure paths in a non-production environment before cutover
How we hand over
- Document DAG ownership, deployment steps, configuration keys and runbooks for your operations team
- Provide a mapping from each original state machine to its replacement DAG
- Walk your engineers through monitoring, scheduler health checks and upgrade procedures
- Record the evidence your compliance assessors will ask for, without issuing any audit opinion
How we support
- Ongoing support, version upgrades and continuous testing are available as retainers
- Plan Airflow upgrades around the end-of-life dates published in the support policy
- Keep backups of the metadata database and DAG repository under documented restore procedures
- Review scheduler and executor performance as your job volume grows
