Migrating Step Functions workflows to Apache Airflow: what the move involves
Moving Step Functions workflows to Apache Airflow means rewriting the orchestration logic as DAGs, then deciding which AWS-triggered steps stay where they are. The work is mostly mapping states, transitions, retries, and error paths to Airflow tasks and dependencies, and deciding which Lambda, Glue, EMR, ECS, or Batch steps Airflow should call and which should remain event-driven in AWS. Effort is driven by the number of state machines, how much branching and parallelism they use, how retries and catch paths are configured, and how many of them are shared by other teams. Ask for an inventory of every state machine, its triggers, and its downstream services before agreeing on scope, because a state machine with hidden dependencies on another system is usually the slowest to move.
Apache Airflow is licensed under the Apache License 2.0 (Apache-2.0), and the project offers no paid edition, so the software cost is zero and the cost of the move is labor, infrastructure, and operations. Those costs are one-time for the migration and recurring for running the scheduler, workers, metadata database, and upgrades. The Amazon provider operator guides include operators for AWS Step Functions, Lambda, Glue, EMR, ECS, and Batch. That lets a team call existing AWS jobs from Airflow while individual state machines are retired, so the cutover can happen one workflow at a time instead of all at once.
Confirm early which AWS permissions, VPC routes, and IAM roles the Airflow workers will need to call those services from infrastructure you control, and which Airflow connections will hold those credentials. Plan for two parallel periods: one where Airflow triggers a Step Functions execution and waits for its result, and one where both systems could run the same job. Double runs are a common risk during coexistence, so each workflow needs a single owner and a clear switch date. Some workflows, such as those reacting to queue messages or object-created events, may be better kept event-driven in AWS, and that choice should be made per workflow rather than assumed for the whole estate.
Which Step Functions capabilities map to Airflow and which need redesign
This table compares common Step Functions constructs with their closest Airflow equivalents and marks where the move needs redesign rather than translation. Read each row as a starting point for a design review, then confirm exact operator and feature behavior against the current Airflow 3 and Amazon provider documentation. The main takeaway is that control flow usually translates, while start triggers, execution history and retry semantics need explicit decisions.
| Step Functions capability | Closest Airflow approach | Translation or redesign |
|---|---|---|
| Task state calling Lambda, Glue, ECS or Batch | Operator from the Amazon provider for the target service, wrapped in a task | Maps directly; confirm the operator covers your invocation parameters |
| Choice state branching on input | Branching operator or a task that returns the next task id | Maps with rework; branch logic moves into Python and is tested as code |
| Map state over an array | Dynamic task mapping over a list produced by an upstream task | Maps; confirm concurrency limits and per-item failure handling |
| Parallel state with fixed branches | Explicit parallel tasks or a TaskGroup for related steps | Maps directly; branches become visible dependencies in the graph |
| Wait state with a timer or timestamp | Sensor or deferrable wait, or a schedule-based start | Maps with rework; long waits should use deferrable waits so workers are not held |
| Retry and Catch on a state | Task retries, retry delay and failure callbacks or trigger rules | Redesign; retry counts and backoff settings must be re-entered and reviewed |
| Input and output passed between states as JSON | XCom for small values, parameters, or assets for shared datasets | Redesign; large payloads should move to storage with only references passed |
| Event-driven start from EventBridge or other AWS events | Event trigger with an AssetWatcher, where the trigger inherits from BaseEventTrigger | Redesign; confirm which event sources have compatible triggers before committing |
| Express versus Standard execution history | Airflow task instance and DAG run history kept in the metadata database | Redesign; plan retention and cleanup because history is stored in your own database |
| Long-running executions lasting months | Scheduled DAG runs with a defined retention window and external state where needed | Redesign; confirm how long run history and logs must be kept for audit |
Documented paths for calling and replacing AWS steps from Airflow
Airflow gives you two documented ways to work with existing AWS workflows: operators from the Amazon provider that start Step Functions, Lambda, Glue, EMR, ECS and Batch jobs, and the Airflow REST API for triggering DAG runs from outside. No automated converter from state machines to DAGs is assumed here, so every translation is a manual design step. The cards below cover what each path does and what to check before relying on it.
Amazon provider operators for Step Functions
The Amazon provider includes operators that start a Step Functions state machine from a task and can wait for its result. This is the simplest way to run an existing state machine during coexistence, so the DAG owns the schedule and dependencies while AWS still executes the steps. Check which operator modes your version of the provider supports, how the task reads execution status, and what happens to retries when a DAG task is rerun after a partial failure.
Lambda and Glue operators
Lambda and Glue operators let a DAG invoke functions and ETL jobs that already exist, without rewriting their code. Each call becomes a task with its own retry policy and log, which makes failures easier to locate than a single state machine execution. Confirm the timeout behavior for long Glue jobs, how payloads are passed between tasks, and whether the task waits for completion or returns immediately.
EMR, ECS and Batch operators
Operators for EMR, ECS and Batch cover the compute steps that many state machines delegate to. Using them keeps the cluster, container or batch job definitions in AWS while Airflow controls ordering and alerting. Verify IAM permissions for each operator, the connection configuration used by the DAG, and how job logs are linked back into the Airflow task view so operators do not need to switch consoles.
Airflow REST API triggers
The Airflow REST API lets systems outside Airflow create DAG runs, which is useful when an existing AWS event, API call or application currently starts a state machine. Pointing those callers at the API lets you move the trigger before moving the work. Plan authentication for the caller, the API access controls on your Airflow deployment, and the handling of duplicate triggers when a caller retries.
Manual translation of state machines
No automated tool is assumed to convert Amazon States Language definitions into DAG code, so each state machine is rewritten by hand. Map each Task, Choice, Parallel and Map state to an operator, task group or dynamic mapping, and record every retry, catch and timeout setting so none is lost. Review the translated DAG with the team that owns the state machine before cutover, since behavior differences in error paths are the most common source of surprises.
Event-driven flows that stay in AWS
Some workflows react to messages or events and may be better left in AWS as event-driven pipelines. Airflow can consume events through event-driven scheduling, but that depends on trigger types it supports, so confirm the message source and trigger class before moving a flow. Decide case by case, and keep the event path in AWS when latency or fan-out requirements fit it better than a scheduled DAG.
How the migration sequence runs from inventory to decommissioning
The sequence turns an unstructured set of state machines into a controlled move, where each workflow is classified, piloted, run in parallel and cut over on its own schedule. It ends with the AWS resources removed and a record of what replaced each workflow.
Inventory state machines, triggers and owners
Export every state machine definition in each account and region, and record what starts it: EventBridge rules, schedules, API calls, other state machines or manual runs. For each one, capture the owner, the Lambda functions, Glue jobs, EMR, ECS or Batch tasks it calls, the data stores it reads and writes, and the downstream consumers that depend on its output. The step is done when the team can name the owner and the upstream and downstream dependencies of every workflow, including the ones nobody has run in months. Inventory also surfaces Step Functions that are triggered by other Step Functions, which must move as a group.
Classify each workflow as move, keep or split
Sort each workflow into one of three groups. Move workflows are scheduled, batch-oriented pipelines with clear task dependencies that suit Airflow's model. Keep workflows are tightly coupled to AWS events, such as S3 notifications or message-driven fan-out, and are better left event-driven in AWS. Split workflows contain both kinds of work and are divided at a clear boundary. Record the reason for each decision with the owner's sign-off. The output is a classification register that the team reviews before any code is written.
Stand up Airflow on a supported stack
Install Airflow 3 on infrastructure you control using a Python version and a PostgreSQL version from the Airflow 3.3.2 prerequisites list. Run the API server and the DAG processor as separate components, and configure more than one scheduler only after the single-scheduler setup is verified. Confirm that the metadata database is not SQLite or MariaDB. Connect the AWS provider with the IAM roles and connections the pilot needs, and confirm the executor choice against the expected task volume. The step is done when a test DAG runs end to end, its logs are readable and the database backup job has been restored once in a test environment.
Pilot one low-risk workflow
Choose one workflow that is scheduled, has few dependencies, has a known good output and can be rerun safely. Translate its states into Airflow tasks, keeping Lambda, Glue and other AWS calls as provider operators, and compare each rerun's outputs with the Step Functions output for the same input. Set retries, alerting and timeouts to match the original behavior, and document every difference. The pilot is complete when outputs match over the agreed number of runs and the operations team has handled at least one failure through the Airflow runbook.
Run Airflow alongside Step Functions
During the transition, let Airflow call the existing state machines through the Amazon provider's Step Functions operators while the translated DAG is validated. This keeps the original logic running as the source of truth and gives the team a single place to see both paths. Make sure only one path writes to production targets at any time, using flags, separate output locations or a paused schedule, so that double runs do not corrupt data. The parallel run ends when the owner confirms the results match and the dependency map shows no unplanned callers.
Cut over workflow by workflow
Move each classified workflow on its own date, starting with dependencies that have the fewest downstream consumers. For each cutover, pause the Step Functions trigger, enable the Airflow schedule, confirm the first production run and update every downstream consumer that reads from the old path. Keep the state machine definition and its trigger disabled rather than deleted until the rollback window the owner agreed has passed. Each workflow closes with a signed cutover record listing the trigger changes and the verification evidence.
Decommission state machines and IAM roles
Once the rollback window has closed for a workflow, delete its state machine, its EventBridge rules and any log groups the retention policy allows you to remove. Review the IAM roles that the state machines assumed and remove permissions and roles that no remaining workflow uses. Confirm the change in the inventory register and retain the final definitions and run history in an archive that the audit and recovery procedures cover. The sequence is complete when the inventory shows no active Step Functions workflows that belong to the migrated scope and each removed role has a recorded reason.
Which workflows should stay event-driven in AWS
Some Step Functions flows should stay in AWS even after the scheduled data pipelines move to Airflow. The clearest candidates are workflows that must react within seconds to AWS service events, such as an object landing in an S3 bucket that a Lambda function must validate and route immediately, or an EventBridge rule that starts a state machine on a resource state change. Airflow is a scheduler and orchestrator for batch and asset-driven work, so a flow whose latency requirement is measured in seconds and whose trigger is native to an AWS service is usually cheaper and simpler to keep where the event originates. Tightly coupled Lambda chains with retry semantics that already work, and flows owned by application teams rather than the data platform, are also reasonable to leave alone. Map each state machine against its trigger, its latency requirement and its owner before deciding anything.
Airflow 3 offers a middle path. An AssetWatcher attaches an event trigger to an asset, so when the trigger fires, the asset is updated and downstream DAGs that depend on it can run. The trigger class must inherit from BaseEventTrigger, and the Airflow documentation names Kafka, Amazon SQS, Google Pub/Sub and Azure Service Bus as message broker sources. Confirm with your own design which of these sources, and which trigger types, your team will actually use, since the documentation lists sources without guaranteeing a particular AWS event path. A common pattern for an illustrative design is to publish S3 or EventBridge events to SQS and let Airflow react to the queue, but test delivery behavior, duplicate handling and the trigger's own failure modes before depending on it.
Decide per workflow, then record the boundary. Workflows that stay event-driven in AWS need an agreed interface with the Airflow estate, such as an SQS queue, an EventBridge target or an asset that a DAG consumes, so that operators can see both sides in one place. Ask the team who owns the trigger's IAM role, dead-letter queue and alerting when the event path breaks, because those controls do not move with the DAG. Revisit the boundary when the Airflow version changes, since event-driven scheduling is a newer Airflow 3 capability and its trigger classes and behavior should be checked against the release in use.
What drives the effort and cost of moving Step Functions workflows
This table lists the factors that change how much work a Step Functions to Airflow move takes, and separates one-time migration work from recurring Airflow operating costs. Use it to structure a request for a written scope, where each item should be itemized with its own effort estimate.
| Driver | Why it changes effort or cost | How to keep it down or what to ask for |
|---|---|---|
| Number of state machines | Each state machine needs mapping, a DAG design, tests and a cutover step, so the count sets most of the one-time work. | Ask for a per-state-machine line in the scope, grouped by domain so shared patterns are built once. |
| Nested and parallel branches | Nested state machines and parallel states often turn into TaskGroups or dynamic task mapping, which need more design review and testing. | Identify nested calls early and decide which ones become separate DAGs, which are inlined. |
| Map state concurrency | Map states with high concurrency or large item sets change how Airflow dynamic task mapping and worker capacity are sized. | Record the current MaxConcurrency settings and typical item counts, and ask how each will be set in Airflow. |
| Error handling and retry policies | Retry, catch and fallback paths in the state language must be translated into task retries, trigger rules and alerting, which is detailed work per branch. | Export the Retry and Catch definitions for every state and ask the partner to map each one explicitly. |
| Glue and EMR job count | Glue and EMR calls can usually be kept as operators in Airflow, but each job needs its connection, role, parameters and wait behavior checked. | Inventory each job with its IAM role and wait pattern. Ask whether Airflow waits in deferrable mode or occupies a worker. |
| Lambda code kept or rewritten | Lambda functions that stay in AWS need an invocation path from Airflow. Functions that move into tasks need dependency isolation, packaging and tests. | Mark each Lambda as keep, call or rewrite. Ask for rewrite work to be priced separately from invocation work. |
| IAM and cross-account access | Airflow runs with its own identity, so each AWS account and role needs a connection, trust policy and least-privilege review. | Ask for a list of accounts and roles, and whether the scope includes the security review of those trust relationships. |
| Number of environments | Development, staging and production each need connections, secrets, testing and promotion, and parallel running multiplies the checks. | Ask how many environments the scope covers and whether parallel runs are included. |
| One-time migration work | Mapping, DAG development, testing, the parallel run and cutover are paid once for the project, and the quote should show them as separate items. | Request a written scope that itemizes mapping, build, test, parallel run and cutover with acceptance criteria. |
| Recurring operating costs | Airflow hosting, the PostgreSQL or MySQL metadata database, upgrades, patching and support continue after cutover. Airflow 3 is required on a supported version, and the project's own code is free under Apache-2.0. | Ask the partner to separate recurring infrastructure from support and upgrade retainers, and to state who pays for each. |
Where Step Functions migrations usually go wrong
Most problems in a Step Functions to Airflow move appear during the parallel run, when both systems can act on the same data, and during the switch to a production metadata database. The cards below cover the risks that repeatedly surface in these projects, with a prevention step for each. Treat them as a checklist to review before the first scheduled run in Airflow.
Double execution during parallel run
While both Step Functions and Airflow are enabled, the same source data can be processed twice, leading to duplicate loads, double notifications or repeated writes to downstream tables. Decide per workflow which system owns the schedule and which one is paused, and confirm this in writing before the parallel run starts. Use one owner per workflow at any moment and make each Airflow task idempotent so a rerun does not change results. Check that Lambda and Glue jobs triggered from either side use the same input partitions.
Lost retry semantics
Step Functions retry blocks, catch branches and timeouts each behave in their own way, and a straight port often loses them. Map every Retry and Catch definition to explicit Airflow task settings for retries, retry delay, execution timeout and trigger rules, then test failure paths deliberately. Note which steps are safe to retry, since a retried Glue job or Lambda call with side effects can duplicate work. Document the mapping so reviewers can compare both definitions line by line.
Timezone and schedule differences
Step Functions schedules come from EventBridge rules, which are often set in UTC, while Airflow DAGs define schedules and logical dates that the team may read in local time. Daylight saving changes and the difference between logical date and run start can shift a job by hours or cause a skipped or doubled window. Confirm the timezone of each source schedule, set the DAG timezone explicitly, and compare the next several planned runs in both systems before cutover.
Unmapped IAM permissions
A Step Functions state machine runs under its own execution role, and an Airflow worker runs under whatever credentials it is given, which often differ in scope. Tasks that worked in AWS can fail in Airflow with access denied errors, or worse, succeed with broader rights than the original role allowed. Inventory each state machine's execution role, grant the Airflow worker only the actions each task needs, and test every task path in a non-production account first.
Secrets moved insecurely
Step Functions often passes configuration through state input or environment settings, and teams sometimes copy those values into DAG files, variables or connection strings during the move. Plaintext credentials in DAG code, logs or the metadata database create exposure that outlasts the migration. Store secrets in a managed backend your organization controls, reference them by name from DAGs, and check that task logs do not echo them. Confirm the chosen secrets backend with your security team before any DAG reads a production credential.
Running Airflow on SQLite or an unsupported database
Airflow's prerequisites recommend PostgreSQL or MySQL for the metadata database, and the production guidance warns that SQLite is for testing only and can cause data loss in production. MariaDB is not supported as the metadata database. A pilot built on SQLite can work for weeks and then fail when concurrency rises or when multiple schedulers run. Choose PostgreSQL 14 to 18 or a supported MySQL version from the start, and confirm the version against the prerequisites for your Airflow release.
Questions teams ask before moving Step Functions to Airflow
What does a Step Functions to Airflow migration cost, and what drives the price?
Cost is driven by the number of state machines, how many states and branches each contains, how many Lambda and Glue integrations must be preserved, and how much of the Airflow platform has to be built first. Your own team pays for infrastructure, staff time and the cutover window, and those costs recur for as long as you run Airflow. An external partner is paid against a written scope, so ask for a quote that lists each state machine, the target Airflow version, the environments involved and whether a coexistence period is included. Ask separately for the one-time migration work and any ongoing support or upgrade retainer, because they are priced differently. DeployOpen does not publish a price list; a quote follows from the written scope.
How long does moving Step Functions workflows to Airflow take?
Duration depends on the count and complexity of state machines, the number of retry, choice, map and parallel patterns, how many external AWS services each workflow calls, and whether the target Airflow platform already exists. Teams usually move workflows in batches, which means a coexistence period where some pipelines run in Airflow while others still run in Step Functions. Confirm with any provider what is included in the timeline, which environments are covered, and what testing and parallel-run period is assumed. Avoid planning around a single cutover date for the whole estate unless the inventory is small and well understood.
What must our team provide before a migration starts?
The work goes faster and costs less when the inputs are complete. Typically you need:
- Exports of each state machine definition in Amazon States Language, with the version or alias in use
- The IAM roles and policies attached to each state machine and to the Lambda functions and Glue jobs they call
- Names of the owners for each workflow, the people who approve changes and the people who receive failure alerts
- Run history for the last several months, including schedules, input shapes, timeouts and known failure modes
- Access to the AWS accounts, a non-production environment and the data sources each workflow reads and writes
Missing owners are the most common blocker, because someone must decide whether a workflow should move, stay or be retired.
Can our internal team migrate Step Functions to Airflow without outside help?
Yes, many teams can do it with their own staff, provided they have engineers who know Python, Airflow operators, AWS IAM and the data pipelines themselves. The harder parts are usually not the syntax of the DAGs but the decisions: which workflows should stay event-driven in AWS, how retries and timeouts translate, how credentials are handled, and how to prove that a cutover did not cause double runs. An outside partner can take on the inventory, the target platform setup, the test plan and the parallel run, while your team keeps ownership of the business logic. Whichever route you choose, agree on who owns each workflow after cutover.
Do Lambda and Glue jobs need to be rewritten during the migration?
No. The Amazon provider for Airflow includes operators for AWS Step Functions, Lambda, Glue, EMR, ECS and Batch, so existing jobs can be called from a DAG while the state machines are retired. Rewriting is a separate decision that should be made per job, based on whether the code is maintained, whether it has hidden dependencies on Step Functions payload formats, and whether it would be better as a native Airflow task. A common pattern is to keep Lambda and Glue steps unchanged and move only the orchestration. Check IAM permissions, timeouts and the way job results are passed between steps, since those often break when the caller changes.
Is Airflow 3 the right target now that Airflow 2 reached end of life on April 22, 2026?
For a new migration, Airflow 3 is the appropriate target. Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. According to the Airflow version support table, Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it receives no fixes. Building new DAGs on Airflow 2 would mean planning a second upgrade soon after go-live. Airflow 3 changes the architecture, including a separate API server and DAG processor, and removes direct database access from task code, so DAGs written for Airflow 2 need review. Confirm the Python, PostgreSQL and Kubernetes versions you run against the prerequisites page before you commit to a platform design.
How DeployOpen scopes and delivers a Step Functions migration
DeployOpen starts with an inventory of your state machines, the Lambda and Glue steps they call, and the event sources that trigger them, then agrees a written scope before any build begins. The work runs in your own AWS accounts and your self-hosted Airflow, with coexistence and cutover planned together.
How we scope
- We inventory each state machine, its states, retries, timeouts and error paths before choosing a target.
- The written scope lists which workflows move to Airflow, which stay event-driven in AWS, and which are retired.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- We map Task and Choice states to Airflow tasks and dependencies, and keep Lambda, Glue and EMR calls as operator steps.
- We run Step Functions from Airflow during the transition, with IAM roles, connections and secrets set up in your accounts.
- We test each migrated pipeline against the outputs of its Step Functions predecessor before any schedule is switched.
How we hand over
- You receive DAG code, configuration, IAM policy definitions and a runbook for the self-hosted Airflow environment.
- Documentation covers each workflow's mapping, retained event triggers, rollback steps and the cutover record.
- Knowledge transfer sessions walk your platform and data engineers through running and changing the DAGs.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- We can plan the upgrade path for Airflow and the Amazon provider as part of a retainer scope.
- Support scope, response expectations and review cadence are written into the agreement before work starts.
