Moving from Amazon MWAA to self-hosted Apache Airflow

Moving from Amazon MWAA to self-hosted Apache Airflow: what the move involves

If your Amazon MWAA environment has hit its scheduler or worker limits, or your organization needs the Airflow platform inside infrastructure it controls, the move is mostly a platform handover rather than a DAG rewrite. Apache Airflow is released under the Apache License 2.0 and has no paid edition from the project itself, so the license cost of the move is zero and the expense shifts to the people and infrastructure that run it. The effort is driven by configuration parity, connections and secrets, and cutover discipline. Most DAG code can move largely unchanged once the target version and its dependency constraints match what MWAA runs.

Start by confirming the MWAA environment's Airflow version and the exact provider and requirements constraints it uses. The MWAA versions page lists supported releases and notes that MWAA supports only minor-version in-place upgrades and points to its migration guide for major moves, so a major-version change needs a planned path. From Airflow v2.7.2, MWAA requires a --constraint statement in requirements files, so inventory every requirements.txt and pin the target Airflow version with its matching constraints file. Also list every AWS connection, variable and secret reference, since these must be recreated in your secrets backend and tested against the new environment.

The move also transfers operations to your team. MWAA runs the schedulers, metadata database, workers and upgrades for you, while a self-hosted Airflow leaves those with you, including the PostgreSQL or MySQL metadata database, multi-scheduler configuration, worker scaling and each minor and major upgrade. Confirm who will own these roles, how alerts reach them, and how backups are tested before you commit to a cutover date. Plan a parallel run where both environments are active against a non-overlapping set of DAGs or a paused schedule, so each pipeline has one owner at any moment and no double runs reach production data.

What maps cleanly from MWAA and what does not

This table compares how common MWAA features map to self-hosted Airflow, from requirements files and executors to secrets, authentication and multi-team operation. Read the right-hand column as the work your team takes on, and treat each provider package behavior as something to confirm in your own environment before cutover.

Area	Amazon MWAA	Self-hosted Airflow
Python dependencies	requirements.txt, which MWAA adds a --constraint statement to from Airflow v2.7.2	requirements file plus the official constraints file for your exact Airflow version, installed into a built image or virtual environment you control
Plugins and custom code	Plugins zip and startup scripts managed through the MWAA environment	Plugins and dependencies baked into your container image or delivered through your own DAG bundle process, with image builds in your CI
Scheduler count	2 to 5 schedulers set by the environment class	Multiple schedulers are supported using database row-level locks, sized and monitored by your team
Worker capacity	1 to 25 workers managed by MWAA scaling	Executor-dependent: Kubernetes pods, Celery workers, AWS Batch or ECS, sized and scaled by your platform
Executor choice	CeleryExecutor only, as managed by the service	CeleryExecutor, KubernetesExecutor, AWS Batch and ECS executors, and LocalExecutor; several executors can run in one deployment since Airflow 2.10.0
Secrets	Environment-level secrets management in MWAA	Configure a secrets backend for your store (for example, AWS Secrets Manager); confirm provider package support and access policies in your environment
Authentication	MWAA-managed authentication	Choose one auth manager (Simple by default in Airflow 3, FAB or AWS via provider packages); only one can be active at a time
Multi-team mode	Not supported; incompatible with MWAA authentication, CeleryExecutor and environment-level secrets	Available as a deployment option, with team isolation, role design and access limits to plan and test before use
DAG code	Python DAGs deployed to the environment bucket	Most DAG Python carries over, but Airflow 3 blocks direct database session use in task code and removes SubDAGs in favor of TaskGroups, so review and retest

Matching Airflow versions and constraints before the first DAG moves

A team running a supported 2.x environment on MWAA has to cross the major version boundary before its DAGs can run unchanged on Airflow 3. Airflow 2 (current patch 2.11.2) entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, so it gets no fixes from the project. The Airflow 3 upgrade guide requires being on Airflow 2.7 or later and recommends moving to the latest 2.x release first. Confirm which MWAA environment version you run today and whether it is one of the versions MWAA lists as supported.

Start from the latest 2.x line: Move the DAG code to the latest 2.x release before attempting Airflow 3, since the upgrade guide recommends this as the intermediate step.
Run the Ruff checks: Scan DAG code with the AIR301 and AIR302 rules to find removed or renamed features before the code reaches an Airflow 3 scheduler.
Plan for the MWAA major-move path: MWAA supports only minor-version in-place upgrades and points to its migration guide for major moves, so the 2.x to 3.x step is a planned rebuild, not a button press.
Track the end-of-life date for the old line: Any 2.x environment kept after April 22, 2026 runs without fixes, which affects how long a parallel run can safely last.

Before the first DAG moves, check that every custom operator, plugin and import in the DAG code still resolves on the target release.

Dependency pins are where many self-hosted Airflow moves fail quietly. From Airflow v2.7.2, MWAA requires a --constraint statement in requirements files and adds one when it is missing, so MWAA environments already resolve provider packages against a constraints file for their Airflow version. A self-hosted install needs the same discipline: install Airflow and its providers with the constraints file that matches the target version, then add the DAG dependencies on top of that set. Without this, a provider can upgrade and change operator behavior even when the DAG code has not changed.

Use the constraints file for the target version: Install with the constraints published for the exact Airflow release and Python version you run, not a file copied from another release.
Carry over the MWAA requirements list: Export the requirements file from the MWAA environment and reconcile each package against the target constraints, dropping packages the target already includes.
Pin provider packages explicitly: Pin AWS and other provider versions in your own requirements so that a rebuild six months later resolves the same set.
Build one image per release: Bake the constraints into the container image or virtual environment so that scheduler, workers and the DAG processor share one dependency set.
Check the Python version: Airflow 3.3.2 supports Python 3.10 to 3.14, so confirm the interpreter version used in MWAA is inside that range before reusing wheels.

Before the first DAG moves, confirm that every provider and third-party package listed in your requirements resolves against the target constraints file without conflicts.

Documented import paths and export sources to use

Moving from Amazon MWAA starts with collecting what the environment actually holds: DAG code, requirements, variables, connections and pools. The cards below cover the documented routes for getting that material out and the target options for running it. No vendor-specific importer is assumed, so confirm which exports your MWAA version exposes before planning the work.

DAG folder and requirements export

The DAG folder, plugins and requirements file live in the S3 bucket attached to your MWAA environment. Copy them with the AWS tooling your team already uses, and keep a versioned snapshot so the source of truth is clear during the parallel run. Check that the requirements file carries the --constraint statement MWAA adds for Airflow 2.7.2 and later, because the target install depends on it. Record which bucket path and object versions each snapshot came from.

Metadata inventory through the CLI

Use the Airflow command line and REST API against your existing environment to list variables, connections and pools. Export only names, types and non-secret fields first, since connection passwords and secret values should move through your secrets backend, not a spreadsheet. Confirm that the MWAA version you run exposes the CLI and API calls you need, and note any call that returns masked values so you know what must be re-entered by hand.

Secrets and AWS connection review

Connections that point at AWS services often rely on the MWAA execution role rather than stored keys. Map each one to the identity your self-hosted workers will use, such as an instance profile, a workload identity or static credentials you control. Confirm this per connection with your security team, because an export will not show whether a connection depends on the MWAA role.

Official Docker image as the target

The official Airflow image from the apache/airflow repository gives you a base you can extend with your provider packages and pinned constraints. Build it in your own registry and version it alongside your DAG snapshot. Check that the image tag matches the Airflow version you chose, since MWAA environments may run a different release line from the one you target.

Official Helm chart as the target

The Airflow community Helm chart deploys the same official image onto Kubernetes, with values files you keep in Git. It suits teams that already run clusters and want scheduler, worker and API server replicas defined as code. Review the chart values for the database, executor and secret references against your inventory before the first deployment, and confirm the chart version pairs with the Airflow release you selected.

Version and constraint confirmation

Before choosing a target, list the Airflow version your MWAA environment runs and the versions it supports for in-place minor upgrades. Plan the target to match the DAG code's Airflow version first, then move to a newer release as a separate step. Ask the MWAA migration guide and your own environment to confirm which exports are available for that version, and document any gap you must fill manually.

How the parallel run and cutover are sequenced

The sequence moves DAGs from Amazon MWAA to self-hosted Airflow in controlled stages, so each DAG has exactly one active schedule and one writer to production targets at any point. It ends with a decommissioned MWAA environment only after a retention window closes.

Inventory DAGs, connections, variables and secrets
Export the list of DAGs from the MWAA environment, along with each DAG's schedule, owner, upstream and downstream dependencies, and the connections, variables and secrets it reads. Record which production systems each DAG writes to, such as warehouse tables, buckets or APIs. Your team and the MWAA environment owners sign off on this inventory. It is done when every DAG has a row that names its schedule, its writes and its secrets source, and when any DAG without a clear owner is flagged for a decision before the pilot.
Build the target environment on a supported version
Install self-hosted Airflow on a release that is still supported by the project, and choose the Airflow version to match your MWAA DAG code where possible, then plan the move to a current release. Use PostgreSQL or MySQL as the metadata database, since SQLite is not suitable for production. Set up the secrets backend and connections to your own systems before any DAG is copied. It is done when a test DAG runs end to end on the target, the metadata database passes backup and restore checks, and the platform team can show the installed version and provider packages against the constraints file used by the DAG code.
Pilot a low-risk DAG group
Choose a small group of DAGs with low business impact, few external writes and no hard deadlines. Copy them to the target, pause the MWAA copies, and let the self-hosted copies run on their schedule with writes pointed at a staging or shadow location. Compare task logs, retries, run durations and output row counts with the MWAA history. Fix differences in connections, environment variables or dependency versions before continuing. It is done when the pilot group has run through several scheduled cycles with outputs that the data owners accept.
Run both environments in parallel with one writer
Keep the remaining DAGs active in MWAA, and run their copies on the target with writes redirected to shadow tables, buckets or schemas. Compare outputs on a fixed schedule and record every mismatch with its cause. The key control is that only one environment writes to each production target: the copy on the target writes to shadow locations until cutover, and MWAA keeps the production writes. Confirm this by reviewing each DAG's connection IDs and target paths. It is done when the comparison report shows no unexplained differences for the agreed set of DAGs.
Prevent double runs during the switch
For each DAG, change the target side to write to production only after its MWAA schedule is paused. Pause the DAG in MWAA first by setting it to paused, then unpause the DAG on the target only after confirming the MWAA copy shows no queued or running tasks. Check that no manual triggers or external sensors in the other environment will launch the same work. Keep a written change record with timestamps for each DAG. It is done when a daily report shows one scheduled run per DAG per interval across both environments, with no duplicate writes to production.
Cut over by pausing MWAA schedules
Work through the remaining DAGs in the order your change process approves, pausing each DAG in MWAA and enabling it on the self-hosted Airflow with its production write targets. Run a final output comparison for the cutover window. Keep MWAA in a read-only state for history, and do not delete DAGs there. It is done when every DAG in the inventory is active only on the self-hosted environment, and the owners have confirmed the first production cycles.
Decommission MWAA after a retention window
Keep the MWAA environment, its logs and its metadata available for a retention window agreed with your compliance and data owners, so that audits and reruns can use the old history. Confirm the window is long enough for your reporting cycles and any audit requests before deleting anything. Then export the retained run history and logs to storage you control, remove the MWAA environment, and revoke the IAM roles and secrets it used. It is done when the retention window has closed, the exports are verified, and the access and environment removals are recorded in your change system.

Scheduler and worker limits that push teams off MWAA

Amazon MWAA caps the capacity you can buy inside one environment: each environment runs 2 to 5 schedulers and 1 to 25 workers. Teams that outgrow that range, or that need more parsing headroom than a fixed scheduler count allows, usually find the ceiling before they find a fix. Self-hosting removes the ceiling, but it also moves every sizing decision to your team. Before you commit to a design, confirm the exact limits for your MWAA version in the Amazon MWAA Airflow versions page, because those limits are what you are replacing.

Multi-team mode is the other constraint that pushes teams off MWAA. MWAA does not support Airflow multi-team mode, and its documentation states that the mode is incompatible with MWAA authentication, the CeleryExecutor and environment-level secrets management. For an organization that wants separate teams to share one Airflow with separate DAG ownership, that combination forces either several environments or a design that gives up one of those three features. In a self-hosted Airflow, you choose the executor, the auth manager and the secrets backend together, so you can test whether multi-team isolation works with the combination you need before cutover.

Self-hosting gives you the executor choice and the scheduler count, but it does not make the design easier. Airflow supports running more than one scheduler at the same time, coordinated through database row-level locks, so you can add schedulers for resilience and throughput. That only helps if the metadata database can handle the lock contention and query load, and if the DAG processor keeps up with parsing. Airflow 3 also allows only one auth manager at a time, set by the auth_manager option, so a team that wants both separate team access and SSO must decide which manager covers which need. Plan the database sizing, parser capacity and worker pool as one exercise, and test it with your largest DAG set before you move production schedules.

Questions teams ask before moving off MWAA

What drives the cost and effort of moving off MWAA?

The biggest drivers are how many DAGs and connections you have, how many of them depend on AWS-specific operators, plugins or environment-level secrets, and how many teams share the environment. Effort also rises with the Airflow gap you must cross: MWAA supports only minor-version in-place upgrades, so a move from an older line usually means a version step first. On the cost side, you pay for the platform build (infrastructure, database, executor, secrets and SSO), the DAG migration and testing, and the parallel run. Ongoing costs are your own cloud spend and the team time to operate the platform. Ask any quote to list each of these as a separate line, with the Airflow versions in scope and the assumed number of DAGs, so you can compare it with your current MWAA bill and staffing.

How long does a parallel run need to last?

The parallel run should cover at least one full cycle of your slowest or most business-critical schedule, plus the release calendar your data owners depend on, such as a month-end or quarter-end close. Set that window from your own calendar, not from a general rule. During the run, both environments execute the same DAGs against the same inputs, and the team compares outputs, timings and failures before any schedule is switched. Plan for duplicate writes to be blocked in one environment, because double runs are the most common cutover risk. Confirm the window with your data owners before the scope is signed, since it sets how long both environments must be funded and staffed.

What does the move need from our own staff?

Expect three groups to be involved. An AWS administrator is needed to review the IAM roles, VPC networking, S3 DAG and log buckets, and the secrets store that replaces MWAA's configuration. Data owners must confirm which pipelines matter, validate outputs during the parallel run and sign off on cutover. Your on-call or platform operations team must take over alerts, restarts and upgrades once the platform is live, so they should join runbook reviews early. Ask the partner to list these roles by name in the written scope, with the hours each one is needed for, so that your own commitments are visible before work starts.

Can our team do this migration without a partner?

Yes, a capable platform team can do it alone, especially if the environment is small and the team already runs Kubernetes or another container platform. The gaps that usually matter are the Airflow 3 changes that affect DAG code, such as the separate DAG processor and the restriction on direct database access from task code, plus the secrets and SSO design and a documented cutover. Teams that lack recent Airflow 3 operating experience often use a partner for the design and first DAG conversions, then take over operations. Weigh your own capacity against the parallel run and on-call load, not only the build effort.

Does self-hosting Airflow cost less than MWAA?

It depends on your scale and on what you already run. Apache Airflow is licensed under Apache 2.0 with no paid edition from the project, so the software itself carries no license fee. The comparison is therefore between MWAA's managed service charges and your own infrastructure, database, storage, staff time for patching and upgrades, and on-call coverage. MWAA also limits you to 2 to 5 schedulers and 1 to 25 workers per environment, so a self-hosted setup can scale differently, but it also needs more capacity planning from your team. Build a total-cost model over a fixed period using your actual MWAA invoices, your staffing estimates and a quote for the move, and treat any single figure as unverified until you have checked it.

What happens to our MWAA history and logs?

MWAA run history, task logs and metadata do not transfer automatically to a self-hosted Airflow. Before cutover, decide which history you need for audit, troubleshooting or reporting, then export or archive it from the MWAA environment, typically task logs in its CloudWatch log groups and any DAG run records you must retain. Your new Airflow starts with its own metadata database, so plan the retention and backup policy for it from day one. Keep the MWAA environment, its logs and its S3 bucket read-only for an agreed retention period after cutover, and confirm that retention window with your compliance owner. The written scope should name each history set, where it will be stored and who can access it.

What does the quote need to cover?

A quote should follow from a written scope that lists the Airflow version and target executor, the number of DAGs and connections to convert, the AWS services each DAG uses, the secrets and SSO design, the multi-team model, the parallel-run window, the cutover plan and the history you must keep. Ask for the deliverables and acceptance criteria for each phase, the assumptions about your environment, and what is excluded, such as ongoing support or upgrades. Most engagements are fixed-scope, so a change to the assumptions should be recorded as a scope change with its effect on cost and timing.

What usually goes wrong when DAGs leave MWAA

Most failed moves from MWAA are not caused by DAG logic. They come from the parallel run, the environment around the scheduler, and the ownership gap that appears once AWS stops operating the platform. Each card below names one risk and the step that prevents it.

Duplicate runs during cutover

If MWAA and the self-hosted Airflow both keep DAGs unpaused during the parallel run, the same job can write the same data twice or send duplicate notifications. Decide which environment owns each DAG and record the switch date per DAG. Pause the MWAA copy before unpausing the self-hosted one, and check that DAGs with side effects are excluded from the parallel run or pointed at a sandbox target.

Missing connections and secrets

MWAA stores connections and variables in its own configuration, and many teams never inventory them. Self-hosted Airflow will fail on the first task that reads a missing connection ID, often at 2 a.m. Export a list of connection IDs and variable names from MWAA, map each one to the secrets backend you will use, and run a test task in the new environment that resolves every AWS connection before cutover.

Unpinned dependencies that resolve differently

A requirements file without a constraints reference can install different provider or library versions on the new workers than MWAA resolved. MWAA has required a --constraint statement since Airflow 2.7.2, so a DAG that worked there may not have been pinned in your repository. Pin every install to the constraints file for your exact Airflow version and Python version, and rebuild the image rather than installing at container start.

SQLite or undersized metadata databases

A local test setup or a small database instance can work for a few DAGs and then fail under the scheduler load of a production schedule. The production guide says SQLite is for testing only and can lose data, and the prerequisites exclude MariaDB as a metadata database. Choose PostgreSQL or MySQL from the supported version list, size it for your DAG count and run history, and confirm backups and a restore test before the first production run.

Airflow 3 breakage from direct database access

Airflow 3 task code can no longer use Airflow database sessions or models directly and must use the REST API or Task SDK. DAGs that query the metadata database from a task, and SubDAGs, will fail after an MWAA-era port. Run the Ruff AIR301 and AIR302 rules across the repository, replace database calls with Task SDK or API calls, and move SubDAGs to TaskGroups before the first parallel run.

Unowned upgrades after handover

On MWAA, AWS controls the Airflow patch cadence and only offers in-place minor upgrades. Once you run Airflow yourself, the security fixes and end-of-life dates become your team's responsibility. Airflow 2 reached end of life on April 22, 2026, and limited-maintenance releases receive only security and critical fixes. Assign a named owner for version tracking, schedule a quarterly review of the support table, and agree on the upgrade window before cutover.

How DeployOpen scopes and delivers the MWAA move

DeployOpen plans and implements the self-hosted Airflow in your own infrastructure, whether that is your AWS accounts, a data center or a restricted network. The work starts from a written fixed scope that names the deliverables, the DAGs and connections in scope, the environments involved and the timeline, all agreed before any build begins. For a move off MWAA, the scope usually has to settle questions the managed service answered for you: which Airflow version and constraints file the DAGs will target, how the metadata database and schedulers will be sized, where secrets will live, and what the parallel run and cutover criteria are. Most engagements are fixed-scope, so changes to those decisions go through a written change to the scope rather than an open-ended hourly arrangement.

Because the scope is written, the quote follows from it. DeployOpen does not publish a price list, so the reader should ask for a quote that itemizes the components: discovery of the existing MWAA environments and DAG inventory, platform build, DAG porting and testing, the parallel run, cutover support and handover documentation. Ask which parts are one-time and which recur, and what each assumes about your team's availability for reviews and access approvals. Ask also which items depend on your own decisions, such as identity provider choice or whether multi-team isolation is required, since those change the effort more than most technical details.

Once the migration is complete, the platform does not stop needing attention. Patch releases, minor-version moves and the Airflow 2 to 3 path all recur, and DeployOpen offers retainers for ongoing upgrades, support and continuous testing. Ask whether a retainer covers Python and provider package updates, backup verification and on-call response, and agree in writing how an upgrade is tested against your production DAGs before it is applied.

Moving from Amazon MWAA to self-hosted Apache Airflow