Why an Apache Airflow 3.0 to 3.1 upgrade needs its own plan
If you run Apache Airflow 3.0 in your own infrastructure, the question is when to move to 3.1, and the answer depends on what 3.1.0 changed for your deployment. Released on September 25, 2025, 3.1.0 removed Python 3.9 support, reduced the default number of API server workers to 1, moved logging to structlog everywhere, and decoupled the Task SDK so it can be upgraded on its own. Each of these changes can break a running environment even when your DAG code is unchanged, so the move needs an inventory of your Python interpreters, your API server settings, your log parsing and your Task SDK installs before anyone edits a deployment file.
Plan the move as a step within the 3.1 line. The current patch release is 3.3.2, and newer features and fixes are there, but a jump from 3.0 straight to 3.3.2 combines this upgrade with the 3.2 and 3.3 changes, which makes failures harder to attribute. Confirm the exact 3.1.x patch you will install against the release notes, and decide separately, with its own test cycle, when to advance beyond 3.1.
Airflow is released under the Apache License 2.0, so there is no vendor license to buy and no upgrade fee from the project. The cost is your team's engineering time for testing, configuration changes, rollback preparation and operations after cutover, and that work is the same whether a partner or your staff carries it. Deadline Alerts and human-in-the-loop tasks are new in 3.1, but they are optional. Adopt them only when a pipeline needs them, and keep them out of the upgrade scope so that a stable base move is not blocked by new feature work.
Which 3.1 changes affect your deployment
This table lists each Airflow 3.1.0 change, its likely effect on a self-hosted deployment, and the check your team should run before and after the move. Read it top to bottom against your current environment, Python versions and DAG code. The main takeaway is that the removals and default changes can fail a deployment outright, while the new features only matter once you choose to use them.
| Change | Impact | Action needed |
|---|---|---|
| Python 3.9 removed, Python 3.13 added | Airflow 3.1.0 no longer runs on Python 3.9. Images and virtual environments on 3.9 will not install the new release. | Confirm the interpreter version on schedulers, workers, API server and DAG processor. Rebuild images and rerun the constraint install on 3.10 or later. |
| Default API server workers reduced to 1 | A single worker can slow the UI and REST API under concurrent use if you relied on the old default. | Load test the API server with your real user and client count. Set the worker count explicitly in configuration and record the value. |
| Logging moved to structlog everywhere | Log line format changes, which can break log parsers, alert rules and custom logging handlers. | Inventory log consumers and custom logging config. Run a staging deployment and confirm that log shipping and searches still match. |
| Task SDK decoupled and upgradable independently | Core and Task SDK versions can now differ, so mismatched installs on workers may behave unexpectedly. | Pin both packages in your constraints file. Confirm worker images install the SDK version you tested with core. |
| Human-in-the-loop tasks | New task type that pauses a run for a person to decide. It affects runbooks and on-call expectations once used. | Decide whether any DAG will pause for approval. If so, define who can respond and test the resume path in staging. |
| Deadline Alerts | New alerting on task or DAG deadlines. Existing alerting does not change unless you add these alerts. | Identify the pipelines with delivery deadlines. Agree alert recipients and test that notifications reach the on-call channel. |
| React plugin system (AIP-68) | Plugins built on the older UI approach need review before use. The new plugin model is an addition, not a drop-in replacement for FAB-era plugins. | Inventory custom UI plugins. Check each one against the release notes and test it on the 3.1 UI before production. |
| Synchronous DAG execution for inference | A new way to run DAGs synchronously for inference workloads. It matters only for teams running model inference through Airflow. | Confirm whether any team needs this mode. If so, check resource limits and timeouts in a staging run. |
| ALL_DONE_MIN_ONE_SUCCESS trigger rule | A new trigger rule that runs a task once all upstream tasks finish and at least one succeeded. Existing DAGs keep their rules. | Review joins or cleanup tasks where the rule could fit. Test the rule with failure scenarios before adopting it. |
Choosing the upgrade path from 3.0 or from Airflow 2
If you already run Airflow 3.0, the core move to Airflow 3 is done and this upgrade is a minor version step within the 3.x line. Your DAG code has already been written or ported for the Airflow 3 model, so the main changes you face are in the 3.1 release itself: Python 3.9 is no longer supported, the API server defaults to one worker, logging moves to structlog, and the Task SDK can be upgraded on its own schedule.
- Python runtime: confirm that schedulers, workers and images run Python 3.10 or later, since 3.9 support was removed in 3.1.
- API server workers: check whether your deployment relied on the previous default of more workers and set the count explicitly if you need it.
- Logging pipeline: confirm that log parsers and forwarders expect structlog output.
- Task SDK version: decide whether to move the Task SDK with core or on a separate cadence, and pin it in your constraint files.
- Feature adoption: Deadline Alerts and human-in-the-loop tasks are opt-in, so plan their rollout separately from the upgrade.
Before starting, confirm that your installed providers and images are compatible with 3.1 and that you have a tested backup of the metadata database.
If you still run Airflow 2.x, you cannot move straight to 3.1. The Airflow 3 upgrade guide requires Airflow 2.7 or later before upgrading, and recommends moving to the latest 2.x release first. Airflow 2 reached end of life on April 22, 2026, so it receives no fixes at all, which makes a planned path off 2.x more urgent than it was when limited maintenance began.
- Minimum version: get to at least 2.7, and ideally the latest 2.x patch, before any Airflow 3 step.
- Ruff checks: run the AIR301 and AIR302 rules against your DAG repository to find code that will break in Airflow 3.
- Removed features: review SubDAGs, removed context keys and direct metadata database access in tasks.
- Deployment changes: plan for the webserver to be replaced by the API server and the DAG processor to run as a separate component.
- Authentication: Airflow 3 uses the Simple auth manager by default, so decide how you will handle the FAB provider and your identity setup.
Confirm your current version and the path your team can take in the upgrade guide before you schedule anything.
If your Airflow runs on a managed service, the provider controls part of the upgrade timing, the images and the supported version range. You will not run the database migration or the Helm chart yourself, so the work shifts toward confirming what the provider offers and whether it matches the version you want. Ask the provider which Airflow versions it offers and when it plans to offer 3.1, because your options depend on their release schedule rather than yours.
- Available versions: confirm which Airflow versions the service supports and whether 3.1 is among them.
- Upgrade mechanism: find out whether upgrades are in-place, side-by-side or require migrating to a new environment.
- Custom images and plugins: check whether your plugins, providers and Python dependencies are supported on the provider's images.
- Data and logs: confirm where metadata, logs and backups are stored and who can restore them.
- Exit path: if you plan to self-host later, understand how DAGs, connections and variables would be exported.
Before committing, ask the provider for its upgrade documentation and its support window for each Airflow line.
What to prepare before touching 3.0 in production
Most upgrade problems are decided before the first package changes. The six items below cover the data, runtime, code and people you need to protect or confirm, and each one feeds the staging run that proves the upgrade works.
Metadata database backup
Take a consistent backup of the metadata database before any migration step, and confirm you can restore it into a scratch instance. Airflow 3.1 runs against PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or Innovation releases, and SQLite is not suitable for production because it can lose data. Check that MariaDB is not your current backend, since it is not supported as a metadata database. Plan disk and memory for the database host with the recommended 4 GB of RAM as a floor, not a target.
Python and image inventory
List every Python interpreter and container image that runs Airflow components, including schedulers, workers, DAG processors and the API server. Airflow 3.1.0 removed Python 3.9 support and added Python 3.13, so any environment still on 3.9 must move before or with the upgrade. Record the base image tags and confirm each one matches the Python version your constraint file targets.
Custom plugins and operators inventory
Identify every custom operator, hook, sensor and plugin, and note which ones import Airflow internals or call the metadata database directly. Airflow 3 requires task code to use the REST API or Task SDK instead of database sessions, so these components are the most likely to fail in staging. Assign an owner to each item and record the test that proves it still works after the upgrade.
Staging environment that mirrors production
Build a staging stack with the same executor, database engine, DAG delivery method and secrets backend as production, then restore a recent copy of the metadata database into it. Run representative DAGs through a full cycle, including retries, sensors and any scheduled runs that fall during the test window. A staging stack that differs in executor or database engine will not show the failures you need to see.
Task SDK and provider version pins
Airflow 3.1 decouples the Task SDK, so it can be upgraded on its own cadence, but it still has to match the core version your workers run. Pin the Task SDK, core and provider package versions in a constraint file, and record which provider releases your DAGs depend on. Confirm the minimum Airflow version each provider requires before you change any pin in production.
Named rollback owner
Assign one person with authority to stop the rollout and restore the database backup, and write down the trigger conditions, such as failed migrations or scheduler heartbeats that stop. Decide in advance who approves a rollback and how long the team will wait before choosing it. Without a named owner, teams often hesitate while a partially migrated environment keeps running.
The 3.0 to 3.1 upgrade sequence with rollback points
This sequence moves a self-hosted Airflow 3.0 deployment to 3.1 in an order that keeps a restore point before every database change and after every redeployed component. Each step ends with a check you can confirm before moving on.
Freeze DAG and configuration changes
Pause merges to the DAG repository, deployment manifests and Airflow configuration for the upgrade window, and record the currently running Airflow 3.0 version, provider versions and Helm chart values. Name one owner who approves any exception. The step is done when the frozen commit or tag, the image digests in use and the active configuration are written down and can be restored from version control.
Back up the metadata database
Take a consistent backup of the metadata database while schedulers and workers are quiet, using your database team's standard tooling. Because the Airflow production guide warns that SQLite is for testing only, confirm you are on PostgreSQL or MySQL before relying on this backup. This is the first rollback point: before any schema change, you should be able to restore the database to its pre-upgrade state and confirm the restore in a scratch environment.
Rebuild images on a supported Python version
Build new Airflow images that use a Python version still supported by 3.1, since Python 3.9 was removed in that release. Pin the Airflow constraint file that matches 3.1 so dependencies resolve consistently. Confirm the build passes import checks for every DAG dependency. The output is a tagged image set that is stored in your registry and not yet deployed.
Update the Task SDK and providers
Upgrade the Task SDK and the provider packages your DAGs use, following the constraint file for 3.1. Because the Task SDK can now be upgraded independently of core, confirm which versions your workers need and record them. The step is done when a test environment installs the new packages cleanly and a sample of operators from each provider imports without errors.
Run the migration in staging
Restore the backup into a staging database, then run the database migration against it and time the run in your own environment. Reserialize a representative set of DAGs and run them end to end. This is the rollback point for staging: if the migration or DAG runs fail, you discard the staging copy and fix the build before touching production. Do not start production until staging completes without schema or DAG errors.
Set API server workers and logging configuration
Airflow 3.1 reduced the default number of API server workers to one, and it moved logging to structlog everywhere. Review your configuration to decide whether one worker is enough for your traffic or whether you need to set the worker count explicitly, and update any custom log handlers, parsers or log-shipping rules that assumed the older format. Confirm the settings in the staging deployment and check that logs from the scheduler, workers and API server reach your log store in the expected structure.
Redeploy components and verify
Apply the production migration after a fresh backup, then redeploy in this order: scheduler, DAG processor, API server and workers. Before each component rolls out, confirm the previous one is healthy, and keep the prior image tag so you can roll back that single component. After the last component is up, verify that scheduled runs start, the API responds, a sample DAG completes and logs appear. The sequence is complete when those checks pass and the restore point is retired by your standard retention policy.
What changes for the teams who run Airflow day to day
Airflow 3.1 moves logging to structlog everywhere, so the log lines your team parses, ships to a log platform, or alerts on will change shape. Plain-text patterns that matched Airflow 2 or early 3.0 output may stop matching, and the fields you filter on may be named or nested differently. Treat this as a log contract change: before the upgrade, list every parser, saved search, dashboard and alert rule that reads Airflow output, then run a staging instance on 3.1 and confirm each one still fires on known test events. Ask your observability owner which of those rules depend on exact message text. The release notes for 3.1 are the place to confirm the exact output changes for your components, so check them against your own log samples rather than assuming.
The API server now starts with one worker by default, down from the higher count many teams had been running. Under light load this is rarely visible, but the UI, the REST API used by CI jobs and integrations, and the task execution traffic all pass through the same server processes. If you have set a worker count explicitly, the default change does not affect you, so check the api_server worker setting in your configuration. If you rely on the default, measure request latency and error rates on the API server during peak scheduling windows in staging, and decide whether to raise the worker count or scale out replicas before go-live. Confirm the exact configuration key and its behavior in the 3.1 release notes and your deployment method.
The Task SDK can now be upgraded independently of the core scheduler and API server, which lets two groups move on different schedules. DAG authors can adopt a newer SDK to pick up task-side changes without forcing a platform upgrade, and platform teams can hold the core at a tested version while they validate the SDK. The practical risk is version drift: confirm which SDK versions your worker images and DAG repositories pin, and make sure the combination you run is one the release notes describe as supported. Set a rule in your change process that records the core and SDK versions together, so a failed task can be traced to the right pair.
What makes a 3.0 to 3.1 upgrade take longer
This table lists the factors that most often stretch a move from Airflow 3.0 to 3.1, along with why each one matters and what to check before planning the work. Read each row against your own estimate, since a single high-risk driver such as Python-version-dependent dependencies or custom plugins can change the whole plan.
| Driver | Why it matters | What to check |
|---|---|---|
| Custom plugins, operators and hooks | Airflow 3.1 removes Python 3.9 and moves logging to structlog, so custom code and third-party packages that rely on older behavior need retesting. | Count custom plugins, operators and hooks, and confirm each one imports and runs on the target Python version. |
| Python-version-dependent dependencies | Dropping Python 3.9 and adding 3.13 means some pinned libraries may lack wheels or behave differently on the new interpreter. | Compare your dependency lock or constraint files against the target Python version and list packages without confirmed support. |
| Logging integrations that parse plain-text output | Logging moved to structlog everywhere, so log shippers, alert rules or scripts that parse plain-text lines may stop matching. | Inventory log parsers, dashboards and alerts, and test them against structured output in staging. |
| API server load profile | The default number of API server workers dropped to one, which can slow UI and REST API traffic if the old setting was relied on. | Review your current worker settings and measure UI and API response times under realistic usage in staging. |
| Database size and history retention | A large metadata database makes migration and backups slower and raises the risk of a long downtime window. | Check table sizes, retention settings and when the last clean-up ran, and confirm a tested backup exists before migrating. |
| Number of environments and teams | Each environment (development, staging, production) and each team's DAG repository must be upgraded and validated separately. | List every environment and owner, and agree on the promotion order and sign-off steps before the first change. |
| Executor and deployment model | Kubernetes with the official Helm chart, or virtual machines with separate components, each need their own rollout, restart and rollback steps. | Confirm how the API server, scheduler, DAG processor and workers are deployed, and which of them run from the same image or package version. |
Questions teams ask before upgrading Airflow 3.0 to 3.1
What does the Airflow 3.0 to 3.1 upgrade cost, and who pays for it?
Airflow itself is free under the Apache License 2.0, so the cost is the labor and infrastructure needed to make the move. Your internal team pays in engineer time for inventory, DAG and code review, staging tests and cutover, and your cloud or data center pays for the extra staging environment and any database backup storage. If you use a partner, the cost is the fixed-scope fee agreed in writing before work starts, which DeployOpen quotes from that scope. One-time costs include the assessment, fixes and the first cutover. Recurring costs appear only if you add a retainer for patching and support. Ask any quote to separate these items and to state which environments and DAG counts it assumes.
How long does the upgrade take, and what drives the schedule?
The schedule depends mostly on how many DAGs, custom operators and plugins you run, how many environments must move in sequence, and how much of your code touches changed areas such as logging, Python versions and the Task SDK. The supplied release information does not give a duration, so treat any figure you are quoted as a claim to verify against your own inventory. Other schedule drivers include the size of your metadata database, the window you can accept for the migration, and whether you need a parallel staging release first. Ask for a schedule that shows the inventory, staging, cutover and rollback checks as separate milestones.
What does the upgrade require from our internal team?
You need people who know your DAGs, your deployment method and your identity and secrets setup. In practice this means a platform owner who controls the environments, DAG authors who can explain intent behind each pipeline, and someone who can approve changes and schedule the downtime window. Security and data owners should review any changes to logging, access and retained data. Your team should confirm the Python version on workers and schedulers, since 3.1.0 removed Python 3.9 support, and should check which custom code relies on logging formats or the Task SDK. Confirm these responsibilities in writing before the work starts so that nothing is assumed.
Can our team complete the 3.0 to 3.1 upgrade without outside help?
Yes, many teams can, provided they have engineers who understand Airflow internals, a staging environment that mirrors production and the time to test every DAG. The main risks are missed changes in custom plugins, logging dependencies and Python version pins, and rollback mistakes after the database migration. A partner is most useful when your team lacks capacity, when the DAG estate is large, or when you need an independent test and cutover plan with evidence for change management. Decide based on your own skills and calendar, and confirm the scope of any outside help in a written, fixed-scope agreement.
Can we skip Airflow 3.1 and move straight to 3.3.2?
Skipping 3.1 is possible in principle, but it changes the work rather than removing it. Airflow 3.3.2 is the current patch release listed in the supported versions table, and your move would include the changes from 3.1 as well as those from later lines, such as the state store and metrics changes in 3.2 and 3.3. Teams often choose a two-step path so each step can be tested and rolled back on its own. Compare the two options by counting the changes in each step, the number of DAGs affected and the test cycles you can afford. Confirm the full list of changes between your current version and the target in the release notes before you choose.
How does rollback work after the database migration?
Rollback depends on the backup you take before running the migration, because a migrated metadata database may not be readable by the older release. Before the migration, take a consistent backup of the metadata database and keep the previous images and configuration so you can redeploy them. Confirm that your database is PostgreSQL or MySQL, since the production guide warns that SQLite can lead to data loss and should not be used in production. Restoring the backup is the usual route back, so the rollback plan should state how much run history would be lost and how the scheduler is paused during restore. Test this restore in staging before the production window.
How DeployOpen scopes and delivers the 3.0 to 3.1 upgrade
DeployOpen plans, builds, tests and documents Airflow 3.0 to 3.1 upgrades inside infrastructure you control, starting from a written scope. The work covers your Python 3.9 exposure, logging changes, Task SDK decisions and the Airflow 3.1 features you choose to adopt.
How we scope
- Inventory your Airflow 3.0 deployment, including Python versions, executors, metadata database and DAG repositories.
- Identify any workers or images still on Python 3.9, which Airflow 3.1 removes, and agree how they move to a supported version.
- Agree in writing which 3.1 features are in scope, such as Deadline Alerts or human-in-the-loop tasks, and which are deferred.
How we build
- Rehearse the upgrade in a staging environment that mirrors production, including the API server worker setting and structlog output.
- Check logging consumers and parsers for structlog-formatted output and update them before the production cutover.
- Test DAG runs, Task SDK versions and provider packages against 3.1, recording each result as evidence for your change record.
How we hand over
- Deliver a runbook covering the upgrade sequence, configuration changes, rollback steps and post-upgrade checks.
- Document the Task SDK version policy so your team knows how to upgrade it independently of the Airflow core.
- Walk your operators through the staging and production results and the open items agreed in the written scope.
How we support
- Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after the upgrade is complete.
- Ask for a written quote based on your scope; DeployOpen does not publish a price list, so cost follows from the agreed work.
