Why upgrading Airflow provider packages safely starts with a pinned, tested baseline
Keep provider packages current by building a pinned, tested baseline first, upgrade providers on their own cadence, and test operator changes before any rollout. Provider packages such as Amazon, Slack, Snowflake and Databricks are separate PyPI packages, so they can drift away from the Airflow core version without anyone noticing until a DAG fails. Most of the providers documented by the Airflow project list a minimum Airflow version of 2.11.0, which matters if you are still running Airflow 2, because Airflow 2 reached end of life on April 22, 2026 and no longer receives fixes.
The baseline is a constraints file matched to your Airflow version and Python version. Airflow publishes one per release, and a plain pip install apache-airflow can produce an installation that does not work. Install providers in a separate step pinned to the Airflow version, and freeze and host your own constraints file on an internal server so every build resolves the same versions. Confirm with your team which Python version your workers and images use, since the constraints file is specific to it, and record which provider versions each environment runs.
Once the baseline is frozen, upgrade providers in a staging environment that mirrors production, and run the DAGs that use the changed operators, such as durable BigQuery jobs, against real connections before rollout. Read the release notes for each provider package, because operator arguments and default behavior can change between versions. DeployOpen plans and carries out this work inside your own cloud accounts, data centers or air-gapped networks. It does not issue audit reports, attestations or certifications, so any evidence you need for an assessment should be confirmed with your independent auditor.
Which provider and core changes break DAGs, and what to do about each
This table pairs each provider or core change that can break DAGs with its impact and the action to take before upgrading. Read it as a checklist: confirm the Airflow version floor and constraints first, then test any operator or hook change against your DAGs before it reaches production.
| Change | Impact on DAGs | Action needed |
|---|---|---|
| Minimum Airflow version per provider | Most listed provider packages, including Amazon, Snowflake, Databricks and Slack, require Airflow 2.11.0 or later, so older 2.x installs cannot take the newer provider. | Check each provider's minimum Airflow version before pinning. Plan the core upgrade first if your Airflow is older than 2.11.0. |
| Separate provider install step with a constraints file | A plain pip install of apache-airflow can produce an unusable installation. Constraints apply only to the pip command they are used with. | Install providers in a separate step pinned to your Airflow version, using the constraints file for your Airflow and Python versions. Freeze and host an internal copy if your network requires it. |
| Operator and hook signature changes in a provider release | A release can change how a task waits, retries or reports status. For example, durable BigQuery jobs change how long-running queries are tracked, which can alter DAG timing and retry behavior. | Read the provider's release notes for each version. Run DAG-level tests against the new provider in staging before promotion. |
| Airflow 2 end of life on April 22, 2026 | Airflow 2 receives no fixes after end of life, so provider fixes on the Airflow 2 line stop being available for that core. | Confirm which provider versions your current Airflow 2 release can still use. Plan the move to Airflow 3 with a dated provider upgrade path. |
| Airflow 3 FAB provider requirement | The FAB auth manager comes from the FAB provider, not core. Without the provider installed, a FAB-based auth setup will not load. | Add the FAB provider explicitly when you choose the FAB auth manager, and confirm the auth_manager setting in staging. |
| Provider changes on a separate cadence from core | Provider releases can change independently of Airflow core, so an unpinned provider can change behavior between deployments. | Pin provider versions in version control, upgrade them on their own schedule, and record each change in your change log. |
What to inventory and back up before bumping any provider
Provider upgrades fail most often because the team cannot say what is installed, what depends on it, or how to return to the previous state. The cards below cover the records, copies and rehearsal environment to prepare before changing a single provider version. Work through them in order so that every later step has a known baseline.
Inventory every installed provider
List each provider package in use, its installed version, and the Airflow version it targets. Pull this from the running environment, not from the requirements file alone, because transitive installs and image builds can differ from what was declared. Record the minimum Airflow version each provider requires, since most documented provider packages, including Amazon, Snowflake, Databricks and Slack, list 2.11.0 as their minimum. Mark any provider that targets a newer Airflow line than the one you run.
Freeze and host the constraints file
Save the constraints file that matches your current Airflow release and Python version, then keep a copy on an internal server or artifact store. Airflow publishes constraints per release and Python version, and a plain pip install of the core package can produce an unusable installation. Constraints apply only to the pip command they are used with, so store the exact file alongside the image build that uses it. Confirm that your build pipeline can reach the internal copy if the public source is unavailable.
Map DAGs and operators to each provider
Build a list of every DAG and operator, hook and sensor that imports from each provider package. Search the DAG repository and any shared libraries for provider imports, then note the owning team and the schedule of each DAG. Operator changes, such as moving a BigQuery job to a durable configuration, affect only the DAGs on this list, so the map tells you which pipelines need a test run before release and which can wait.
Build a staging Airflow that matches production
Stand up a staging Airflow with the same Python version, the same provider pins and a metadata database on the same engine as production. Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, and do not use SQLite, which the production guide says is for testing only. Mismatched engines or Python versions hide errors that appear only in production, so treat any difference as a finding to close before testing provider changes.
Copy connections and variables used by hooks
Export the connections, variables and secrets backend entries that provider hooks read, such as cloud credentials, warehouse accounts and Slack webhooks. Record the connection type and the fields each hook expects, because a provider upgrade can rename parameters or change which fields are required. Store the export in your secrets system with restricted access, and never place plaintext credentials in a ticket or a shared repository.
Keep a rollback image and pin ready
Retain the previous container image and the constraints pin that produced it, and confirm that you can redeploy both within your change window. Rolling back a provider usually means reverting the pin and rebuilding the image, so test that build path before the upgrade. Record which DAGs have already run on the new provider version, because a rollback does not undo writes made by an operator that changed behavior.
How to upgrade one provider at a time with checks and rollback points
This sequence moves one provider package at a time from a known pin to a new version, with a named check at each stage and a previous pin kept ready so any step can be reversed without touching core Airflow.
Confirm the minimum Airflow version
Before changing anything, read the target provider's package page and confirm its minimum supported Airflow version against the Airflow core version that is running now. Most provider pages in the documentation list a minimum of 2.11.0 for the packages reviewed, but each provider sets its own floor, so check the exact version for the provider you plan to move. If the core is older than the minimum, stop here and plan a core upgrade first. The output is a written note recording the provider, the target version, the core version and the result of the comparison.
Update the pin in a staging constraints file
Change the provider's version in a staging constraints file built for the same Airflow release and the same Python version that production uses. Constraints apply only to the pip command they are used with, so the file must be passed explicitly in every install. If your team hosts its own frozen constraints file on an internal server, create a new copy for staging rather than editing the production file in place. The output is a staging constraints file that differs from production only in the provider line being tested, and the previous production file stays untouched as your rollback reference.
Install in a clean environment
Build a fresh virtual environment or container image for staging. Install Airflow core first, pinned to its version and the constraints file, then install the provider as a separate step pinned to the new version with the same constraints file. Keeping the provider install separate avoids pip resolving the provider into an incompatible combination. Record the output of pip freeze from the built environment. The output is a clean environment whose frozen package list matches the staging file, with no unexpected package changes.
Run DAG parsing and import checks
Run a full DAG parse across the repository in the new environment and fail the build on any import error, including imports from the upgraded provider. Run the Ruff checks for Airflow rules where they apply to your code, and confirm that every DAG file loads and that the provider's operators and hooks import under the expected names. Check the Airflow import error list in the staging environment as well. The output is a clean parse report with zero import errors, saved with the build record so a later failure can be compared against it.
Run operator tests against sandbox accounts
Run operator-level tests for the affected provider against sandbox accounts or test projects in the target system, such as a non-production warehouse, bucket or Slack workspace. Focus on the operators your DAGs actually use and on changed parameters or default behavior noted in the provider's release notes. Where a provider change affects how jobs are submitted or polled, confirm that tasks retry and recover correctly after an interrupted run. Your team owns the sandbox credentials and the test data. The output is a test report per operator with a pass or fail result and a list of any behavior changes accepted or blocked.
Promote one provider group to production
Promote the change to production for one provider group at a time, not the whole provider set in a single change. Use the approved production constraints file, add the new pin, and deploy through your normal change process using the same clean-install steps used in staging. Watch the first scheduled runs of the DAGs that use the provider and compare task results with the baseline from before the change. The output is a production deployment record and a short run review confirming that the affected DAGs completed with expected results.
Keep the previous pin as the rollback point
Retain the previous constraints file and the previous provider pin in version control until the new version has run in production through a full cycle of your DAG schedules. If a failure appears, roll back by reinstalling the previous pin with its matching constraints file in a clean environment, then redeploy and rerun the affected DAGs. Roll back one provider at a time so the cause stays isolated. The output is a documented rollback procedure tied to the exact previous pin, and a decision record that closes the change once the new version is accepted.
How to test operator changes such as durable BigQuery jobs before production
A provider release can change how an operator behaves even when your DAG code has not changed at all. The operator's submission and tracking logic lives in the provider package, so a new version may submit work differently, wait on it differently, or record different identifiers and log lines. Durable BigQuery jobs are a good example of this kind of change: an operator that moves from a fire-and-forget submission to a job the operator tracks through its own identifier alters retries, timeouts, cost attribution and what appears in task logs. Illustrative only: a task that once re-ran a query after a worker restart might now reattach to the existing job instead. Confirm the exact behavior change for each provider version in that provider's release notes before you assume the old and new behavior differ, or that they do not.
Test in layers, from cheap checks to real runs. Start with parsing tests that import every DAG against the pinned provider set, so that renamed parameters, moved imports and removed arguments fail in CI rather than at the scheduler. Next, write operator unit tests that mock the BigQuery hook and assert the calls the operator makes, such as the job configuration, the job identifier, and whether it polls or returns early. These tests catch changed call patterns quickly, but they cannot prove that Google's service behaves as expected. For that, run integration tests against a non-production project with a dedicated dataset, service account and small sample tables, and confirm that job names, labels, retries and cleanup behave as your pipelines require. Keep the credentials and project scoped to the test environment so that a misconfigured run cannot touch production data.
How the provider upgrade differs for Airflow 2.x and Airflow 3 teams
Teams still on Airflow 2.x have the most room to choose, but the room is shrinking. Most provider packages in the documented set list a minimum Airflow version of 2.11.0, so a provider upgrade on a 2.x line below that version is usually blocked by the package metadata before any code runs. Airflow 2 reached end of life on April 22, 2026, and after that date it gets no fixes, so provider upgrades on that line are a bridge to the Airflow 3 move rather than a long-term setup.
- Minimum core version: Check each provider's minimum Airflow version before you pin it. Most documented providers, including Amazon, Snowflake, Databricks and Slack, require 2.11.0.
- Constraints per release: Install providers with the constraints file for your exact Airflow version and Python version, so pip resolves a set that works together.
- Early move to the standard provider: Confirm that the standard provider is installed and pinned, since Airflow 3 expects it and it is easier to validate on 2.x.
- Operator behavior changes: Test operators whose arguments or defaults changed between provider releases, such as BigQuery job handling, against real staging data.
- Schedule for the move: Treat any provider upgrade as a step toward Airflow 3 and plan the core move, not just the provider move.
Before upgrading, confirm the core version, the minimum version of each provider you depend on, and the date your organization plans to leave Airflow 2.
Teams on Airflow 3 pin providers against the core line they run, and the provider version must match what that core release supports. Upgrading providers is therefore part of every core patch or minor upgrade rather than a separate cycle, although providers can still move on their own schedule once the core is stable. Airflow 3 changes the auth picture as well: the Simple auth manager is the default, and the FAB auth manager comes from the FAB provider, so that provider is only needed if you use that auth manager.
- Match the core line: Pin providers to versions that declare support for your Airflow 3 release, and re-check them after each core patch.
- Install FAB only when used: Add the FAB provider only if the
auth_managersetting points to the FAB auth manager, and remove it otherwise to reduce the pinned surface. - Check DAG code with Ruff: Run the Ruff AIR301 and AIR302 rules across the DAG repository, so operator imports and removed features are caught before a provider change reaches production.
- Test against the API server: Verify operators in a staging environment that runs the API server and DAG processor as separate components, since task behavior depends on the Task SDK path.
- Pin in one place: Keep the constraints file and provider pins in version control so a provider upgrade is a reviewed change with a rollback path.
Before upgrading a provider on Airflow 3, confirm the core version, the auth manager in use and the Ruff results for the affected DAGs.
Security and audit teams care less about which provider version is current and more about whether each change is traceable, reviewed and reversible. A provider package changes the code that runs with your task credentials, so each upgrade needs a recorded scope, a test result and an owner. The approach differs by starting point: on Airflow 2.x the review centers on minimum core versions and the end of life date, while on Airflow 3 it centers on the match between provider and core, and on which auth manager the package brings in.
- Record the pinned set: Keep the constraints file, the provider list and the core version together in a change record for each upgrade.
- Review provider changelogs: Read the release notes for providers that touch credentials, connections or data movement before approving a pin.
- Confirm the auth dependency: Verify that the FAB provider is present only where the FAB auth manager is configured, so the access model stays as reviewed.
- Keep test evidence: Store the results of operator tests and DAG checks alongside the change record, so auditors can see what was verified.
- Plan rollback: Retain the previous constraints file so a provider change can be reverted without rebuilding the environment from memory.
Before approving a provider upgrade, confirm the change record, the reviewed changelog and the rollback file exist.
Platform teams usually build the images or virtual environments that run Airflow, so they see provider upgrades as a build and release problem. The main difference between the two starting points is how the image is composed: on Airflow 2.x the build must resolve against a constraints file for an older core, while on Airflow 3 the build must match the Task SDK and API server components that now run alongside the scheduler. Either way, a plain install of the core package without constraints can produce an installation that does not work, so the build should always reference a constraints file.
- Build from constraints: Pass the constraints file for your core and Python version in every build, including local development images.
- Separate the provider step: Install providers in a distinct step pinned to the core version, which makes failures easier to isolate.
- Host an internal constraints copy: Freeze and host a constraints file on your own server, so builds do not change when upstream files are updated.
- Rebuild and scan images: Rebuild all images, including workers and the API server, when a provider changes, and rescan them.
- Test in staging first: Deploy the new image to a staging release that mirrors production components before promoting it.
Before promoting an image, confirm the constraints file, the provider step and the staging test results match the core version in use.
What makes provider upgrades take longer
This table compares the factors that expand or shrink a provider package upgrade and what each one changes in effort. Use it to size the work before committing to a date, and to find which checks your environment needs before the upgrade starts. The main takeaway is that the number of DAGs and custom subclasses matters less than how many environments, credentials and owners must agree before a change ships.
| Driver | Effect on effort | What to check |
|---|---|---|
| Number of provider packages installed | Each provider adds its own release notes, minimum Airflow version and possible operator changes to review | List every installed provider and its pinned version from your constraints file |
| DAGs that import provider classes | More imports mean more code paths to test after each provider release | Search DAG repositories for provider imports and count affected DAGs |
| Custom operators or hooks that subclass provider classes | Subclasses break when a parent class changes signature or moves, and they are often untested | Identify every internal class that extends a provider class and confirm who maintains it |
| Distinct Python and Airflow versions across environments | Each combination needs its own constraints file and test run, so effort multiplies | Inventory dev, staging and production versions and the constraints file used for each |
| Credentials and sandbox accounts for integration tests | Without test accounts, operator changes can only be checked by reading code | Confirm which cloud, warehouse and messaging accounts exist for non-production testing |
| Teams that own affected DAGs | Each owning team needs to review behavior changes and approve the rollout | Map DAG owners and agree on a sign-off path before the upgrade window |
| Internally hosted constraints files | A hosted file lets you freeze versions and test them, while public-only use leaves resolution to upstream changes | Confirm whether a frozen constraints file is hosted on a server you control and who updates it |
What buyers ask before paying for provider upgrade help
How much does a provider package upgrade cost?
The cost depends on how many provider packages you run, how many DAGs import them, and how many operators in those DAGs change behavior between versions. You pay for inventory, compatibility testing, code changes, test runs in a sandbox and the written handover. Costs recur only if you keep a retainer or repeat the upgrade on a new cadence. Ask any partner for a quote that lists each of these as a separate line, with the provider packages in scope named. No public price list exists, so the written scope is the basis for a figure.
How long does a provider upgrade take?
Duration is set by the number of providers, the number of DAGs and operators that touch changed APIs, and how quickly test accounts and owners respond. Teams with a current constraints file and good test coverage move faster than teams that have never pinned their providers. Ask for a timeline tied to named milestones in the written scope rather than a single date, and confirm which steps wait on your access approvals. No general duration figure is given here because it depends on your environment.
What does our team need to provide for a provider upgrade?
Plan to supply four things before work starts: an inventory of installed provider packages and the DAGs that import them, sandbox cloud or SaaS accounts that mirror production, a named owner for each DAG or group of DAGs, and a window for reviewing changes. Without an owner, behavior changes in an operator can go unreviewed. Without sandbox accounts, tests cannot exercise real connections. Confirm these responsibilities in the scope document so delays are visible early.
Can our internal team upgrade providers alone?
Yes, many teams can. Internal teams usually handle the pinning, the constraints file and simple import changes well. Outside help tends to pay off when a provider release changes an operator in a way that affects production data, such as a move to durable BigQuery jobs, or when the team has no time to build test coverage around the change. Ask a partner to show how they would separate the parts they handle from the parts your team keeps.
Is a fixed-scope provider upgrade engagement available?
Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. A written scope for a provider upgrade should name the providers and Airflow versions covered, the DAGs and operators in scope, the test environments used, the changes DeployOpen will make, the handover documentation, and what is excluded. Anything not listed should be treated as a change request with its own quote.
Does DeployOpen issue audit reports for a provider upgrade?
No. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies do that work. If your upgrade falls under a compliance program, ask for change records, test results and the configuration history that your auditor will request, and confirm with your auditor which evidence they need.
What should we ask a partner in a provider upgrade quote?
Ask which providers and Airflow versions the quote covers, and whether it includes the constraints file, the pinning policy and the minimum Airflow version check for each provider. Ask how they test operator changes, which environments they use, and who approves changes to production DAGs. Ask what is one-time and what recurs, what is excluded, and how new findings are priced. A clear answer to each point is a better signal than the total.
How are provider upgrades supported after the work ends?
Provider packages keep releasing, so an upgrade is not a one-time event. A retainer can cover ongoing support, scheduled upgrades on a separate cadence from Airflow core, and testing of new provider releases before they reach production. Ask what the retainer includes, how release notes are reviewed, how urgent fixes are handled, and what documentation is updated after each change. Confirm the response expectations in writing before you sign.
How DeployOpen works on this
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, and provider upgrades follow the same path. We scope the provider inventory and pinning approach first, build and test the upgrade in a staging environment you control, and hand over the constraints, runbooks and test evidence your team needs to repeat it.
How we scope
- We inventory installed provider packages, their pinned versions and the operators your DAGs call today.
- We confirm your target Airflow version and the minimum Airflow version each provider package requires.
- The written scope names the providers, environments, constraint file location and test coverage before work starts.
How we build
- We build a constraints file for your Airflow and Python versions and host a frozen copy on your internal server.
- We upgrade providers on a separate step in staging, then run DAG tests against changed operators such as BigQuery jobs.
- We record each provider version change in a reviewed pull request so that rollback to the previous pins stays simple.
How we hand over
- You receive the pinned provider list, constraint file, and the upgrade runbook covering install order and rollback steps.
- You receive test results and a list of operator behavior changes that your DAG authors need to review.
- We walk your platform and data engineering teams through the cadence for future provider upgrades, separate from core Airflow.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing.
- A retainer can cover regular provider reviews against each Airflow release and its supported provider versions.
- Before any retainer starts, confirm response expectations, change windows and which environments the support covers.
