Why tasks that query the Airflow metadata database break in Airflow 3
Airflow 3 removes direct metadata database access from task code, so any DAG task that opens a database session or uses an Airflow model to read or write the metadata database must be moved to the REST API or the Task SDK before the upgrade. Teams leaving Airflow 2 often find this in tasks that check run state, reset task instances, or record custom status in the tables behind the UI. These tasks do not fail gracefully at import time. They fail when they run, so a clean DAG parse does not prove the code is safe. The supported path starts from Airflow 2.7 or later, and the upgrade guide recommends moving to the latest 2.x release first. The current 2.x patch is 2.11.2, and it is the last Airflow 2 line you should plan around. Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. Confirm your current version and the date your team plans to leave 2.x before you scope the rewrite, because a rewrite that finishes after end of life leaves you running unpatched scheduling in the meantime.
Before you estimate the work, run a search across your DAG repository and plugins for session and model imports from the Airflow package, direct calls to hooks that point at the metadata database, and custom operators that wrap them. Sort each finding into one of three groups: state reads that the REST API can serve, writes that belong in the Task SDK or an external system you own, and logic that should move out of the DAG entirely. Ruff AIR rules can help flag some patterns, but they will not find every custom query, so the inventory needs a code review. Ask any implementation partner to show how they will produce and sign off this inventory before they quote a fixed scope.
What changes for database-touching tasks in Airflow 3
This table compares each Airflow 3 change that affects tasks reaching into the metadata database, with its impact on existing DAGs and the action to take. Read it as a checklist: find the patterns in your code, then fix them before the upgrade rather than after the first failed run.
| Change | Impact | Action needed |
|---|---|---|
| Operators lose metadata database sessions | Operator code that opens a database session or runs queries against Airflow tables stops working in Airflow 3 | Find operators that use Airflow sessions, then move each query to the Airflow Python client or REST API |
| Airflow models not for task use | Task code can no longer import and use Airflow database models directly | Replace model reads and writes with API calls or Task SDK methods, and test each replacement |
| Hooks pointed at the metadata database | Hooks such as PostgresHook aimed at the Airflow database may run for now, but the upgrade guide calls this a workaround that will break in future versions | Treat as technical debt, plan a replacement, and avoid building new logic on it |
| Recommended REST API and Python client | The upgrade guide recommends the Airflow Python client over the REST API for task-side access | Confirm which client fits each use case and check that the calling identity has the needed permissions |
| SubDAG removal | SubDagOperator is removed in favor of TaskGroups, so DAGs that nest pipelines with SubDAGs fail to load | Rewrite SubDAGs as TaskGroups and confirm task IDs and run history behave as expected |
| Ruff AIR301 and AIR302 checks | These rules flag Airflow 3 breaking changes in DAG code, including removed database access patterns | Run the checks across the DAG repository, review each finding, and add them to CI before upgrading |
Why Airflow 3 removes database access from tasks
Airflow 3 changes who can reach the metadata database. According to the Airflow security model, workers do not receive database credentials and cannot access the metadata database directly. The metadata database holds connections, variables, DAG state and run history, so a task that can read or write it can affect the whole platform. Moving database access out of task code, and out of the worker process entirely, means a task runs with only the access it needs to do its own work, and it communicates with Airflow through the API rather than through a shared database session.
The same security model is more limited for two other components. The DAG File Processor and the Triggerer rely on software guards, and those guards do not stop a deliberately malicious DAG author. In practice, anyone who can write DAG code should be treated as partly untrusted, because a hostile or compromised author can write code that attempts to reach the database through those components. A platform team should plan for that boundary: restrict who can merge DAG code, require code review for DAG repositories, and keep credentials for external systems out of DAG files. Confirm the exact guard behavior for your Airflow version in the Airflow security model before you write your own trust policy.
For a platform team, the practical decision is how far to trust DAG authors. If authors are fully trusted internal engineers, the worker boundary still reduces accidental damage from a bad query or a leaked connection. If authors span several teams or include contractors, treat DAG delivery as a privileged change path, and review the deployment controls around DAG repositories, the DAG File Processor and the Triggerer as part of the design. Ask your Airflow team to document which identities can change DAG code and which components hold any credential that could reach the metadata database.
Choosing the REST API, Python client or Task SDK for each task
Most scripts that check whether a DAG is paused, when a run last succeeded or which task failed only need to read metadata. In Airflow 3 this work belongs outside the task and should go through the Airflow Python client, which calls the REST API, rather than through a database session. The client keeps your script on a documented interface that Airflow maintains across releases, so a schema change inside the metadata database does not silently break a dashboard or audit job.
- Use the Python client first: wrap read-only calls in a small service account role so monitoring scripts never hold write permissions.
- Confirm the API version: scripts written for
/api/v1need migration to/api/v2before they will run against Airflow 3. - Confirm the authentication method: token-based access replaces basic auth, so check how each script obtains its credential.
- Confirm the data scope: check which DAGs and runs the account can see, especially if you use multi-team features.
- Log the calls: record who queried what, since these reads are often the evidence auditors ask for.
Before retiring any direct database query, list each script, its owner and the fields it reads, then confirm every field is exposed by the API your version provides.
External systems such as CI pipelines, ticketing tools and upstream data services often need to trigger a DAG run, pause a DAG or clear failed tasks. These actions change scheduler state, so they carry more risk than reads. The Airflow Python client is the recommended path, because it sends each change as an authenticated API request that Airflow validates and records. Direct inserts or updates to the metadata database bypass those checks and are no longer supported in Airflow 3.
- Authenticate as a dedicated principal: give each external system its own identity and the narrowest role that allows the action.
- Confirm idempotency: check whether a retried trigger call could start a duplicate run, and pass a stable run identifier where the API supports it.
- Confirm the permitted actions: verify that the role allows only the operations the integration needs, such as triggering but not deleting runs.
- Test failure paths: simulate timeouts and rejected requests to confirm the caller reports failure instead of assuming success.
- Keep change records: tie each external trigger to a change ticket or pipeline run so audit logs can be traced back to a requester.
Confirm with your security team which external systems may change run state, and whether any of them currently write to the database directly.
When a task itself needs a value that Airflow stores, such as a connection, a variable, an XCom or information about its own run, the Task SDK is the path intended for that work. Task code in Airflow 3 runs with a narrow execution interface and does not receive database sessions or models. Hooks such as PostgresHook that connect to the metadata database still work in some setups, but the upgrade guide describes them as a workaround that will break in future versions, so they should not become the default pattern.
- Use the Task SDK for run context: read connections, variables and XCom values through the supported interfaces rather than through database queries.
- Check for hooks against the metadata database: search DAG and plugin code for hooks or sessions that point at the Airflow database and plan their removal.
- Confirm data access separately: a task that reads a business database still needs its own connection, which should live in a secrets backend, not in the metadata database.
- Review task credentials: confirm workers receive only the connections their task needs, since workers do not get metadata database credentials.
- Test with the target version: run each rewritten task against Airflow 3 in staging, because errors often appear only at execution time.
Confirm which tasks currently touch the metadata database, and assign an owner to each rewrite before the upgrade window.
Risks of keeping PostgresHook calls against the metadata database
The hook workaround keeps old DAGs running for a while, but it carries four risks that grow over time. Each card names one risk and the control to ask your team or partner to put in place before the workaround becomes a permanent part of your pipelines.
Breakage in future versions
The upgrade guide describes calling hooks such as PostgresHook against the metadata database as a workaround that will break in future versions. A DAG that works on Airflow 3 today can fail on a later patch or minor release with no code change on your side. Ask for an inventory of every task that touches the metadata database through a hook, with an owner and a dated plan to rewrite each one.
Database credentials reach workers
A hook that connects to the metadata database needs connection details that workers would otherwise never hold. The Airflow security model states that workers do not receive database credentials and cannot reach the metadata database directly. Restoring that path puts a high-privilege secret into task execution, where any code in the task can read it. Ask that the connection be removed from worker environments and that any remaining use be reviewed by the security team.
Authorization and audit paths bypassed
Requests made through the REST API or Task SDK pass through Airflow's authorization checks and produce API-level records. A hook that writes straight to the metadata database skips those checks, so an action such as clearing a task or changing a state may not appear where your auditors look. Ask for a written list of what each hook call changes and confirm where that change is logged.
DAGs coupled to internal schema
Hook code reads and writes Airflow's internal tables, whose columns and relationships are not a stable public interface. A release that renames a column or changes how run state is stored can break queries silently, or write incorrect data without raising an error. Ask that every direct query be covered by a test that runs against each target Airflow version before the upgrade is promoted.
Workaround becomes permanent
Temporary workarounds tend to outlive the project that created them, especially when a DAG still produces correct results. Once the hook call is in production, nobody may schedule the rewrite, and the code will be copied into new pipelines. Ask for a tracked removal date for each workaround, a review gate that blocks new hook calls against the metadata database, and a check in CI that flags them.
Rewriting database-touching tasks in the upgrade sequence
This sequence moves each task that touches the Airflow metadata database off direct access before the Airflow 3 cutover, so the upgrade has a tested rollback point at every stage.
Move to the latest Airflow 2.x
The platform team upgrades the Airflow 2 deployment to the latest 2.x release before any Airflow 3 work begins, because the upgrade guide requires Airflow 2.7 or later and recommends the latest 2.x first. Confirm the target version against the upgrade guide and the version support table, then run the existing DAGs for a full cycle on that release. The step is done when scheduled runs complete without new errors and the running version is recorded in the change ticket.
Run Ruff AIR checks on the DAG repository
The data engineering team runs Ruff with the AIR301 and AIR302 rules across the DAG repository to list Airflow 3 breaking changes. Each finding is logged with its file, DAG and owning team, and the team reviews which findings are automatic fixes and which need a person. The step is done when the repository has a complete findings list with no unowned entries, and the check runs in CI so new code is covered.
Classify each database-touching task
Engineers review every task and operator that opens metadata database sessions, imports Airflow models, or calls hooks such as PostgresHook against the metadata database. Each task is placed in one category: replace with the Airflow Python client or REST API, replace with Task SDK calls, or retire because its purpose no longer applies. The output is a classification register with an owner and a reason for each entry. The step is done when the register covers every finding from the previous step.
Rewrite tasks on the REST API or Task SDK
Developers rewrite each classified task in a feature branch, using the Airflow Python client or REST API for control-plane actions and the Task SDK for task-level logic. The upgrade guide recommends the Python client over the REST API, and it treats hook calls against the metadata database as a workaround that will break in future versions. Code review checks that no branch imports Airflow database sessions or models. The step is done when the branch passes review and the static checks show no remaining direct database access.
Test rewritten DAGs in staging on Airflow 3
The team deploys the branch to a staging environment running Airflow 3 with a copy of the production metadata database and the same executor and worker setup. Testers run each rewritten DAG against representative inputs, compare outputs with the Airflow 2 results, and confirm that the API server and DAG processor run as separate components. The step is done when every rewritten DAG produces matching results, and any difference is logged with a fix or an approved exception.
Remove hook-based workarounds
Engineers delete the remaining hook-based workarounds and any configuration that exposed metadata database credentials to task code. Workers are designed not to receive database credentials, so a task that still needs the connection signals a design gap that must be resolved before cutover. The security team confirms the change against the Airflow security model. The step is done when the search for hook calls to the metadata database returns no results and the credential removal is reviewed.
Back up the metadata database and run the upgrade
The operations team takes a consistent backup of the metadata database and verifies that it can be restored in a test environment before the upgrade starts. They then run the Airflow 3 upgrade in the agreed window, including the database migration and the separate DAG processor. The step is done when the scheduler, API server and workers report healthy, a test DAG completes, and the backup restore check is recorded.
Define the rollback point to Airflow 2
Before cutover, the operations team records the Airflow 2 image tag, the configuration set and the verified backup as a single rollback point, and rehearses restoring it in staging. If production checks fail, the team reverts to that image and restores the backup. The rollback steps and the decision owner are written into the runbook. The step is done when a rollback rehearsal has completed and the runbook names who can call a rollback.
Questions teams ask before rewriting database-touching tasks
How much rewriting effort should we expect for tasks that touch the metadata database?
The effort depends mostly on how many tasks open a database session or use Airflow models directly, and how many of those tasks have logic that is hard to separate from the query. A small set of tasks that read a single status value is usually a short change. Tasks that update run state, write to custom tables or chain several queries need design work, testing and a review of what they should do in Airflow 3. Start with an inventory: search your DAG repository for direct session and model imports, then sort the hits by complexity. Ask any partner to base the quote on that inventory, the number of distinct patterns found, and the test coverage you already have.
Can we keep a hook workaround such as PostgresHook for a while after upgrading?
You can keep it temporarily, but the upgrade guide treats hooks that call the metadata database as a workaround that will break in future versions. Running on that workaround means you carry a known breaking point into every later release, and the code still gives workers a path to the metadata database. Use it only as a bridge for a short, tracked list of tasks, with an owner and a removal date for each. Confirm the exact behavior on the version you run, because support for the workaround may change between releases.
Why does Airflow 3 block direct metadata database access from tasks?
The Airflow security model says workers do not receive database credentials and cannot reach the metadata database directly. The DAG File Processor and Triggerer have software guards, but those guards do not stop a deliberately malicious DAG author. Removing database sessions from task code keeps the database behind the API boundary, so a task that is compromised or written with bad intent has less reach into scheduler state, connections and other DAGs' history. Your security team should review the trust model for DAG authors as part of the plan.
Can our own engineers do this work without an implementation partner?
Yes, if your team already owns the DAG code, has test coverage and understands the Airflow 3 task execution model. The work is mostly code change and testing, and the upgrade guide and Ruff checks give a clear starting list. A partner adds value when many DAGs are affected, when the database queries encode business rules nobody documented, or when the team lacks time to run staging tests alongside normal delivery. Decide based on your inventory, your capacity and the number of owners who must sign off on changes.
How long does the rewrite take and what does the timeline depend on?
No supplied figure gives a duration for this work, so treat any number you hear with caution. The timeline depends on the size of the inventory, the complexity of each database-touching task, the quality of existing tests, how many teams must review changes, and how often staging environments are available. Ask for milestones in the quote: inventory complete, rewrites per DAG group, staging test passes, and production cutover. Ask what is assumed about staging access and code owner availability, because those are common causes of delay.
What must our team provide before the rewrite starts?
Plan for three inputs. First, named code owners for each DAG group who can approve behavior changes and answer questions about business rules. Second, credentials and service accounts that allow the work in development and staging, scoped to the minimum needed, such as access to the Airflow REST API or the Task SDK in a test environment. Third, a staging Airflow 3 environment that mirrors production enough to run the rewritten tasks against realistic connections and data. Confirm how these will be granted and removed, and who keeps records of the access.
How should we handle tasks whose data source is the metadata database itself?
Some tasks read run history, DAG state or other Airflow data to make a decision. In Airflow 3, these should call the REST API or the Task SDK for the information they need, not query tables directly. Where the API does not expose a value, document the gap and decide whether the logic can move to a dedicated reporting copy, an asset or an explicit input, or whether the design should change. Engineers should confirm current API coverage for each value against the version they run, and keep a list of these cases for review before cutover.
What makes rewriting database access take longer
This table compares the factors that stretch a rewrite of metadata database access in Airflow 3, why each one adds effort, and how to reduce it. Read it before estimating scope, since most of the time usually goes to finding every affected task and replacing the data access they depend on, not to the code change itself.
| Driver | Why it adds effort | How to reduce it |
|---|---|---|
| Number of DAGs and operators touching the database | Every direct session or model query in task code fails under Airflow 3, and teams often do not know which DAGs do this until they run them. | Search the DAG repository for session and model imports, then group findings by owner before rewriting anything. |
| Custom hooks and plugins that wrap sessions | A hook such as one built on PostgresHook against the metadata database still runs, but the upgrade guide calls it a workaround that will break in future versions, so each one needs a planned replacement. | Inventory shared hooks and plugins first, then move them to the REST API or Task SDK in one reviewed change. |
| Credentials distributed across teams | Database connection details stored in DAG-side connections or variables must be found, rotated and removed from task reach, which touches several owners. | Confirm with the security model which components should hold database credentials, then retire task-side copies as each DAG is migrated. |
| Missing test coverage | Without tests on the affected tasks, a rewrite can change results silently, and teams may not notice until a scheduled run fails. | Write tests for current outputs before rewriting, and run old and new versions side by side in staging. |
| Tasks that need data the REST API does not expose | Some tasks read or write state that has no equivalent endpoint, so the work may need an API gap analysis or a redesign of where the data lives. | List required data per task early, and confirm endpoint coverage for your Airflow 3 version in the API reference before committing a design. |
| Number of environments to validate | Development, staging and production can run different provider versions, connections and data, so a fix that passes in one may fail in another. | Validate each environment with the same test set and promote changes through the same pipeline. |
When to rewrite now and when a planned stopgap is acceptable
Rewrite the tasks now if you can, and treat the hook workaround as a bridge rather than a destination. Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. A team that keeps running Airflow 2 while leaning on direct database access inherits every future vulnerability in that line without a patch path. The upgrade guide also calls hooks such as PostgresHook against the metadata database a workaround that will break in future versions, so a stopgap adds a second migration on top of the first. If your task inventory is small, with a handful of DAGs that query or update metadata tables, the rewrite to the REST API or Task SDK is usually the cheaper path, because it removes the workaround instead of postponing it.
A planned stopgap is defensible only when the rewrite cannot fit before the upgrade window and you accept the trade-off in writing. Before you accept any hook-based path, confirm with your vendor or partner which Airflow versions the workaround covers, whether it remains supported in the Airflow 3 line you plan to run, and what happens when the hook is removed. Then check your own environment: whether workers can reach the metadata database over the network, which credentials they would need, and how your security team will record that exception. The Airflow security model states that workers do not receive database credentials and cannot reach the metadata database directly, so a hook path gives workers a capability the design deliberately withholds. Write down the owner, the removal date and the tasks covered, and track them in the same change process as the upgrade.
How DeployOpen works on this
DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure. For Airflow 3 without direct metadata database access in tasks, we start by inventorying which tasks reach the metadata database and then rebuild them on the REST API or Task SDK inside a scope agreed in writing before work starts.
How we scope
- Inventory every DAG, operator and hook that opens metadata database sessions or imports Airflow models directly.
- Classify each access as a read, a write, or a workflow that needs a new design, with an owner for each.
- Record the agreed scope, deliverables and timeline in writing, since most engagements are fixed-scope.
How we build
- Rewrite affected tasks to use the Airflow Python client or REST API, or the Task SDK, in your repository.
- Replace any hook calls against the metadata database, and document the temporary workaround and its exposure where one is kept.
- Run Ruff AIR checks and automated tests in your CI before each change is promoted to staging.
How we hand over
- Deliver a written record of each rewritten task, the API or SDK calls it uses, and the credentials it needs.
- Document the security reason for the change: workers do not receive metadata database credentials, so tasks must not depend on them.
- Walk your engineers through the code, tests and remaining exceptions so your team can maintain them.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover.
- Review new DAGs and provider changes against the agreed rules so direct database access does not return.
- Confirm with your own environment and the Airflow project which versions and APIs apply before you schedule work.
