Apache Airflow and dbt Cloud integration

How Airflow can trigger and monitor dbt Cloud jobs

A team that runs its dbt transformations in hosted dbt Cloud can let Airflow own the schedule and the dependencies around those jobs. In practice, Airflow starts a dbt Cloud job at the right point in a larger pipeline, waits for it to finish, and then decides what runs next. The dbt Cloud provider package supplies DbtCloudRunJobOperator to start a job and DbtCloudJobRunSensor to wait on a run that was started elsewhere. Together they let a single dependency graph hold ingestion, dbt transformations and downstream checks, instead of splitting them across two schedulers.

The provider targets the hosted dbt Cloud service, not dbt Core. If your transformations run from dbt Core on your own servers or in your own containers, this package does not apply, and you would invoke dbt directly from Airflow tasks instead. Airflow itself stays in your infrastructure under the Apache-2.0 license, so the orchestration layer is self-hosted while the transformation runtime sits with the dbt Cloud vendor. Confirm with your dbt Cloud account team which account, project and job identifiers, and which API token scope, your Airflow connection will use, and where your data governance rules allow dbt Cloud to run.

Waiting is the part that affects capacity. Deferrable waiting lets the triggerer hold the wait for a long-running dbt Cloud job instead of a worker slot, so you should check that your triggerer is sized and monitored before you rely on it for many concurrent runs. Artifact download, such as the run results and manifest that dbt Cloud produces, is also available through the provider, and you should decide early which artifacts your downstream checks and audit records need. Ask your team to define what a failed dbt Cloud run should do in Airflow, including whether retries re-trigger the job, since that choice determines whether transformations run twice.

What you need before connecting Airflow to dbt Cloud

The integration depends on four layers working together: a supported self-hosted Airflow 3 deployment, the dbt Cloud objects it will call, credentials scoped to those objects, and network and package settings that let workers and the triggerer reach the dbt Cloud API. Confirm each layer before writing the first DAG, because a gap in any one shows up as a failed run or a silent wait.

Supported Airflow 3 deployment

Start with a self-hosted Airflow 3 deployment on a supported Python version. The Airflow 3.3.2 prerequisites list Python 3.10 to 3.14 and Kubernetes 1.30 to 1.35, so confirm your interpreter and cluster version against that page before installing providers. Airflow 2 reached end of life on April 22, 2026, so a dbt Cloud integration built on it inherits no fixes.

Production metadata database

Airflow stores task state, connections and run history in its metadata database. Use PostgreSQL or MySQL rather than the default SQLite backend, which the production guide says is for testing only. The Airflow 3.3.2 prerequisites list PostgreSQL 14 to 18, and MariaDB is not supported. Ask your platform team who backs up this database, because dbt Cloud run IDs recorded in it are what let a retry resume monitoring instead of starting a new job.

dbt Cloud account, project and job IDs

The operator and sensor address work by numeric account ID and job ID, so record both before building DAGs. Confirm with your dbt Cloud admin which project owns the job, whether the job runs on a schedule or only on demand, and whether a separate job exists for each environment. Jobs that other tools trigger can still be monitored from Airflow, but Airflow should own only the runs it starts.

Scoped dbt Cloud service token

Create a service token in dbt Cloud with the narrowest permissions that can start and read the chosen jobs, then store it as an Airflow connection secret rather than in DAG code. Ask your dbt Cloud admin whether the token belongs to a service account that survives staff changes. Rotate it on the same schedule as other pipeline credentials, and confirm that the Airflow UI masks it in logs.

Outbound HTTPS to the dbt Cloud API

Airflow workers run the operator, while the triggerer runs deferred sensors, so both need outbound HTTPS to the dbt Cloud API host. Egress rules are often set per cluster or per subnet, and a blocked triggerer can leave runs looking stuck. Confirm the exact API host for your dbt Cloud region with the provider documentation and your dbt Cloud admin, and test it from the pods or VMs that will run the tasks.

Compatible dbt Cloud provider package

Install the dbt Cloud provider package at a version compatible with your Airflow release, and pin it with the Airflow constraints file so upgrades do not pull in an incompatible build. Check the provider documentation for the minimum Airflow version it supports, and rerun the install in a staging environment after each Airflow upgrade. Keep this package list under version control with the rest of your DAG dependencies.

Configuration sequence for the dbt Cloud provider

This sequence takes a self-hosted Airflow environment from a bare install to a DAG that triggers a dbt Cloud job, waits for it without holding a worker slot, and optionally pulls run artifacts back. Each step closes with a check so your team can confirm the integration works before any production schedule depends on it.

Pin the dbt Cloud provider package
Your platform engineers install the dbt Cloud provider package into the same Python environment or container image that runs your Airflow components, pinned to a specific version and built against your Airflow constraints file. Install it in every image that runs a scheduler, DAG processor, API server, or worker that will import DAGs using these operators. Confirm the install by running the Airflow provider listing command in each image and checking that the dbt Cloud provider appears at the version you pinned. Record that version in your change record so future upgrades start from a known state.
Store the dbt Cloud token in your secrets backend
An administrator creates a dbt Cloud service token with the access the jobs require, then stores it through the secrets approach your organization has chosen, such as a secrets backend or an encrypted connection store, rather than in a file or DAG. The Airflow connection of the dbt Cloud type references that stored value and holds the account details the operators need. DAG code should never contain the token in plain text, since DAG files are readable by anyone with access to the repository and may appear in logs or rendered views. Confirm the setup by opening the connection's test function in the Airflow UI and seeing a successful authentication result against your dbt Cloud account.
Reference the connection and IDs in a DAG
A DAG author adds a DAG that calls DbtCloudRunJobOperator with the connection ID, the dbt Cloud account ID, and the job ID, and sets a task ID and retry policy suited to your schedule. Keep account and job IDs as configuration values rather than hard-coded throughout the file, so the same DAG can move between environments with different dbt Cloud projects. Parse the DAG in your development environment to confirm there are no import errors. Confirm the wiring by triggering the DAG manually from the Airflow UI and checking that the task starts a run in dbt Cloud, with the run ID appearing in the task log.
Wait on the run with the deferrable option
The DAG author sets the operator to deferrable mode or adds DbtCloudJobRunSensor after the trigger task, so that the wait for the dbt Cloud run does not hold a worker slot for the full duration of the job. Deferrable waiting depends on a triggerer process being available in your deployment, so platform engineers confirm that component is running and monitored before enabling it. Choose the terminal statuses the DAG should treat as success or failure and make sure a non-success status fails the task. Confirm the setup by checking in the Airflow UI that the waiting task moves into a deferred state while the dbt Cloud run is in progress, then completes after the run finishes in dbt Cloud.
Confirm a success and a failure path
The DAG author runs the DAG once against a dbt Cloud job that is expected to succeed and checks that the run status in dbt Cloud shows success and that the Airflow task is marked successful. Then the team runs a controlled failing job, either a job known to fail in a non-production project or a test run that the dbt Cloud job is configured to fail, and confirms the sensor or operator raises an error and the Airflow task is marked failed. Both checks matter because a waiting task that silently passes on a failed dbt run will let broken models flow into downstream pipelines. Record both results in the change record before any schedule is enabled.
Enable artifact download where it is needed
If downstream steps or audit processes need dbt run artifacts, the DAG author adds the artifact download step after the run completes and stores the files in a location your data governance rules allow. Confirm with your dbt Cloud administrator which artifacts the jobs generate and whether your plan and job settings make them available for download, since this varies by configuration. Keep artifact storage access as narrow as the data it contains requires. Confirm the step by opening a completed run in the Airflow task log and checking that the expected artifact files arrive in the target location with the expected names.

How dbt Cloud identifiers and run states map into Airflow

This table lists the identifiers and run states a team must keep consistent between dbt Cloud and Airflow, with the owner of each value and where it should be recorded. Read it as a checklist for the DAG repository and configuration review. The main takeaway is that the Airflow connection, account and job IDs must agree across environments, and that a dbt Cloud run status only reaches Airflow as a task outcome when the operator and sensor are set up to wait on it. Confirm exact field names in the provider documentation before writing code.

Item	Owner and where it is recorded	What to confirm
dbt Cloud account ID	Platform team; recorded in the Airflow connection or environment configuration, not hard-coded in DAG files	Matches the dbt Cloud account that owns the target project in each environment
dbt Cloud project ID	Analytics engineering; recorded in a configuration file or DAG repository variable	Points to the intended project after any project is cloned or renamed
dbt Cloud job ID	Analytics engineering; recorded in the DAG repository alongside the schedule that calls it	Refers to a job that runs the expected dbt commands and is not triggered by a second schedule
Airflow connection ID	Airflow administrators; stored in the metadata database or a secrets backend, referenced by name in DAG code	Connection holds the dbt Cloud API token and is named identically across dev, staging and production
Run ID returned by the operator	Airflow runtime; returned by the trigger operator and passed to the sensor through XCom or the operator's output	Confirm the exact return key in the provider documentation before a downstream task reads it
dbt Cloud run status values	dbt Cloud service; read by the sensor and mapped to Airflow task state	Success, error and cancelled outcomes map to task failure paths; confirm how intermediate states are handled
Run metadata in XCom	DAG authors; passed from the trigger task to downstream tasks such as artifact download or alerting	Keep payloads small, avoid secrets, and confirm which fields the provider exposes

Choosing between blocking sensors and deferrable waiting

A sensor that waits for a dbt Cloud run occupies a worker slot for the whole wait while it polls the dbt Cloud API. Deferrable mode hands the waiting to the triggerer, so the worker slot is released while the run executes. The choice is mostly a capacity decision: how many long dbt runs will overlap, and how many slots your executor can spare for other DAGs.

Slot consumption: A blocking sensor holds one worker slot for each waiting task until the dbt Cloud job finishes, so long transformations can starve other DAGs on a shared pool.
Triggerer capacity: Deferred waits run in the triggerer process. Confirm how many triggerers you deploy, their resource limits, and how waiting tasks recover after a triggerer restart.
Executor fit: Confirm how your executor handles deferred tasks in Airflow 3, using the executor documentation as the starting point, because pool sizing differs between executors.
Scheduler and parser load: Multiple schedulers are supported with database row-level locks. Measure polling and scheduling load after rollout instead of assuming the default settings will hold.

Before setting the default mode for all dbt DAGs, check triggerer health alerts and worker slot utilization over a period that includes your longest dbt Cloud runs.

What the dbt Cloud provider does not do for you

The dbt Cloud provider targets the hosted dbt Cloud service. It triggers and monitors jobs in dbt Cloud, and it does not run dbt Core inside Airflow. If your transformations run dbt Core, the provider is not the right tool, and you need a separate pattern such as a container image or a bash-based dbt invocation from a task. That pattern is your design to build and test, including how the project, profiles and warehouse credentials reach the worker, and how dbt artifacts are stored. Confirm the current scope of the provider in its documentation before you commit to either path.

Several operational gaps stay with your team. Alerting on failed dbt Cloud runs is not something the operator replaces. Decide which failures page on-call, how retries avoid duplicate notifications, and what each alert links back to. Lineage and metadata beyond what dbt Cloud exposes also need a plan, since a dbt Cloud job run does not give Airflow a full picture of your data assets. Promotion between dbt Cloud projects for development, staging and production is handled outside Airflow, so define how job IDs and connection settings change across environments. Rotation of the dbt Cloud service token is your process as well, with a named owner, a rotation schedule and a tested update of the Airflow connection.

Finally, check for schedules that already exist in dbt Cloud. A job with its own schedule can run alongside a DAG that triggers the same job, which produces duplicate or overlapping runs. Inventory every dbt Cloud schedule before you cut over, assign each job to one scheduler, and disable the duplicate. Ask your dbt Cloud administrator who can edit these schedules, since that access is separate from the Airflow connection. Write this ownership down as part of the migration plan so that the first failed run does not reveal an unowned schedule.

Securing the token, the network path and the run history

The dbt Cloud integration gives an Airflow deployment a credential that can start production transformations and read their results. The controls below cover where that credential lives, how little it can do, where traffic can go, who changes the DAGs that use it, and what evidence remains afterward. Each one must be confirmed against your own environment and your dbt Cloud plan.

Keep the token out of DAG code

Store the dbt Cloud service token in a secrets backend or in Airflow connection storage, never as a literal in a DAG file or an environment variable baked into an image. Confirm that connection passwords are encrypted at rest with the Fernet key, that the value is masked in task logs and the UI, and that the metadata database backups do not expose it in plain text to people who should not see it. Also confirm who can read connections in the API server.

Scope the token to the jobs

Use a service token with the fewest permissions that still let Airflow start and read the jobs it owns. Ask dbt Cloud administrators which permission sets are available on your plan and whether the token can be limited to specific projects or environments. Separate tokens for development and production make revocation and audit simpler. Record who owns each token and when it was last rotated.

Restrict egress to the dbt Cloud API

Limit outbound traffic from Airflow workers and the triggerer to the dbt Cloud API host over HTTPS, and deny general internet access from those hosts. Because deferrable waiting runs on the triggerer, that component needs the same restriction. Confirm the exact host names and any regional endpoint with dbt Cloud documentation, and test the rule with a failing connection to an unlisted host before relying on it.

Review changes to production DAGs

Any DAG that triggers a production dbt Cloud job can change what runs in your warehouse, so edits should pass through the same review as other production code. Require pull request approval, protect the branch that Airflow syncs from, and separate the people who can merge from those who can change Airflow connections. Confirm how your DAG delivery method, such as Git sync or a bundle, enforces this in practice.

Retain run IDs and logs

Keep the dbt Cloud run ID that each Airflow task returns, along with the Airflow task log, the triggering user or DAG version, and the timestamps. Together these connect a warehouse change to the scheduled job that caused it. Confirm your log retention in Airflow, how long dbt Cloud keeps run history on your plan, and whether these records meet the retention period your auditors expect.

Common dbt Cloud failures in Airflow and how to fix them

This table lists six failure modes teams hit when running dbt Cloud jobs from Airflow, with the likely cause and the fix for each. Use it as a triage checklist, and confirm exact parameter names against the dbt Cloud provider documentation before changing code.

Failure	Likely cause	Fix
Authentication errors	The dbt Cloud API token has expired, was revoked, or lacks access to the target account or project.	Create a token with the scope the job needs, store it in an Airflow connection, rotate it on a schedule, and confirm the connection's account ID matches the token.
Wrong job or account ID	The job ID or account ID points at a development or staging environment, or at a job in another dbt Cloud account.	Copy IDs from the target environment's job page, pin them in configuration per Airflow environment, and add a startup check that fails fast if the job name does not match.
Sensor timeouts on long runs	The sensor's wait limit is shorter than the dbt run, so the task fails while dbt Cloud keeps running the job.	Set the sensor timeout above the longest normal run, add retries only for transient API errors, and check the run status in dbt Cloud before rerunning.
Deferred tasks not resuming	The triggerer is not running, is not reachable by the scheduler, or is too small to hold the waiting triggers.	Confirm a triggerer process is running in each deployment, watch its logs for stalled triggers, and size it for the number of concurrent deferred dbt waits.
Duplicate job runs after retries	A task retry re-sends the trigger request, so dbt Cloud starts a second run of the same job.	Use the provider's option to wait for completion within the same task where it fits, make downstream steps depend on the first run's ID, and avoid blind retries on the trigger step.
Artifact download fails	The run had not finished when the download was requested, or the artifact file name or path does not match what dbt Cloud produced.	Wait for a successful run status before downloading, list the run's artifacts first, and check the artifact path against the provider's documented parameters.

Questions teams ask before connecting Airflow to dbt Cloud

Can our team set up the Airflow to dbt Cloud integration without outside help?

Yes, for a small set of DAGs with one clear owner, provided the team already runs Airflow in its own infrastructure. The work itself is modest: a dbt Cloud API token, an Airflow connection, and DAG code that triggers jobs and waits for their results. The harder parts are operational. Someone has to own the connection, rotate the token, review failed runs, and decide how sensors or deferrable waiting behave under load. If your team has not yet operated Airflow 3 in production, a fixed-scope engagement can cover the build and handover while your engineers learn the system.

What makes the dbt Cloud integration effort larger or smaller?

Effort grows with the number of dbt Cloud jobs that Airflow must trigger, the number of DAGs that depend on them, and how many environments exist, such as development, staging and production, each with its own dbt Cloud project and credentials. Waiting strategy matters too: a sensor holds a worker slot while it polls, while deferrable waiting hands the wait to the triggerer, which needs its own capacity planning. Other drivers include how secrets are stored, which network paths must be approved between Airflow and the hosted dbt Cloud service, and how much artifact handling the pipelines need. Each of these should be listed in the written scope so the quote reflects them.

What costs apply when connecting Airflow to dbt Cloud?

There are three separate cost components. Airflow is open source under the Apache License 2.0, so there is no license fee from the project itself, but your infrastructure, storage, and staff time are real costs. dbt Cloud is a hosted service with its own subscription, billed by dbt Labs, and that subscription is independent of Airflow. The implementation effort is a one-time fixed-scope quote from DeployOpen, based on the written scope of jobs, environments, and handover. Ongoing support or upgrades are optional retainers. Ask your vendors for their current subscription terms directly, since DeployOpen does not publish a price list.

How long does the Airflow and dbt Cloud integration take?

The duration depends on the agreed scope: how many jobs and DAGs are involved, how many environments need wiring, and how quickly your network and access approvals arrive. DeployOpen does not give a fixed duration before scope is known. Once the scope is written down, the timeline is set in the same document and agreed before work starts. Ask for milestones that separate the connection setup, the test project validation, and the production cutover, so you can see where time is spent and where delays could come from.

What do we need to provide before the integration work starts?

The work needs several items from your side. You need dbt Cloud admin access, or an admin who can create a service token with the permissions the jobs require. A dbt Cloud test project or non-production environment should exist for validation before anything touches production. Your security team must approve the network paths from the Airflow environment to the hosted dbt Cloud API. You also need an Airflow environment you control, running a supported version, with a place to store the connection secret, such as a secrets backend. Confirm the exact token scopes and network rules with your dbt Cloud administrator and security team, since these depend on your account setup.

Does the integration work with dbt Core?

No. The Airflow dbt Cloud provider targets the hosted dbt Cloud service and its job API, so it does not run or monitor dbt Core projects. If your transformations run dbt Core, you will need a different approach, such as running dbt commands in a container or on a worker through a separate operator, and that is a different scope. Confirm which dbt product your team uses before you plan the work, because the choice changes the operators, credentials, and where the transformations actually execute.

How DeployOpen scopes and delivers this integration

DeployOpen plans, implements, documents and supports self-hosted Apache Airflow inside your own infrastructure, including the dbt Cloud provider connection, job triggers and run monitoring. Work starts from a written scope that states which dbt Cloud projects, jobs and downstream dependencies are covered.

How we scope

Inventory the dbt Cloud accounts, projects, environments and jobs that Airflow will trigger or monitor
Record which DAGs depend on each dbt job and which downstream tasks wait on its results
Agree in writing how dbt Cloud API tokens are issued, stored and rotated in your secrets system
Confirm which artifacts, if any, Airflow must download and where they are stored afterward

How we build

Configure the dbt Cloud connection and test it against a non-production dbt Cloud account first
Build DAGs with DbtCloudRunJobOperator and DbtCloudJobRunSensor, using deferrable waiting where the triggerer is sized for it
Test failure paths, retries and timeouts so a failed dbt run cannot be mistaken for success
Document where dbt Core work, which the provider does not cover, will run and how it is orchestrated

How we hand over

Deliver the DAG code, connection configuration and runbook in your own version control
Walk your platform and data teams through triggering, monitoring, rerunning and troubleshooting dbt Cloud jobs
Record the scope, decisions and open items in a handover document your team can maintain
Agree acceptance criteria in the written scope before the handover is signed off

How we support

Offer ongoing support, provider upgrades and Airflow upgrade planning as a retainer after handover
Review provider version changes and dbt Cloud API changes before they reach production
Keep fixed-scope engagements separate from retainer work so each has its own written terms
Ask any partner to confirm support hours, response expectations and terms in the quote