How the Apache Airflow Databricks integration runs jobs, notebooks and SQL
Teams that already schedule pipelines in Airflow and need Databricks work in the same dependency graph should understand one division of labor first. The Databricks provider package calls the Databricks REST API from Airflow tasks. The processing itself runs on Databricks compute, so Airflow holds the schedule, the dependencies and the run status, while Databricks runs the jobs, notebooks and SQL. This page covers three patterns: submitting a one-time run with submit-run, triggering an existing Databricks job with run-now, and building a Databricks Workflow from Airflow tasks with the DatabricksWorkflowTaskGroup. It also covers the choice between them, the setup, security and the failures you should plan for.
Before you choose a pattern, confirm the provider package version against your Airflow 3 release, since provider packages are versioned separately from Airflow itself. Also confirm the operator names, parameters and the Databricks workspace API endpoints against the vendor documentation for the workspace you will call, because Databricks changes its API and job model over time. A useful first check is whether your team already defines Databricks jobs as code or in the workspace UI, because that decides whether Airflow should trigger existing jobs or create runs with their own configuration.
The provider runs its waits on the Airflow side, so the operators you choose determine how much Airflow worker capacity a long Databricks run consumes. Deferrable operators hand the waiting to the triggerer, which frees the worker slot while the Databricks run is in progress. Decide early whether your deployment has a triggerer component sized for the number of concurrent waits you expect, and plan for a Databricks run that outlasts a worker restart. Retries need care too: a retried submit can start a second run unless the task is designed to detect an existing run, so review idempotency before enabling automatic retries on these tasks.
What a team needs before connecting Airflow to Databricks
The Databricks integration depends on decisions made on both sides before the first DAG runs. Airflow needs a supported version, a metadata database and workers that can reach the workspace, while Databricks needs an identity, permissions and a compute policy that the pipelines are allowed to use.
Airflow release and Python runtime
Plan on Airflow 3.3.2, the current patch release in the supplied version table, and run Python 3.10 to 3.14 on the workers, triggerer and scheduler. Confirm the provider package version that matches your Airflow release before pinning anything, because the Databricks provider is versioned separately from Airflow core. Check the compatibility notes in the provider documentation rather than assuming the newest provider works with an older Airflow line.
Metadata database and sizing
Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release as the metadata database. MariaDB and SQLite are not suitable for this role, and the production guide warns that SQLite can lose data. Allow at least 4 GB of RAM for the Airflow components, and size workers separately if Databricks polling runs alongside heavier tasks.
Non-personal Databricks identity
Create a service principal or service identity in the Databricks workspace for Airflow, not a named user account. Grant it only the job, run and cluster permissions that the DAGs need, such as running existing jobs or submitting one-time runs on approved compute. Ask your Databricks administrator to confirm which workspace objects the identity can read and modify, and record that list with the DAG owners.
Agreed compute policy
Agree with the Databricks administrators which cluster policies, instance pools and runtimes Airflow-triggered runs may use. Without an agreed policy, a DAG can start expensive clusters or runtimes that nobody approved. Decide who owns cluster sizing, who can change the policy and how cost is attributed to each pipeline, so that the Airflow connection points at policy-controlled compute only.
Network reachability to the workspace
Confirm that every Airflow worker and triggerer can open outbound HTTPS connections to the Databricks workspace host, and that egress rules, proxies and firewall allowlists permit it. Deferrable operators depend on the triggerer reaching the workspace while it polls run status, so a worker that can reach the host but a triggerer that cannot will fail in a way that is harder to diagnose. Test both components from their real network segments.
Connection storage and secrets
Store the Databricks host and credentials in an Airflow connection backed by your chosen secrets store, not in DAG code or environment files. Decide who can create and rotate the connection, and confirm how token rotation will be handled without redeploying DAGs. Review the Airflow security documentation for how connection values are masked in logs and the UI.
Submit-run or run-now: which call fits the team's job design
The choice between submit-run and run-now comes down to who owns the job definition. With a submit-run call, the DAG sends a complete run specification to Databricks each time it executes, including the compute, libraries and task definition. This suits pipelines where the data engineering team owns everything about a run, such as one-off backfills or jobs whose cluster shape changes with each input. With a run-now call, the DAG only starts a job that already exists in the workspace, identified by its job ID. This suits jobs that a data platform team defines, versions and governs in Databricks, where Airflow's role is scheduling and sequencing rather than defining the work.
Parameters travel differently in each case. With submit-run, the specification and its parameters live in DAG code, so a change to cluster size or a library version is a change to the Airflow repository and goes through its review process. With run-now, the job's settings stay in Databricks and the DAG passes only run-time parameters, so the platform team can change the underlying job without touching the DAG. Confirm the current operator names, parameter fields and how each call handles job parameters and task-level overrides in the Databricks provider documentation before writing DAG code, since these fields change between provider releases.
Each choice shapes audit evidence. A run-now design leaves a single authoritative definition in Databricks, with its own version history and permissions, which is usually easier to show an auditor. A submit-run design puts that evidence in the Airflow repository, so change control depends on code review, branch protection and the DAG deployment process described elsewhere on this page. Many teams use both: run-now for governed production jobs and submit-run for exploratory or per-run work, with a written rule for which pattern a new pipeline must use and who approves exceptions.
Choosing between submit-run, run-now and Databricks Workflows
Use this pattern when the pipeline definition belongs in the DAG. A submit-run style task sends a one-time run with its full specification, so the DAG file becomes the record of what the Databricks job runs. Changes to libraries, parameters or compute move through the same review and promotion path as other DAG changes.
- Ownership: The team that writes the DAG also owns the Databricks run definition, and changes ship with DAG releases.
- Cluster spec: The compute and task configuration sits in DAG code or in a file the DAG reads, so it is versioned with Airflow. Confirm how your provider version expects that spec to be passed.
- Failure surfaces: A failed Databricks run appears as a failed Airflow task, and the run details are available from the task logs. Confirm which run identifiers are pushed to XCom.
- Retries: Confirm whether a retry starts a new Databricks run before enabling automatic retries on non-idempotent jobs.
- What to test: Run the DAG in a non-production workspace with the same spec used in production, and check that cluster settings and libraries match what the code declares.
Check that every task in this pattern has a clear owner for the Databricks side, since no one else reviews the run definition.
Use this pattern when the platform or analytics engineering team already maintains a Databricks job and Airflow should only trigger it on schedule. A run-now style task starts an existing job by its identifier, so the job definition stays under Databricks controls. Airflow contributes the schedule, the dependencies and any per-run parameters.
- Ownership: The Databricks job owner controls the definition, and the data team controls when it runs and what depends on it.
- Cluster spec: Compute is defined on the Databricks job, so the DAG does not carry cluster sizing. Confirm which per-run overrides your provider version accepts.
- Failure surfaces: Airflow reports the run state that Databricks returns. A job that fails inside Databricks shows as a failed Airflow task, and the error detail lives in the Databricks run output.
- Drift: Job edits made in Databricks can change behavior without a DAG change, so record the job identifier and the expected job version in the DAG documentation.
- What to test: Trigger the job with each parameter set the DAG passes, and confirm that the identity behind the Airflow connection has permission to run the job.
Confirm who is paged when the Databricks job fails, because the Airflow task alone does not show the cause.
Use this pattern when several related Databricks tasks should run as one Databricks Workflow and Airflow should schedule that workflow with its own upstream and downstream dependencies. The DatabricksWorkflowTaskGroup lets the group appear in Databricks as a workflow, so the task graph is visible on the Databricks side as well as in Airflow. Ownership is shared, so the agreement between teams needs to be written down.
- Ownership: Both teams touch the workflow. Decide in advance who edits task definitions, who changes shared compute and who handles repairs.
- Cluster spec: Compute for the group is declared with the task group, and tasks in the group are expected to share it. Confirm the exact settings your provider version supports.
- Failure surfaces: A failed task inside the workflow is reflected in the Airflow task group. Confirm how individual task reruns are handled from either side.
- Visibility: Check that the workflow and its runs appear where the Databricks operations team expects to look.
- What to test: Verify dependency order, cluster reuse across tasks, behavior when one task fails midway, and the rerun path from both Airflow and Databricks.
Record which system is the source of truth for run history before the first production schedule is enabled.
Configuring the Databricks connection in the order that avoids rework
This sequence moves from identity and secrets to a verified first run, so each step rests on a check the previous one has already passed. Teams that follow it in order find permission and credential problems in staging rather than in production schedules.
Create a dedicated Databricks service identity
A Databricks administrator creates a service principal used only by Airflow, separate from any personal user account. Grant it only the job, cluster and SQL permissions that the planned DAGs need, scoped to the workspace folders, jobs and SQL warehouses those DAGs touch. Confirm the scope by listing the jobs and warehouses the identity can see in the workspace and checking that nothing outside the planned set appears. Confirm the exact permission names and how the workspace applies them with Databricks documentation and your workspace administrators, since they vary by workspace setup.
Store the credential in the Airflow secrets backend
Generate the service principal's credential in Databricks and write it to the secrets store your Airflow deployment already uses, such as a Vault or cloud secrets manager configured as the Airflow secrets backend. Keep it out of the metadata database, DAG files and environment variables baked into images. Check the result by confirming that the secret path resolves from the Airflow scheduler and worker environments, and that the value does not appear when you inspect connection records in the Airflow UI.
Define the Airflow connection with a secret reference
Create a Databricks connection in Airflow that holds the workspace host and a reference to the stored credential, not the credential itself. Platform engineers or the Airflow administrator do this, and the connection is shared across DAGs only through its connection ID. Close the step by running the connection test from the Airflow UI or CLI and reading the result. A successful test shows the workspace responded to the authenticated request, which is the first evidence the identity and secret path both work.
Pin the Databricks provider in the Airflow image
Add the Databricks provider package to the Airflow image at a specific version that your team has tested against your Airflow 3 release, installed with the same constraints file used for the rest of the image. Do not install it ad hoc on running workers. Verify by starting the scheduler, worker and triggerer containers from the new image and checking that the Databricks operators import without errors in each one, since deferrable operators need the triggerer to load the same code.
Build one staging DAG with run-now or submit-run
A DAG author builds a single small DAG in staging that runs one existing Databricks job with run-now, or submits a one-time run with submit-run when the job definition should live in the DAG. Choose run-now when the job is already defined and managed in Databricks, and submit-run when Airflow should own the cluster specification for that run. Trigger the DAG once. The check is a log line from the task that shows the Databricks run ID returned to Airflow, and the same run visible in the Databricks run history with matching parameters.
Enable deferrable mode and confirm the triggerer
Switch the long-running Databricks task to deferrable mode so the worker slot is released while Databricks runs the job, and the triggerer polls for completion. Confirm the triggerer process is running and healthy in the deployment, then trigger the DAG again. The check is that the task enters a deferred state in the Airflow UI, the worker slot is free during the wait, and the task resumes and completes once the Databricks run finishes. If the task fails to resume, the triggerer is not running or cannot reach the workspace.
How Airflow concepts map to Databricks objects
This table matches each Airflow object to the Databricks object it drives, the team that should own it, and where to edit it. Read it before splitting work between the platform and data teams, because retries and parameters exist in both systems and a change on one side can quietly override the other. The main takeaway is to decide which layer owns each setting before the first DAG goes live.
| Airflow concept | Databricks object | Who owns it | Where to change it |
|---|---|---|---|
| Airflow connection | Workspace host and service identity (service principal or token) | Platform team | Connection store in Airflow, backed by your secrets backend, not DAG code |
| DAG run | One job run, started by a submit-run or run-now operator | Pipeline code (data team) | DAG schedule and operator choice in the DAG file |
| Task instance | A task inside a Databricks Workflows job, when using DatabricksWorkflowTaskGroup | Data team | Task definitions in the DAG file, with job settings applied on deploy |
| Operator retries | Retry attempts for the Airflow task that launches or waits on the run | Pipeline code (data team) | retries and retry_delay arguments on the operator |
| Run retries | Retry attempts for the job run or task inside Databricks | Data team | Job or task retry settings in the Databricks job definition |
| Run output | Values the provider returns or pushes to XCom, such as the run identifier | Pipeline code (data team) | Confirm the returned fields for your provider version before reading them downstream |
| DAG parameters | Job parameters or notebook parameters passed into the run | Data team | params and templated arguments in the DAG file, with defaults set in the job |
How deferrable operators free worker slots and what they require
A non-deferrable Databricks operator holds an Airflow worker slot for the whole time a Databricks run is executing, which means a few long jobs can consume the pool or concurrency limits that every other DAG depends on. A deferrable operator hands the waiting to the triggerer, a separate Airflow component that polls the external run asynchronously, so the worker slot is released until the run reaches a terminal state. The saving is real only if the triggerer is deployed, healthy and sized for the number of runs that will be waiting at the same time, and that count is set by your schedule, not by the number of DAGs in the file system.
Decide which tasks should defer by asking how long each run waits and how many of them overlap. Long Databricks jobs and notebook runs that sit behind a daily or hourly schedule are the strongest candidates, while short SQL statements that finish in seconds gain little and add triggerer load. Check the deferrable argument on each operator in the Databricks provider release you have installed, since support differs between operators and versions, and confirm it in the provider documentation for that release before you rely on it. Keep the decision in code review so that a new DAG does not quietly add waiting load to a shared triggerer.
Monitor the triggerer as a production dependency. Track how many triggers are active, how long tasks sit in the deferred state before they resume, and whether the triggerer process restarts or falls behind on its event loop. Confirm the triggerer count and placement in your deployment, whether that is a Kubernetes Deployment or a process on a virtual machine, and plan a restart and failover procedure for it. Ask your platform team to alert when deferred tasks stop resuming, because a stalled triggerer fails silently from the DAG's point of view and can leave runs apparently waiting for hours.
Securing the Airflow to Databricks connection
A Databricks connection from Airflow carries a workspace token, compute rights and access to production data, so it needs the same controls as any privileged integration. The six cards below cover where credentials live, what identity they carry, how traffic and audit trails flow, and who is allowed to change the DAGs that spend money and touch data.
Store credentials in a secrets backend
Keep the Databricks host and token in an Airflow secrets backend or an external vault, not in the metadata database or DAG files. Name one team that owns rotation, set an expiry on every token, and document how a rotated value reaches running workers and the triggerer. Ask whether the backend you plan to use can be read by workers only, and confirm how Airflow connection lookups behave during a rotation window.
Scope the service identity narrowly
Create a dedicated Databricks service principal for Airflow rather than reusing a personal account. Grant it run permission on named jobs, the specific clusters or policies it may use, and only the catalogs and schemas its pipelines read or write. Review those grants whenever a DAG adds a new job, because a broad identity turns any DAG change into a wider access change.
Keep secrets out of DAGs and logs
Reference connection IDs in operators and never paste tokens, cluster IDs with embedded settings or notebook parameters that contain credentials into DAG code. Check task logs, rendered templates and XCom values for leaked secrets, and confirm that masking is applied to connection fields in your Airflow version before you rely on it.
Limit egress to the workspace host
Allow workers and the triggerer to reach only the Databricks workspace endpoint and the storage or secrets services they need. Deferrable operators poll from the triggerer, so that component needs the same outbound rule as workers. Ask your network team to confirm the exact hostnames for your cloud and region, and test a blocked destination to prove the rule holds.
Correlate audit trails across both systems
Record the Airflow run ID, task ID and try number alongside the Databricks run ID for each submitted job, and keep both logs for the retention period your policy requires. Without that mapping, an incident review cannot tie a Databricks job back to the DAG change or manual trigger that started it. Confirm where each system stores its logs and who can read them.
Review who can change launching DAGs
DAG authors can choose which job runs, which cluster spec is passed and which tables a task reads, so they can affect compute spend and data access even without direct Databricks rights. Apply code review and protected branches to any DAG that launches jobs with cluster creation rights, and require a second approver for changes to cluster specs or service identity settings.
Common Databricks failures in Airflow and how to fix them
This table lists the failures teams most often hit when Airflow drives Databricks, with the likely cause, the check to run and the fix for each. Read each row alongside the Airflow task log and the Databricks run page named in it, since the two usually show the same problem from different sides. Start with the authentication and parse rows if nothing runs at all, and with the triggerer and duplicate-run rows if work runs but behaves oddly.
| Failure | Likely cause and check | Fix and where to look first |
|---|---|---|
| Authentication failure on connection test or first task | The Databricks connection uses a token or client secret that has expired, been rotated or been copied with whitespace. Check the connection's host and credential fields and test with a non-production token. | Replace the credential in your secrets backend, not in DAG code, then rerun the task. Open the Airflow task log for the 401 or 403 message and the Databricks account audit or token list. |
| Run stuck in a waiting state | The triggerer process is down, unreachable or undersized for the number of deferred tasks, so no one polls the Databricks run status. Check triggerer health and its logs. | Restart or scale the triggerer and confirm it is running in your deployment. Then check the Databricks run page to confirm the job is actually progressing, and clear the task only after the triggerer is healthy. |
| Duplicate Databricks runs after a retry | A submit-run task created a new run each time Airflow retried it, because the earlier attempt's run was not reused. Compare run IDs for one Airflow task instance across attempts. | Prefer run-now for jobs already defined in Databricks, and review retry settings on submit-run tasks. Check the Databricks jobs run list and the Airflow task log for repeated run IDs. |
| Task timeout while the cluster starts | Cluster start or library installation takes longer than the task's execution timeout, so Airflow marks the task failed while Databricks is still provisioning. Compare cluster event logs with the Airflow timeout. | Raise the task timeout for the Databricks step, use a pool or an existing cluster where your policy allows, and reduce library installation at startup. Open the Databricks cluster event log and the run page for the pending state. |
| DAG import errors after a provider upgrade | The Databricks provider version changed operator names, parameters or import paths, or it does not match the installed Airflow version. Check the DAG import error list and the provider's installed version. | Pin the provider to a version compatible with your Airflow release, update changed imports, and test parsing in staging before promotion. Open the Airflow DAG processor log and the provider changelog. |
| Missing run output or empty XCom | The Databricks task finished but returned no value, or the notebook did not call exit with a result, so downstream tasks received nothing. Check the notebook's exit value and the XCom entry for the task. | Have the notebook or job write results to a known table or path and pass only identifiers through XCom. Open the Databricks notebook output on the run page and the Airflow XCom tab for the task. |
Questions teams ask before connecting Airflow to Databricks
How much effort does an Airflow to Databricks integration take?
The effort depends on how many jobs move, how the workspace is governed, and how much you rely on deferrable operators. A small set of notebook or job runs with one workspace and one service identity is a modest build. Effort grows with each additional job family, each environment you need to test in, each team that authors DAGs against shared Databricks resources, and each Databricks Workflows task group you want to keep in step with Airflow scheduling. The main drivers are:
- Number of jobs and whether they use submit-run or run-now patterns
- Governance model: who may create clusters, jobs and connections
- Use of deferrable operators, which needs triggerer capacity planned alongside workers
- Staging and test coverage before production promotion
Ask any partner for a written scope that lists these items so the quote reflects your actual footprint.
Can our in-house team connect Airflow to Databricks without a partner?
Yes, a team with working knowledge of Airflow 3 and Databricks administration can build this integration. The provider covers job submission, notebook runs, SQL operations and workflow task groups, so the core work is configuration, identity, testing and operations rather than custom code. Outside help usually adds value in three places: a first design review of the job pattern and identity model, a tested staging pipeline before the first production DAG, and a documented runbook for failures and retries. If your team has not yet run a self-hosted Airflow in production, the deployment and metadata database decisions often matter more than the Databricks connection itself, and those are worth covering in the same review.
What do we need to provide before the integration work starts?
You need to provide four things: workspace administrator access, or a named administrator who can create and scope identities and connections; a decision on the service identity Airflow will use to call Databricks; network paths from your Airflow environment to the Databricks control plane and workspace endpoints, including any proxy or private connectivity rules; and a named owner for DAG code who approves changes and promotions. Without the owner, pipeline changes tend to drift between teams. Confirm with your Databricks account team which authentication methods your workspace supports before you choose the identity model, since that choice shapes how credentials are stored and rotated.
How long does the integration take to deliver?
The duration follows from the agreed scope and cannot be stated as a fixed figure before that scope is written. Timing depends on the number of jobs, the number of environments, how many identity and network decisions are still open, and how much testing your change process requires. Work that is already documented and has a named owner moves faster than work that still needs design decisions. Ask for a timeline tied to the written deliverables, with the dependencies on your access, network and approval steps listed separately so you can see which parts of the schedule you control.
How do Databricks compute charges relate to Airflow hosting costs?
Databricks bills its compute separately from the cost of running Airflow itself. Airflow hosting costs cover the infrastructure that runs the scheduler, API server, DAG processor, workers, triggerer and metadata database in your environment. Databricks charges come from the clusters, jobs and SQL resources that your runs consume. Deferrable operators change the Airflow side of this picture by letting the triggerer wait on a run instead of holding a worker slot, but they do not change Databricks compute usage. Confirm the billing model, any discounts and the way job compute is metered with the Databricks account team, because those terms determine the total cost of the pipelines.
Does DeployOpen support the integration after go-live?
Yes, ongoing support, upgrades and continuous testing are available as retainers. A retainer can cover provider package upgrades, Airflow version changes, review of new DAG code that touches Databricks, and investigation of failed or stuck runs. Scope, response expectations and the list of covered components are agreed in writing before the retainer starts. Because provider packages and Airflow releases change over time, ask for the upgrade and test steps to be documented as part of the handover so your own team can carry routine changes between retainer periods.
How DeployOpen scopes and delivers the Airflow to Databricks integration
DeployOpen plans and builds the Apache Airflow to Databricks integration inside infrastructure you control, under a written scope agreed before work starts. The work covers provider setup, job patterns, secrets placement and operating runbooks, handed over to your team with documentation.
How we scope
- Inventory the Databricks jobs, notebooks and SQL statements Airflow must start, and which ones need Workflows instead
- Confirm the Airflow 3 deployment, executor model and metadata database the integration will run on
- Agree which Databricks workspaces, service principals and connections are in scope before build begins
- Engagements are fixed-scope, with deliverables and timelines written down before work starts
How we build
- Install and configure the Databricks provider, then set connections with credentials kept out of DAG code
- Build example DAGs for submit-run and run-now patterns, and deferrable waiting that frees worker slots
- Test failure, retry and cancellation behavior against a non-production Databricks workspace first
- Document job ID handling so reruns do not start duplicate Databricks runs
How we hand over
- Deliver DAG code, connection definitions and configuration in your version control system
- Provide a runbook covering reruns, stuck runs, credential rotation and provider version upgrades
- Walk your platform and data engineering teams through the design decisions and trade-offs
- Hand over test results and a list of open items that your team should confirm in your environment
How we support
- Ongoing support, provider upgrades and continuous testing are available as retainers
- Retainer scope is agreed in writing, including which Airflow and Databricks changes it covers
- Revisit the integration when you upgrade Airflow or change Databricks workspace configuration
- Quotes follow from the written scope, so ask for each component to be priced separately
