Apache Airflow and HashiCorp Vault secrets backend

Teams running self-hosted Airflow often keep connection passwords in the metadata database. The Vault secrets backend moves connections and variables into Vault, and this page explains how to plan and verify that move.

Book a callTalk to us
On this page

How the Airflow Vault secrets backend moves connections out of the database

Airflow stores connections and variables in its metadata database by default, which means every DAG author and operator who can read that database can also read the passwords, tokens and keys that pipelines use. The Vault secrets backend moves that lookup out of the database and into HashiCorp Vault. It is configured through Airflow's secrets backend option, so the switch is a configuration change on the scheduler, workers, DAG processor and API server rather than a change to DAG code. Airflow is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0), and the project itself offers no paid edition, so the cost of this work is the cost of the Vault service you run and the engineering time to wire it in.

At runtime, when a task or hook asks for a connection or variable, Airflow consults the configured backend first and reads the secret from a Vault path built from a mount point, a path prefix and the object's name. Airflow then falls back to the metadata database only if the backend does not return a value. That fallback matters during migration, because it lets you move secrets in stages, but it also means a secret left in the database can keep working silently after you believe it has moved. Decide early whether the fallback stays on in production, and plan a check that lists any connection still stored in the database. Confirm the exact option names, lookup order and fallback behavior for your Airflow version in the Airflow documentation on secrets backends before you write configuration.

What you need in place before the Vault backend is enabled

The Vault secrets backend only works once the Airflow platform, the metadata database and the Vault side are ready. Each card below names one prerequisite and the question to settle with your team before any configuration changes are made.

Airflow 3.3.2 runtime versions

Confirm that the Python interpreter on your scheduler, workers and API server falls within the range that Airflow 3.3.2 supports, which lists Python 3.10 to 3.14. Check that the same interpreter version is used across every component, since a mismatch between the scheduler and a worker image is a common source of failures that only appear after a secrets lookup starts.

Supported metadata database

Connections and variables move to Vault, but the metadata database still stores DAG runs, task state and other operational data. Confirm that you run PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release. MariaDB is not supported, and SQLite should not be used in production, so ask your team which database is already in place and who owns its backups.

Memory for scheduler and API server

The prerequisites recommend at least 4 GB of RAM. Confirm this for each Airflow component that will resolve secrets, since each process fetches connections and variables from Vault as it needs them. Ask whether the Vault client calls will be sized alongside existing parsing and scheduling load, and whether the hosts have headroom for growth.

Reachable Vault server with TLS

Airflow components must reach the Vault API over the network, so confirm the network path from every scheduler, worker and API server host to the Vault address. Use TLS for that connection and confirm which certificate authority signs the Vault endpoint, so each Airflow host trusts it without disabling verification. Ask the Vault team who owns the listener and its certificate renewal.

Vault auth method and Airflow policy

Decide how Airflow authenticates to Vault, such as AppRole, Kubernetes or another method your Vault team already runs, and create a policy that grants read access only to the mount and paths Airflow uses for connections and variables. Confirm which identity each environment uses, so staging and production do not share a token or role, and who rotates the credential.

HashiCorp provider package

The Vault backend ships in a separate HashiCorp provider package rather than in core Airflow. Confirm the package name, its version compatible with Airflow 3.3.2, and the configuration keys it expects by reading its own documentation, then pin it in the same constraints file as the rest of your Airflow environment so it upgrades with a planned change.

Airflow 2 end-of-life date

If any environment still runs Airflow 2, note that Airflow 2 reached end of life on April 22, 2026, after which it gets no fixes. Plan the Vault change on Airflow 3 where possible, because the secrets configuration you build now would otherwise need to be redone during the upgrade. Confirm the upgrade date with the platform owner before scheduling the Vault cutover.

Choosing KV version 1 or KV version 2 for Airflow secrets

KV version 1 stores one current value per path. Writing to a path replaces the secret that was there, and the previous value is gone unless you kept a copy elsewhere. For Airflow, each connection or variable path holds exactly one secret, and the backend reads whatever is stored at that path when a task runs.

  • Single value per path: Each write overwrites the stored secret, so Airflow only ever sees the current value.
  • Rotation without rollback: A bad password pushed to a path cannot be reverted from Vault. Recovery depends on the source system or your own backup process.
  • Deletion is final: Removing a path removes the secret for every DAG that reads it. Confirm how your team responds to a missing connection before the next scheduled run.
  • Limited per-secret history: Little change history is kept on the secret itself, so evidence of who changed a value comes mainly from Vault audit logs and your change records.
  • Flat mount layout: Paths sit directly under the mount, so the path Airflow builds closely matches what you see in the Vault interface.

Before relying on this version, check that your Vault audit logging captures writes and deletes, and test that a deleted path fails visibly in a DAG run.

Configuring the Vault secrets backend in order

This sequence moves Airflow connections and variables out of the metadata database and into Vault, with each component able to read its own secrets. Each step ends with a check so the team knows the layer works before anything is migrated.

  1. Create the KV mount and path layout

    The Vault administrator enables a KV secrets engine for Airflow and agrees on one path layout for connections and variables, such as separate prefixes for each. Confirm with your own Vault team whether the mount uses KV version 1 or KV version 2, because the Airflow lookup settings and the secret paths you write depend on that choice. The step closes when a test secret written by the administrator can be listed and read at the intended path, and the layout is written down for the team.

  2. Create a policy and auth method for Airflow

    The Vault administrator writes a policy that allows read access only to the Airflow connection and variable paths, and binds it to an auth method that each Airflow component can use, such as AppRole or Kubernetes auth where your platform supports it. Scheduler, API server, DAG processor and worker identities should be separate so access can be reviewed per component. The step closes when each identity logs in and reads a test path, and a read attempt against a path outside the policy is denied.

  3. Configure the secrets backend and Vault settings

    The platform team sets the Airflow secrets backend option to the Vault backend in the configuration used by every component, and provides the Vault address, mount point, path prefixes, KV version and the authentication settings. Keep credentials for the Vault login in the deployment's own secret store rather than in DAG code or plain configuration files. The step closes when Airflow starts cleanly on each component and the configuration matches the layout from the first step.

  4. Write and resolve one test connection and variable

    An engineer writes one test connection and one test variable in Vault at the agreed paths, using a value that is safe to expose, such as a dummy host. From a shell in the Airflow environment, a CLI lookup of the known connection ID and variable key should return the stored values. The step closes when both lookups return the expected results using the Airflow component's own identity, not an administrator token.

  5. Confirm resolution from the scheduler and a task

    Run a small test DAG that reads the test variable and uses the test connection in a task, and confirm the scheduler parses it without errors. The task log should show the value resolved through the backend, and the Vault audit log should record the read from the task's identity. This step proves that both the scheduler path and the worker path can reach Vault. It closes when the DAG run succeeds and the audit entries match the expected identities.

  6. Migrate remaining objects and check the lookup order

    Move the rest of the connections and variables into Vault, following the agreed path layout, and record which object moved to which path. Confirm that Airflow resolves each object from Vault and that any lookup order you rely on behaves as expected, especially where a name could exist in both places during the transition. Confirm the exact multiple path lookup behavior with the Airflow documentation for your version before relying on it. The step closes when every migrated object resolves in a test run.

  7. Remove migrated objects from the metadata database

    After the resolution checks pass for each object, delete its copy from the Airflow metadata database through the UI, CLI or a controlled change, and keep a dated record of what was removed. Take a backup of the metadata database before this step so the removal can be reversed if a missed dependency appears. The step closes when a DAG run that depends on the removed objects still succeeds and the database no longer holds the secret values.

How Airflow connections and variables map to Vault paths

This table compares how each Airflow object type is stored in Vault, including the path pattern, KV version behavior, the field that holds the value, and who maintains it. Read it as a starting layout to confirm against the Vault provider settings in your installed Airflow version, since mount names, prefixes and field handling depend on configuration.

Object or settingVault path pattern and KV behaviorValue field and owner
ConnectionsStored under the connections path prefix, with the connection ID as the final path segment. Under KV version 2, the mount needs a data segment that the backend handles for you, so confirm the mount and version settings.Connection fields such as login, password, host and extras are read from the secret. Owned by the platform team that provisions data source access.
VariablesStored under the variables path prefix, with the variable key as the final path segment. KV version 1 and version 2 differ in how versions and metadata are kept, so confirm which one your mount uses.The variable value is read from a single value field. Owned by the DAG team that depends on the variable, with the platform team approving new keys.
Connection ID namingThe connection ID should be unique within the prefix and stable across environments, so DAG code references the same ID everywhere.Owned by the team that defines the data source. Changing an ID breaks DAG references, so record it in your DAG review.
Environment prefixA separate mount or path prefix per environment (for example, development, test and production) keeps credentials for each stage apart. The exact separation pattern is your design choice.Owned by the platform team. Confirm that Airflow deployments in each environment point only to their own prefix.
Team prefixWhere several teams share one Airflow, a team segment in the path can limit which secrets a team's DAGs can reference, but Vault policies must enforce the limit.Owned by the team lead of each prefix, with Vault policy changes reviewed by security.
Multiple path lookupsThe backend can search more than one prefix for a lookup. Confirm the order of search, because a value in an earlier prefix will shadow a later one.Owned by the platform team. Document the search order in the runbook so that a shadowed value is not mistaken for a failure.
Provider-specific settingsMount point, KV version, path prefix keys and the field names read from each secret vary by provider version. Confirm them in the configuration for the installed provider package before you write any secret.Owned by the Airflow platform team, verified during each upgrade of the Airflow or provider packages.

How multiple path lookups and fallbacks change resolution

When the Vault secrets backend is enabled, Airflow asks Vault for a connection or variable under the configured mount and paths before it reads the metadata database. Whether a second path is searched, and in what order, depends on how the backend and its provider version are configured. If the backend is configured with more than one connections or variables path, a key can be found under more than one location. The first match is what the task receives, so a secret placed under a later path can be silently ignored. Confirm the exact lookup order, and whether a miss in Vault falls through to the metadata database, against the Vault provider release you have installed rather than against memory or older examples.

A key that exists in both Vault and the metadata database is the case that causes the most confusion during migration. During a phased move, a connection may be created in Vault while an older copy remains in the database. Which copy a DAG uses then depends on the lookup behavior you confirmed, so the two sources can disagree without any error. Plan a short inventory of every connection and variable ID present in both places, decide which source is authoritative for each, and remove the duplicate once the cutover is verified. Test the resolution with a non-production DAG that reads each key and records which value it returned.

Document the resolution order as an operational standard before go-live, not as an afterthought. The document should name each path in the order it is searched, state what happens on a miss, identify the Airflow version and Vault provider version it applies to, and list who approves changes to the configuration. Review it again on every Airflow or provider upgrade, because lookup behavior is part of what an upgrade can change. Shadowing risk rises with each additional path, so keep the number of searched paths as small as your layout allows and ask the implementer to show the resolution result for every secret class in your test evidence.

Hardening the Vault connection and the secrets it serves

Moving Airflow connections and variables into Vault shifts the risk to the Vault connection itself. The controls below cover who can read which paths, how Airflow authenticates, how traffic is protected, and what evidence Vault keeps, so the secrets store is not weaker than the metadata database it replaces.

Dedicated policy per environment

Give each Airflow environment its own Vault policy that grants read access only to the connection and variable paths that environment's DAGs need. A production scheduler should never be able to read staging credentials, and a shared policy across environments makes that boundary impossible to audit. Check policies by testing them with the actual Airflow identity, not by reading them alone.

Short-lived tokens from the auth method

Authenticate Airflow to Vault with the auth method your platform already trusts, such as Kubernetes or AppRole, and let Vault issue tokens with short time-to-live values that the client renews. Avoid long-lived static tokens in environment variables or configuration files. Confirm the renewal behavior of your chosen auth method and the Airflow Vault backend version before setting time-to-live values.

TLS with certificate validation

Require HTTPS between every Airflow component that calls Vault, including the scheduler, workers and API server, and validate the Vault certificate against your internal certificate authority. Disabling verification to get a connection working hides a misconfigured trust chain and exposes secrets to interception. Confirm the TLS options the Vault backend accepts in your Airflow version.

KV paths separated by environment and team

Lay out KV paths so each environment and owning team has its own prefix, for example one mount or path segment per environment, and the Airflow connections prefix and variables prefix sit under it. Shared paths make it hard to know who owns a credential and blur policy boundaries. Document the path layout alongside the mount type, since KV version 1 and version 2 use different path structures for reads and metadata.

Vault audit logging

Enable at least one Vault audit device so every authentication, secret read and policy change is recorded with the requesting identity. Send audit logs to a store your security team monitors, and alert on reads of production paths from unexpected identities. Airflow's own logs show task activity, not which secret was read, so the Vault audit trail is the primary evidence for this control.

Remove legacy secrets after cutover

After the backend serves all connections and variables, delete the copies stored in the Airflow metadata database and confirm no DAG or plugin still reads them directly. Keep a dated record of the cutover and of the deletion. Leftover rows in the metadata database and in backups keep credentials alive outside Vault, so include database backups and replicas in the cleanup plan.

Masking and log hygiene in Airflow

Confirm that Airflow masks secret values in task logs and the UI, and check that DAG code does not print connection details, pass secrets as command-line arguments or write them into XCom. Masking covers only values Airflow knows about, so test it with a deliberately logged value in staging before production. This check complements Vault controls because a secret leaked into a log is outside Vault's access controls.

How to test secret resolution before cutover

Before you move any production DAG onto the Vault secrets backend, run a test plan that proves each Airflow component can resolve secrets from Vault under the identity it will actually use. Start with a read check for every component identity that touches connections or variables, including the scheduler, the DAG processor, the API server and the workers. Each identity should read a known test connection and a known test variable from the expected KV path, using the same token or auth method your production configuration will hand to Airflow. A check that passes only from an operator's laptop or from a single component proves very little about the rest of the deployment.

Next, run a small test DAG that reads one connection and one variable through the normal hooks and operators, not through direct Vault API calls. This confirms that the lookup order, the connections_prefix and variables_prefix values and the KV version you chose all resolve as expected in task code. Add a negative test that points at a path which does not exist and confirm the failure is clear, logged and does not silently fall back to a stale value in the metadata database. Then run a rotation test: change a value in Vault, trigger a new run and confirm the task receives the new value. Record how long the change takes to appear, since cached values and token renewal can delay it. If Vault runs as a cluster, add a failover test that stops the active node during a scheduled run and confirms tasks still resolve secrets through the standby, with the failure behavior documented.

Save the outputs of every test (run IDs, log excerpts, Vault audit entries and the configuration version tested) as evidence for change management, so the cutover approval refers to recorded results rather than memory. Confirm the exact behavior on your executor model before you rely on these results. Worker processes on Celery, Kubernetes pods, and local processes can differ in how and when they fetch secrets, and the executor documentation at Apache Airflow executors is the natural starting point for checking where task processes run. Repeat the tests on each executor type you run in production.

Common Vault backend failures and how to diagnose them

This table lists the failures teams most often hit when Airflow reads connections and variables from Vault, with the likely cause, a first diagnostic check and a first fix for each. Work down the rows in order when a DAG fails to find a connection, and use the checks before changing policies or Airflow configuration.

FailureLikely cause and diagnostic checkFirst fix
Connection not foundThe lookup path does not match where the secret was written, or the KV version set in the backend does not match the mount. Check the connections path prefix and the KV engine version on the mount, then read the secret with the Vault CLI using the same path Airflow builds.Correct the connections path prefix or kv_engine_version so it matches the mount, and rewrite the secret at the path Airflow expects.
Permission denied on readThe token policy lacks the read capability on the data path, or lacks list on the metadata path for KV version 2. Test the exact token with a read and a list against the connection and variable paths.Add read and list capabilities for the connections and variables paths only, then retest with the same token.
Token expired during long-running workersThe token used by Airflow has a short time to live or hit its maximum use count, so tasks that run for hours lose access partway through. Compare the token lifetime with the longest expected task duration and check worker logs for authentication errors after a set time.Use a renewable token or an AppRole flow that re-authenticates, and confirm the renewal settings with your Vault administrator.
TLS verification failureThe Airflow worker or scheduler does not trust the internal certificate authority that signed the Vault listener certificate. Check the error text in logs and confirm the CA bundle path available to the Airflow process.Provide the internal CA bundle to Airflow's Vault configuration and to the host trust store, rather than disabling certificate verification.
Values still read from the metadata database after cutoverThe secrets backend is not enabled in the configuration that the scheduler, workers and API server actually load, or a connection with the same ID still exists in the database and takes precedence in some lookups. Confirm the active backend in the running configuration on each component.Set the backend in the configuration every component reads, restart all components, then delete the duplicate database connections after a test run succeeds.
Secrets appearing in logsConnection extras, task parameters or printed debug output include secret values, or a custom operator logs its configuration. Search recent task logs and rendered template views for known secret values after a test run.Remove secret values from printed output and parameters, use connection IDs in DAG code, and confirm masking covers the fields your providers use.

Questions teams ask before connecting Airflow to Vault

How much effort does a Vault secrets backend integration take?

The effort depends mostly on how many connections and variables move, how the Vault KV mount is laid out, and how many environments need their own paths. A small set of connections on one KV mount is a short, contained change. Effort rises when teams have many connections with inconsistent naming, mixed KV version 1 and version 2 mounts, or separate development, test and production Vault namespaces that each need matching paths. Executor model matters less for the backend itself but matters for where workers authenticate to Vault and how they receive tokens. Rotation needs add work if credentials must change without redeploying DAGs. Ask any partner to break the quote down by these drivers, and ask which of them you can settle before the build starts.

Can our own team connect Airflow to Vault without a partner?

Often yes, if you have someone who can administer Vault and someone who owns the Airflow deployment. The backend is configured in Airflow, and the Vault side needs a KV mount, policies that let Airflow read only the paths it needs, and an authentication method for the Airflow components. Teams usually struggle less with the configuration keys than with the path design, the policy scope and testing across environments. If your team has not run a Vault production cluster or has not changed Airflow configuration under a change process, a partner can take on the design and test work while your staff keep ownership. Either way, confirm who will maintain the policies after handover.

What does the work need from our teams?

The integration needs three groups and a few concrete things from them. Vault administrators own the KV mounts, policies, authentication method and audit settings. Airflow platform owners own the Airflow configuration, the components that read secrets, and the connection and variable inventory. A test environment that mirrors production paths is needed so that DAG runs can be verified before cutover. You also need a change window, because the lookup order and path structure take effect when the configuration changes, and rollback should be planned in advance. Ask your partner or internal lead to write the cutover steps and the rollback steps down before the window is booked.

Is the Vault secrets backend free because Airflow is Apache-2.0 licensed?

The Airflow project is released under the Apache License 2.0, and the project itself has no paid edition, so the Airflow side of the backend carries no license fee. HashiCorp Vault is a separate product with its own licensing and support terms, and those terms should be confirmed with HashiCorp or your reseller before you plan around them. Infrastructure is also a separate cost: Vault servers, storage, backups, load balancing and monitoring run on your platform, and their size depends on your availability target and request volume. Partner services are a third cost component and are priced from the written scope. Ask for each of these to be listed separately in any quote.

What changes for DAG authors after the cutover?

For most DAG authors the change is in how they refer to connections and variables, not in the logic of their pipelines. Connection IDs and variable keys usually stay the same, so DAG code that calls them keeps working once the backend is in place. The differences show up in where values are created and updated: they are written to Vault paths by administrators or automation rather than through the Airflow UI or CLI, and a value change takes effect at the next lookup rather than at a deployment. Authors should also learn how the lookup order works, so they know which path a value comes from when two paths define the same key. Document this in the team guide before cutover.

Which Vault KV version should we choose for Airflow?

Choose the KV version your Vault platform already standardizes on, unless you have a reason to change it, and confirm the mount type before writing any paths. KV version 2 adds versioning and soft deletion, which helps with recovery and auditing of secret changes, but its paths include a data segment, so the path configuration in Airflow must match that layout. KV version 1 has a simpler layout but no version history. Mixed mounts are possible, yet they complicate the path settings and the troubleshooting guide. Confirm the choice against the Airflow Vault backend options in the provider documentation for your Airflow version, and test lookups on each mount type before committing.

What should we check when Airflow cannot read a secret from Vault?

Start by separating three causes: the Airflow component cannot authenticate to Vault, the policy does not allow the path, or the path or key name does not match what Airflow builds. Check the Vault audit log for the denied request, which shows the path and identity involved. Then compare the configured mount, KV version and prefix settings against the actual secret location. Confirm that the lookup order checks the paths you expect, because a missing value in an earlier path can be the cause of a wrong or unexpected result. Testing each connection with a known path in a non-production environment is faster than debugging in production DAG runs.

How should we handle secret access and audit once Vault is in place?

Give each Airflow component its own Vault identity and a policy that reads only the paths it needs, so the scheduler, workers and API server do not share broad access. Keep write access to the secret paths with administrators or controlled automation, and record who changes what through Vault audit logging. Plan token lifetimes and renewal, because expired tokens are a common cause of failed runs. The Airflow security documentation and your own security team's standards should define these settings, and your auditors will ask to see them. DeployOpen can prepare the controls and evidence for a compliance assessment, but independent auditors issue any audit conclusion.

How DeployOpen scopes and delivers the Vault secrets backend

DeployOpen plans, builds and hands over the Airflow side of a Vault secrets backend inside infrastructure you control. The work starts with a written scope that fixes which connections, variables, mounts and paths move, and ends with documentation your operators can run without us.

How we scope

  • Inventory every Airflow connection and variable and classify which ones hold credentials that belong in Vault
  • Agree in writing the KV mount layout, whether KV version 1 or 2 applies, and the connection and variable path prefixes
  • Define the lookup order for multiple paths and the fallback behavior if a secret is missing in Vault

How we build

  • Configure the Vault secrets backend in a non-production Airflow 3 environment and confirm each connection resolves from its Vault path
  • Test rotation of a sample credential and confirm running tasks pick up the new value under your Vault policies
  • Verify that Vault access tokens, policies and AppRole or Kubernetes auth roles grant only the paths each Airflow component needs

How we hand over

  • Deliver a runbook covering mount layout, path naming, policy definitions and the steps to add a new connection
  • Walk your platform and data teams through lookup behavior, troubleshooting and the metadata database cleanup of old secrets
  • Document the verification steps and evidence your auditors can review for secrets access and change control

How we support

  • Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
  • Ongoing support for Airflow upgrades, Vault policy changes and secrets backend reviews is available as a retainer
  • Confirm with your Vault and Airflow administrators which versions and auth methods your environment will use before scoping

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com