Apache Airflow secrets backends for connections and variables

A secrets backend moves Airflow connection and variable credentials out of the metadata database into a store you control. This page explains the lookup order, backend choices, worker separation and naming rules.

Book a callTalk to us
On this page

Why Apache Airflow secrets backends matter for connections and variables

A secrets backend is an external store that Airflow queries for connections and variables at runtime, so passwords, tokens and keys do not have to sit in the metadata database. A team needs one as soon as DAGs authenticate to databases, cloud accounts or third-party APIs and the people or systems that can read the metastore should not automatically be able to read those credentials. Once more than a handful of pipelines depend on shared credentials, a backend also gives security and platform teams one place to rotate, revoke and audit them.

Airflow checks sources in a fixed order. It first asks the alternative secrets backend, then environment variables, then the metastore, and this order cannot be changed in configuration. The practical consequence is that a backend value shadows a same-named environment variable or metastore entry, which matters during migration because a stale copy in the database may still be used if the backend lookup misses. Workers can use a different backend through the [workers] section, which is covered in the sibling topic on worker isolation. Confirm the exact lookup behavior against the Airflow secrets backend documentation for the version you run.

This topic is one part of a wider hardening approach for self-hosted Airflow, and it assumes Airflow 3 as the target version. A secrets backend reduces exposure in the metastore, but it does not stop a DAG author from printing a secret or reading it through a task, so it works alongside masking, worker isolation and review controls. Before you choose a backend, confirm which Airflow 3 patch release you will deploy, which provider package supplies the backend you need, and whether your identity and network setup lets the scheduler, API server and workers each reach the store with the least privilege they need.

Choosing a secrets backend for your environment

Vault suits organizations that share secrets across Airflow, other platforms and applications, or that run in several clouds and data centers. The Airflow HashiCorp provider reads connections and variables from paths you define. Your team must already run a Vault cluster, or buy HCP Vault Dedicated, and must operate unsealing, storage, backups and policies for it.

  • Deployment model: Self-managed Vault and HCP Vault Dedicated differ in who runs the cluster, not in the Airflow configuration. HCP Vault Dedicated runs the same binary as self-managed Vault Enterprise.
  • Licensing: HashiCorp announced that future releases move from MPL-2.0 to the Business Source License 1.1. The Vault LICENSE names IBM as licensor for Vault 1.15.0 or later and restricts competing hosted or embedded offerings, so have counsel confirm your use fits.
  • Enterprise features: Replication, HSM support, namespaces and FIPS compliance need a Vault Enterprise license on the cluster. HashiCorp does not publish Enterprise prices, so request a written quote.
  • OpenBao option: OpenBao is a community fork under MPL-2.0 and a Sandbox project of OpenSSF. Its migration guide was tested only against Vault 1.14.1 community edition, so confirm each feature you need before choosing it.

Check the license terms and the feature list against your intended use before you standardize on a Vault distribution.

Secrets backend configuration options and when each fits

This table compares the configuration keys that choose where Airflow reads connections and variables, from the main [secrets] backend to a separate [workers] backend and the environment variable fallback. Read each row as a decision: what the setting controls, when it fits, and what it costs you to operate. The main takeaway is that the lookup order is fixed, so the backend you pick determines where every credential must live.

OptionWhat it controlsWhen it fitsTrade-off to plan for
[secrets] alternative backendSets the secrets backend class that Airflow queries first for connections and variablesCentral credential stores your organization already runs, such as Vault or a cloud secret managerEvery lookup depends on the external store, so its availability and access policy affect scheduling
Environment variablesAirflow checks environment variables after the alternative backend and before the metastoreInjecting credentials through Kubernetes Secrets or container orchestration without an external storeValues appear in process environments, so tighten who can read pod specs and worker environments
Metastore fallbackReads connections and variables stored in the Airflow database when no earlier source returns a valueEarly testing or small installs where the metadata database is already backed up and access controlledCredentials sit in the database, which is the exposure a secrets backend is meant to avoid
[workers] secrets_backendGives task workers a different backend from the scheduler and API serverWorkers need narrow read access to only the secrets their tasks use, separate from control plane accessTwo backends mean two access policies and two sets of credentials to rotate and audit
backend_kwargsPasses provider-specific settings such as connection IDs, prefixes or paths to the chosen backendTuning the backend to your naming scheme, regions, mount points or lookup prefixesMistyped keys can fail silently or fall through to later sources, so verify lookups after each change
Lookup orderFixed sequence: alternative backend, environment variables, then metastorePredicting which value wins when the same connection ID exists in more than one placeOrder is not configurable, so use unique naming conventions to avoid collisions
Provider backend packagesConnect Airflow to Vault, AWS, Azure, GCP and Kubernetes through their provider packagesMatching the backend to the credential store your security team already governsConfirm the provider version supports your Airflow 3 release and the store's current API

Vault, OpenBao and the licensing question

Choosing a Vault-compatible backend for Airflow is partly a licensing decision, so settle it before the first [secrets] setting is written. On August 10, 2023, HashiCorp announced that future Vault releases would move from MPL-2.0 to the Business Source License 1.1. IBM completed its acquisition of HashiCorp on February 27, 2025. The Vault LICENSE names IBM as licensor, covers Vault 1.15.0 or later, and restricts competitive hosted or embedded offerings. Each version converts to MPL-2.0 four years after publication. Your organization's exposure depends on how it uses Vault, so ask counsel to read the HashiCorp adopts Business Source License announcement and the Vault LICENSE file against your planned use, and confirm current terms with the vendor in writing.

Vault is offered in three forms that carry different obligations and costs. Community Edition is available as binaries or source. Vault Enterprise is licensed and adds features such as replication, HSM support, namespaces, performance standbys, seal wrapping, FIPS compliance and Sentinel policy as code, each of which needs a license applied to the cluster. HCP Vault Dedicated is a managed service that runs the same binary as self-managed Vault Enterprise. HashiCorp's pricing pages direct self-managed Enterprise buyers to sales, so ask for a written quote that names the edition, features, support terms and renewal conditions. Airflow's Vault integration works with any of these, but the features you depend on determine which edition you need.

OpenBao is a community fork of Vault under MPL-2.0, managed by the Linux Foundation's OpenSSF, and it is a Sandbox project. Its migration guide was tested only against Vault 1.14.1 Community Edition, not Vault Enterprise, so treat it as untested for Enterprise features and confirm compatibility for your specific auth methods, secrets engines and storage backend with the OpenBao maintainers before committing. Once the license decision is made, record which product, version and edition your Airflow secrets backend will use, and plan how you would change backends later, since moving credentials between Vault products is a migration project in its own right.

Setting up a secrets backend step by step

This sequence moves Airflow connections and variables out of the metadata database and into a secret store you control, with each step ending in a check that proves the change took effect. Work through it in order, because the identity and access model must exist before Airflow can read anything.

  1. Confirm the secret store and its high availability model

    Before touching Airflow, decide which store will hold credentials: HashiCorp Vault, OpenBao, a cloud secret manager or Kubernetes Secrets. Confirm the licensing and support terms of your choice with the vendor, because Vault's license changed and Enterprise features need a license applied to the cluster. Size the store for high availability, choose a storage backend it supports, and define who can administer it. The output is a written design and a store that answers health checks from the network segment where Airflow components run.

  2. Install the provider package in every Airflow component

    Install the provider package that matches your store, such as the Vault, Amazon, Google, Microsoft Azure or Kubernetes provider, into the scheduler, DAG processor, API server, triggerer and worker images. Pin the version with your Airflow constraints file so all components load the same backend code. Check each image by running the Airflow CLI inside it and confirming the backend class imports without error. Components that lack the package will fail silently or fall back to other lookup sources, so check every image, not only one.

  3. Create a dedicated identity for Airflow

    Create a separate authentication identity that only reads the Airflow paths. For Vault or OpenBao, an AppRole for servers with a narrow read policy is a common pattern, and Kubernetes auth fits pods that run in a cluster. For cloud stores, use workload identity bound to the component's service account rather than long-lived access keys. Keep workers and the API server on separate identities where your design allows. The check is that the identity can read one test path and is denied a path outside its prefix.

  4. Set backend_kwargs and the path prefixes

    Configure the backend in airflow.cfg or environment variables under the secrets section, setting backend to the provider class and backend_kwargs to the mount point or project, the connections and variables path prefixes, and the authentication settings your identity needs. Store secret values under the prefixes, for example one path for connections and another for variables, so the backend resolves names predictably. Confirm the settings with the Airflow CLI's configuration output on each component. A correct setup resolves a test connection from the store and reports the backend in use.

  5. Migrate existing credentials and remove the metastore copies

    Inventory connections and variables already in the metadata database, then write each value to the store under the agreed naming convention. Use names that include the team or environment and avoid collisions between a connection and a variable with the same identifier. Update DAG code to reference the logical name only. Delete the database rows after each DAG has run successfully against the store. Keep a record of which credentials were rotated, because copied passwords remain valid until changed at the source.

  6. Verify resolution with a test DAG

    Deploy a small DAG that reads one connection and one variable through the normal Airflow interfaces, then runs a trivial task on a worker. Run it on each executor type your deployment uses, because workers may use a separate backend through the workers section of the configuration. Confirm the task succeeds and that the log shows no secret values. The check is a green run plus logs that show only masked placeholders where a credential would appear.

  7. Confirm the metastore holds no plaintext values

    Query the metadata database directly for the connection and variable tables and confirm that no password, token or key values remain. Compare the row count with your migration inventory, and check backups and replicas too, since a backup taken before cutover still contains old credentials. Record the result and the date in the change record. Repeat this check after each Airflow upgrade or database restore, because a restore can bring back rows that were deleted from the live database.

Naming conventions that prevent lookup collisions

Airflow builds the backend lookup key from the connection or variable name, so the naming scheme you choose becomes the access boundary. When a task asks for a connection, Airflow passes the connection ID to the backend, which combines it with the configured prefix (for example connections_prefix and variables_prefix in the [secrets] section, with the exact option names confirmed in the Airflow secrets backend documentation) to form a path or secret name. A connection ID such as warehouse_prod therefore maps to a single object in Vault, AWS, Azure, GCP or Kubernetes, and anyone who can read that object can use the credential. Before go-live, list every backend and provider you plan to use and confirm how each one maps prefixes and slashes, since the separator rules differ between providers.

Collisions happen because the lookup order is fixed. Airflow checks the alternative secrets backend first, then environment variables, then the metastore, so an environment variable such as AIRFLOW_CONN_WAREHOUSE_PROD can silently override a backend secret with the same connection ID. The reverse is also true: a stale value left in the metastore is never reached while the backend returns a match. Give each environment its own prefix, for example one path or project per environment and per team, so that a dev workspace cannot resolve to a production secret even if IDs match. Keep the environment variable layer for local development only, and treat any AIRFLOW_CONN_ or AIRFLOW_VAR_ variable in a production container as a finding to review.

Write the convention down and enforce it at review time. The document should state the allowed character set, the team or domain segment, the environment segment, and who may request new IDs. Add a review step to the pull request or change ticket that adds a connection ID, checking that it follows the pattern, does not duplicate an existing ID in any layer, and that the backend secret exists in the correct prefix before the DAG merges. Ask your platform team to confirm whether a read-only audit query can list secret names per prefix, so the review can run against the live backend rather than memory. The same review should cover workers: if [workers] secrets_backend points to a different backend, confirm that the worker prefix matches the scheduler's naming exactly.

Security and operational risks of secrets backends

A secrets backend removes credentials from the metadata database, but it creates a new store that attackers, outages and sloppy identities can all affect. The cards below cover the risks we see most often and the control that addresses each one.

The backend becomes a high-value target

Once connections and variables live in Vault, AWS, Azure, GCP or Kubernetes, that store holds every credential Airflow uses. Compromise of the store exposes all of them at once, so it needs the same protection as a production identity system. Check who can administer the backend, whether audit logging is on, and whether the store sits in a network zone that DAG authors cannot reach directly. Short-lived tokens for Airflow components reduce what a stolen token can do.

Airflow identities over-scoped in the store

A single token that can read every path in the store lets any DAG reach any credential. Give each Airflow component its own identity and policy, for example scheduler, API server and worker, and restrict each to the paths its tasks need. Use prefixes per team or environment so a policy can be written as a narrow path rule. Review these policies whenever a new provider or connection is added.

Secrets still leaking into logs or XCom

A secrets backend only protects values that Airflow looks up. If a task prints a retrieved password, passes it through XCom, or writes it to an environment variable that a subprocess echoes, the value lands in logs or the metadata database anyway. Test representative DAGs for these paths before go-live. Treat XCom as a data channel that must never carry credentials, and confirm log masking covers the names your teams use.

Store outages blocking task starts

When a task needs a connection, Airflow asks the backend at the moment it runs. If Vault or a cloud secret manager is unreachable or throttled, tasks fail or queue even though the workflow itself is healthy. Ask the platform team for the store's availability model, its request limits and its failover path before the first production schedule. Decide in advance whether a failing lookup should retry, alert or fail fast.

Caching and rotation behavior

Credential rotation only helps if workers pick up the new value. Lookups can be cached by the provider or by the process, so a rotated password may keep working for old tasks or fail for new ones until the cache expires. Document the cache lifetime for each backend, test a rotation in staging, and confirm which long-running processes need a restart. Rotate the credential in the store first, then in the target system, in that order.

Audit trail gaps

A store audit log shows which identity read a secret, but not which DAG run needed it, and Airflow's own records show the connection use without the store's view. Correlate the two by identity, run ID and time window. Confirm that store audit logs are retained longer than your Airflow metadata cleanup window and that they are shipped to the SIEM your security team already reviews.

Questions teams ask before adding a secrets backend

What drives the cost and effort of adding a secrets backend to Airflow?

The cost comes from four components: the secrets store itself, the identity design around it, the migration of existing connections and variables, and testing. The store may be a licence you buy, a managed cloud service billed by usage, or an open-source product you host yourself, and each carries different recurring costs. Identity design covers how Airflow components authenticate to the store, which is usually the largest unknown. Migration effort rises with the number of connections and variables and how many environments share them. Ask any partner for a quote that breaks out store licensing or hosting, identity work, migration and testing as separate line items, and that states which parts recur each year.

Can an internal team add an Airflow secrets backend without outside help?

Yes, a team with experience in both Airflow and the chosen store can usually do it alone, and the work is well bounded. The hard parts are rarely the Airflow configuration itself. They are granting least-privilege access to the store, deciding how workers authenticate, moving secrets without downtime, and agreeing on naming. Teams without someone who has run the store in production often benefit from a fixed-scope engagement that covers design, a staged rollout and handover. Whichever path you choose, confirm that someone on your staff will own the store after go-live.

How long does it take to move Airflow connections into a secrets backend?

The duration depends on your inventory and environments, not on a fixed figure. Start by counting connections and variables in each Airflow environment, noting which ones are shared, which are unused and which contain credentials that must be rotated during the move. Then consider how many environments (development, staging, production) need the same secrets path, and how many teams must confirm their DAGs still resolve correctly. Ask for a timeline that lists the inventory step, the pilot environment, the cutover window and the cleanup of stored credentials in the metadata database as separate phases.

What does the work need from our team before it starts?

The work needs three things from your side. First, access to the secrets store with enough rights to create paths and policies for Airflow, plus an administrator who can approve them. Second, network paths from the scheduler, API server, DAG processor and workers to the store, including any firewall rules and DNS entries inside your network. Third, a named owner for each group of secrets, someone who can confirm whether a connection is still in use, which value is current and when it must rotate. Without owners, migration stalls at the point where nobody knows which credentials to keep.

Does Fernet encryption still matter once a secrets backend is in use?

Yes, it still matters during the transition and for any secret that remains in the metadata database. Airflow encrypts connection and variable values stored in its database with the Fernet key, so keeping that key protected is necessary until every credential has moved out. A backend removes the need to store new credentials in the database, but it does not clean up old rows automatically. Plan to retire stored values after cutover, then decide how the Fernet key is rotated or retained. Confirm this sequence with your security team before the cutover date is fixed.

Does the Vault licence change affect whether we should choose Vault?

It can, and the answer depends on how you intend to use the product. HashiCorp announced on August 10, 2023 that future releases would move from MPL-2.0 to the Business Source License 1.1, and the Vault LICENSE names IBM as licensor. The licence bars competitive hosted or embedded offerings, and each version converts to MPL-2.0 four years after publication. Internal use by a company running Vault for its own applications is a different question from building a competing service, so have your legal team read the licence against your plans. Enterprise features such as replication, namespaces and FIPS compliance need a licence applied to the cluster, and HashiCorp does not publish Enterprise prices. If you want an MPL-2.0 option, OpenBao is a community fork, though its migration guide was tested only against Vault 1.14.1 Community Edition.

Which secrets store should we choose for Airflow connections?

Choose the store that your organization already operates and can staff, since an unmaintained store becomes a single point of failure for every pipeline. Airflow ships provider backends for AWS Secrets Manager and SSM Parameter Store, GCP Secret Manager, HashiCorp Vault, Azure Key Vault and Kubernetes Secrets, among others. If you run Kubernetes, the Kubernetes Secrets backend may fit small estates, but it does not add the audit and rotation features of a dedicated store. Compare each option on identity integration, audit logging, high availability, and whether workers can use a separate backend through the [workers] secrets_backend setting.

How do we avoid naming collisions between secrets and Airflow variables?

Adopt one naming convention before the first secret moves, because Airflow looks up the alternative backend first and then environment variables and the metastore. A common pattern, given as an illustrative example, is to include the environment, team and connection purpose in the path, such as airflow/prod/finance/warehouse, and to keep connection IDs and variable keys unique across teams. Document which prefix each backend uses, confirm the lookup order in your version of Airflow, and test that a duplicate name in the environment variables does not silently override the store value.

How DeployOpen scopes and delivers secrets backend work

DeployOpen scopes secrets backend work in writing before any build starts. The first deliverable is an inventory of every connection and variable across DAGs, plugins and the metadata database, recording who owns each one, which systems it reaches, and whether it is read by the scheduler, the DAG processor, the API server or workers. That inventory decides the backend choice, the identity each Airflow component uses to authenticate to the backend, whether [workers] secrets_backend needs a separate setting, and the naming standard that prevents a connection and a variable from colliding under the same lookup key. Backend and identity choices are agreed with your security and platform owners, because a Vault, AWS, Azure, GCP or Kubernetes backend each brings its own access model and operating responsibilities.

The written scope also sets out migration and verification. Migration moves credentials out of the metastore in ordered waves, with a rollback path for each wave and a check that each DAG resolves the expected secret from the new backend before the stored copy is removed. Verification covers lookup order behavior, denied access for workers that should not read certain paths, and confirmation that secret values stay masked in logs and the UI. Everything runs inside your own cloud accounts, data centers or air-gapped networks, and DeployOpen does not issue audit reports or attestations, so evidence is delivered for your auditors to review. Ongoing support, backend upgrades and periodic access reviews are available as a retainer after handover. Ask for the scope, deliverables and schedule in writing before committing.

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com