Why Airflow secrets in the Helm chart need a plan before the first install
Plan where each sensitive Airflow value will live before the first helm install, because the official chart does not keep credentials out of values.yaml by default. The short answer is that the metadata database connection, the Fernet key, the API secret key and, if you use the built-in Redis with CeleryExecutor, the Redis password should each sit in a Kubernetes Secret that you create and manage, with the chart pointed at it. The Helm chart documentation advises against putting passwords in values.yaml, so a values file that will be committed to Git, shared with a vendor or copied between environments is the wrong place for any of them. Confirm this with your chart version before you write the first values file, since the exact keys differ between releases.
Each value has a different blast radius. The metadata database connection string carries the credentials for the database that holds every DAG run, task instance and connection your pipelines use. The Fernet key encrypts values stored in the metadata database, such as connection passwords, so losing it or rotating it carelessly can make stored credentials unreadable. The API secret key signs the requests that the API server and workers exchange, and the production guide asks you to set it statically through a Kubernetes Secret and to use a different value per instance. Also plan for clock discipline: the production guide warns that unsynchronized clocks can cause forbidden errors when task logs are fetched, so NTP or your node time service must be verified on every node before you debug log access.
Two chart defaults decide how much work the plan takes. The chart creates secret environment variables for several of these values from its own built-in settings, and to use _CMD or _SECRET variants you must first disable the matching built-in variable through enableBuiltInSecretEnvVars. Once you do, the chart no longer manages that variable, so your team owns its creation, rotation and naming. Kubernetes stores Secrets unencrypted in etcd by default, so the plan should also cover encryption at rest and least-privilege RBAC on Secret reads. DeployOpen plans and implements this configuration inside your own cluster, with the Secrets, RBAC and rotation steps agreed in writing before the install.
Which secret mechanism fits each Airflow value
This table compares the four ways a value can reach an Airflow Helm release: the chart's built-in secret environment variables, the _CMD variant, the _SECRET variant and inline values in values.yaml. Read each row for when the mechanism fits, what it depends on and which built-in variable must be disabled before it works. The main takeaway is that sensitive values should not sit in values.yaml, and the _CMD and _SECRET variants only take effect after the matching built-in variable is turned off.
| Mechanism | When it fits | What it depends on | What must be disabled |
|---|---|---|---|
| Built-in secret environment variables | Default for passwords, the Fernet key and the API secret key when the chart manages them from Kubernetes Secrets | Kubernetes Secrets created in the release namespace with the expected keys, plus RBAC that limits who can read them | Nothing. This is the default behavior |
| _CMD variant | A value must be fetched at container start by a command, such as a vault client or a script that reads a mounted file | A command available inside the Airflow image, credentials for the store it calls, and a mounted or reachable source | The matching built-in variable, for example enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB set to false |
| _SECRET variant | A value should be read from a Kubernetes Secret named by an environment variable, with the secret mounted or referenced directly | The Secret existing before pods start, the correct key name and access granted to the service account running Airflow | The matching built-in variable, for example enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB set to false |
| Inline values in values.yaml | Non-sensitive settings only, such as log levels, replica counts and feature flags | Values committed to a repository, so access to the repository effectively grants access to the value | Not applicable. The chart advises against placing passwords here |
| Static Redis password (CeleryExecutor) | The built-in Redis is used with CeleryExecutor and needs a fixed password across restarts | A Secret holding the password that the chart and the Redis deployment both reference | Not applicable. Set it through a Secret, not values.yaml |
How to create and reference the Airflow Secrets in the chart
This sequence creates the metadata connection, Fernet key and API secret key as Kubernetes Secrets outside the Helm release, then points the chart at them. The result is a deployment where sensitive values never sit in values.yaml and the pods read their keys from Secrets you control.
Inventory secrets and confirm the chart version
The platform engineer lists every sensitive value the release needs: the metadata database connection, the Fernet key, the API secret key and, only if the built-in Redis is used with CeleryExecutor, a Redis password. Confirm the chart version and the secret name settings for that version in the chart documentation, because key names change between releases. Done when you have a written list with one owner and one target namespace for each value.
Create the metadata connection Secret
Create a Kubernetes Secret in the Airflow namespace that holds the connection string for the PostgreSQL or MySQL metadata database. Create it with kubectl or your secrets pipeline, not through a values file, and keep the password out of shell history. Done when the Secret exists, its keys match the names you recorded in step one, and a test connection from a pod in the namespace succeeds with the same credentials.
Create the Fernet key Secret
Generate the Fernet key with a cryptographically secure tool, store it in its own Secret, and record who holds the backup copy. Every Airflow component that reads encrypted connection and variable data must use the same key, so the Secret must be the one all Airflow pods mount. Done when the Secret is present, the key length and format pass a check in your own tooling, and a documented backup exists in your key management process.
Create the API secret key Secret per instance
Create a separate Secret for the static API secret key. The chart guidance is to use a static value set through a Kubernetes Secret, and to use a different value for each Airflow instance, so staging and production never share it. Confirm that clocks on the nodes are synchronized, because the chart guidance warns that unsynchronized clocks cause forbidden errors when task logs are fetched. Done when each environment has its own Secret and node time sync is confirmed.
Reference the Secrets in values without literal values
In values.yaml, set only the Secret names and keys for the metadata connection, Fernet key and API secret key. Leave the literal fields empty so the file can live in Git without credentials. If you run the built-in Redis with CeleryExecutor, set the static Redis password through its own Secret, and skip this if you use an external broker or a different executor. Done when a rendered template with helm template shows Secret references and no plaintext values.
Disable built-in variables for _CMD or _SECRET variants
If you supply connections through _CMD or _SECRET configuration variants, the matching built-in secret environment variable must be turned off, or the chart will still inject its own value. For example, the production guide shows setting enableBuiltInSecretEnvVars.AIRFLOW_CONN_AIRFLOW_DB to false when the metadata connection comes from a _CMD or _SECRET source. Check each variant you use against the chart documentation. Done when a rendered manifest contains only the variant you intended for each setting.
Deploy and confirm pods read the keys
Run the Helm upgrade or install in the target namespace and wait for the API server, scheduler, DAG processor and triggerer pods to become ready. Check the pod logs for database connection errors and confirm a test DAG can read an existing encrypted connection. Confirm that task log retrieval works, which tests the API secret key and clock setup. Done when all components are ready and the test run completes without decryption or forbidden errors.
Verify no sensitive value appears in the release
Run helm get values and helm get manifest for the release, then search their output for the database password, the Fernet key and the API secret key. Review the release history with helm history, since each revision is stored in the cluster. Confirm that Secrets are encrypted at rest and that RBAC limits who can list or watch them, because Kubernetes stores Secrets unencrypted in etcd by default. Done when the search returns no sensitive values and the encryption and RBAC checks are recorded.
How the approach changes by cluster and delivery model
When an operator runs Helm directly, the person or pipeline that runs the install also decides where secret values come from. The usual pattern is to create the Kubernetes Secrets first, with the metadata connection, the Fernet key and the API secret key, and then point the chart at them by name so that no sensitive value appears in values.yaml. This keeps the values file safe to commit and review, but it also means every rotation is a manual step that someone must record. The operator remains the only owner of the Secret objects, so the change history lives in whatever tool or runbook the team uses.
- Create Secrets before install: Build the Secrets in the target namespace before the first
helm install, so the release never starts without its credentials. - Reference, do not inline: Keep passwords and keys out of
values.yamland reference the Secret names instead. - Rotate with a runbook: Each rotation of the Fernet key or API secret key needs a written sequence and a dated record, because no tool will remember the previous value.
- Check the namespace: Confirm that the release and its Secrets share a namespace and that RBAC limits who can read them.
- Verify after changes: Run a
helm upgradein a test namespace first and confirm that task logs still load.
Confirm with your platform team who holds Secret write access and where the runbook is stored before the first production install.
In a GitOps model the controller applies the chart from Git, so the values files and any Secret manifests are reviewed like code. The hard rule is the same as in manual installs: sensitive values must not sit in the Git repository in clear text. Teams usually store only Secret names in values, and deliver the Secret objects through a separate, encrypted or externally sourced path. Hook behavior, drift detection and upgrade timing all depend on the controller, so confirm each of these with the tool's own documentation, because this topic does not assume specific controller behavior.
- Keep values free of secrets: Commit only references to Secret names, never the Fernet key, API secret key or database password.
- Confirm hook handling: Check how your controller treats Helm hooks for migrations and user creation before relying on them for secret-dependent steps.
- Expect drift reports: A Secret changed outside Git may be flagged or reverted, so decide in advance which object is the source of truth.
- Separate rotation from release: Rotate secret values through the secret delivery path, not by editing values and syncing a new chart version at the same time.
- Pin the chart version: Promote chart and image versions through environments so a secret change and a chart change are never mixed in one sync.
Check that your controller's sync and drift settings will not overwrite a Secret that an operator rotated on purpose.
Where the organization already runs a central secret store, the Secrets Store CSI Driver can mount values from that store into pods. In this model the store owns the secret values, and the Kubernetes Secret used by the chart becomes a derived object. Kubernetes stores Secrets unencrypted in etcd by default, so the external store is often chosen partly to reduce how many copies of a value exist inside the cluster. The trade-off is an extra dependency: if the store or the driver fails, Airflow components that need those values may not start, so availability planning must cover the store too.
- Name the owner: Record which team owns the value in the external store and which team owns the cluster object derived from it.
- Limit read access: Apply least-privilege RBAC so that only the components that need a Secret can read it, and restrict list and watch access.
- Plan for store outages: Document what happens to scheduler, API server and workers when the store is unreachable.
- Encrypt etcd anyway: Configure encryption at rest for any Secret objects that remain in the cluster.
- Test rotation end to end: Confirm that a rotated value reaches running pods and that the chart picks up the new value on the next restart.
Confirm the exact provider and mount behavior with your store vendor and the Secrets Store CSI Driver documentation before designing the rotation procedure.
Security controls that keep Airflow secrets from leaking
Secrets in the Airflow Helm chart are only as protected as the Kubernetes cluster that stores them. The controls below cover where secret values are kept, who can read them, which values are shared across instances, and how the pods that use them are configured. Each one is a check to confirm in your own environment before the first production install.
Encryption at rest for Secrets
Kubernetes stores Secret objects unencrypted in etcd by default, so anyone who can read the etcd data directory or a backup can recover the Fernet key, the API secret key and the metadata database password. Confirm with your platform team that encryption at rest is enabled for the Secrets resource in the API server configuration, and that etcd backups are encrypted and access-controlled. Ask the team to show the setting, not describe it.
Least-privilege RBAC on Secrets
Any identity allowed to list or watch Secrets in the Airflow namespace can read every credential in it, including values used by other workloads. Bind the Airflow service accounts only to the Secrets they reference by name, and keep list and watch verbs away from human groups and general-purpose CI roles. Review RoleBindings in that namespace after each chart upgrade, since new service accounts may appear.
No passwords in values.yaml
The chart documentation advises against putting sensitive values such as passwords in values.yaml. Those values are stored in the Helm release history, appear in rendered manifests and often end up in Git. Reference existing Kubernetes Secrets instead, and check that helm get values and helm history output in your environment contain no credentials before you share a release with another team.
Distinct API secret key per instance
The API secret key signs values that the API server and workers exchange, including log-serving requests. Copying one key from a test cluster into production, or reusing it across environments, lets a leaked key from one instance affect another. Generate a separate value for each deployment, store it in a Kubernetes Secret, and record in your runbook which key belongs to which environment so rotation does not break log access.
Clock synchronization across nodes
Task log fetching is validated against time, so clock skew between the API server, scheduler and workers can produce forbidden errors when logs are requested. The failure looks like a permissions problem but is a time problem. Confirm that every node in the cluster runs a time synchronization service with a trusted source, and monitor offset on the Airflow nodes as part of your normal alerting.
Pod Security Standards for the namespace
Airflow pods run user-supplied DAG code, so a compromised task that reaches the service account token or mounted Secrets can move laterally. Apply a Pod Security Standards level to the Airflow namespace, starting with restricted where the chart version allows it, and set automountServiceAccountToken only where a pod needs the API. Test the setting in a non-production namespace first, because some chart components need elevated settings that the policy may block.
How to rotate the Fernet key and API secret key safely
Rotating the Fernet key is the riskiest step because it encrypts values already stored in the metadata database, including connection passwords and variables. A new key alone cannot read those rows, so the old key must stay available until every stored value has been re-encrypted and checked. Treat a fresh backup of the metadata database as the rollback point before anything changes. Confirm the exact procedure for your chart version in the Helm chart documentation and the Airflow release notes, because how old and new keys are listed and how re-encryption is run depends on the release you run.
The API secret key follows different rules. It must match in every component that uses it, and it protects task log fetches between workers and the API server, so a partial change shows up as access errors that are hard to trace back to the key. Update the Kubernetes Secret once, confirm every component reads the same value, and keep both keys out of values.yaml during the change. The Helm chart production guide also warns that unsynchronized clocks can cause forbidden errors when logs are fetched, so check time synchronization on all nodes before restarting anything. Plan the restart order so the API server, scheduler, workers and triggerer move to the new state in one controlled window, then watch for log fetch failures afterward.
Rehearse the whole sequence in a non-production namespace that runs the same chart and Airflow versions, loaded with a recent copy of the metadata database. Record the current Helm revision and Secret values in your password manager, then run the steps in order: back up the metadata database, add the new Fernet key while keeping the old one available, re-encrypt stored values, confirm that a sample connection and variable still decrypt, roll the API secret key, and restart components. Remove the old Fernet key only after the production checks pass. If any step cannot be verified, stop and restore from the backup before going further.
Questions teams ask before changing Airflow secrets
Can we change Airflow secrets in the Helm chart without a partner?
Yes, if your team already runs Helm and Kubernetes in production and can work through the chart's production guide. The changes themselves are small: create Kubernetes Secrets, point the chart at them, and disable the built-in environment variables that would otherwise read values from values.yaml. The work gets harder when several Airflow components share one secret, when secrets must reach pods through a external store, or when auditors need evidence of each change. Decide early which parts your team owns and which need an outside review, and confirm the exact chart version you run against the Helm chart production guide before you start.
What drives the effort of moving secrets out of values.yaml?
Effort rises with the number of components and clusters that consume each secret, and with how your delivery pipeline works. The main drivers are:
- How many Airflow components read the metadata connection, the Fernet key and the API secret key, and whether the Helm release is managed by hand or by GitOps.
- Whether you use Kubernetes Secrets alone or an external store synced through the Secrets Store CSI Driver, which adds an integration and access policy to build and test.
- Whether you need compliance evidence for each change, such as approvals, access records and rotation logs.
Each driver is a one-time build cost, while secret rotation and access reviews recur. Ask any partner to split the quote along these lines.
How long does a secrets change take, and is it fixed-scope?
Duration depends on the scope you agree, not on a standard timeline, and DeployOpen does not publish fixed durations for this work. A change limited to one release, a Kubernetes Secret for the metadata connection and the Fernet key, is much shorter than a rework covering several clusters, GitOps and an external secret store. Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ask for the list of components, environments and verification steps in the written scope, since those items set the timeline.
What must our team provide before a secrets project starts?
Plan to provide access and decisions, not only time. Typical inputs include:
- Cluster access with rights to read and write Secrets in the Airflow namespace, plus Helm release details.
- A namespace design that states which workloads can read which secrets.
- The secret store you will use, if it is not plain Kubernetes Secrets, with its access model.
- Agreed change windows for rolling restarts of the API server, scheduler, workers and triggerer.
- Named owners who can approve values and rotations.
Confirm these with your platform and security teams first, because gaps here usually delay the work more than the technical steps do.
How do we rotate the API secret key without downtime?
Rotate by staging the new value alongside the old one, rolling the components, and then retiring the old value, with clocks kept in sync. The chart production guide asks for a static API secret key held in a Kubernetes Secret, unique to each instance, and warns that unsynchronized clocks can cause forbidden errors when task logs are fetched. Test the rotation in a non-production cluster first, check task log retrieval after each rolling restart, and confirm the exact procedure for your chart version in the Helm chart production guide. Plan a short maintenance window for the first attempt.
Can we recover the Fernet key if we lose it?
Do not plan on recovering it. The Fernet key encrypts values stored in the Airflow metadata database, such as connection passwords, so a lost key can leave those stored values unreadable. Keep a secure, access-controlled copy of the current key in your secret store or backup system before any change, and record which key version encrypted which data. Confirm with your Airflow version and the chart documentation how key rotation handles existing encrypted values before you change it, and test restoring a backup in a separate environment.
Why is Kubernetes Secret storage not enough on its own?
Kubernetes stores Secrets unencrypted in etcd by default, so they need controls beyond creating the object. The Kubernetes secrets good practices page recommends encryption at rest, least-privilege RBAC that restricts list and watch access, and, where needed, external stores through the Secrets Store CSI Driver. Review those controls for your cluster before you move production passwords and keys into Secrets, and document who can read them. Your security team should confirm how your cluster is configured, because the defaults in your distribution may differ.
Limits to confirm in your environment
The secrets pattern in the Helm chart depends on four things: the chart version, the Airflow and Kubernetes versions it runs, the security posture of your cluster, and which Airflow components need secret access. Confirm each one against your own environment before you write the first Secret manifest.
Chart, Airflow and Kubernetes versions
Confirm that the chart version you deploy supports Airflow 3.3.2 and that your cluster falls in the supported Kubernetes range of 1.30 to 1.35. Variable names such as enableBuiltInSecretEnvVars and the secret key names can change between chart releases, so pin the chart version and read its release notes before you copy any values from a guide. Record the exact chart, image and Kubernetes versions together so that a later upgrade does not silently change how secrets are read.
CNCF conformance of your distribution
Ask your Kubernetes provider or distribution vendor whether the cluster passes CNCF Certified Kubernetes conformance. Conformance shows that the required APIs behave as expected, including the Secret and RBAC APIs that this pattern relies on. A distribution that is not conformant may still run Airflow, but you should test Secret handling and the chart's hook jobs on it before relying on the design.
Encryption at rest for etcd
Kubernetes stores Secrets unencrypted in etcd by default. Confirm with your platform team whether encryption at rest is enabled for Secret resources, and who can read etcd snapshots, since those backups contain the same data. Your managed Kubernetes service may handle this differently from a self-managed cluster, so ask for the setting in writing rather than assuming it. Also confirm that etcd backups are encrypted and access-controlled.
External secret store support
Decide whether the Secrets are created by your pipeline, by a secrets operator, or through the Secrets Store CSI Driver pulling from an external vault. Check that your chart version reads the _CMD and _SECRET variants you plan to use, and that the external store's sync mechanism can populate the exact Secret names the chart expects. Test one rotation end to end in a non-production namespace before you rely on it.
Airflow 3 component split
In Airflow 3 the API server and the DAG processor run as separate components, and each one may need access to the metadata connection, the Fernet key or the API secret key. Map which pods mount which Secrets, because a broader mount widens the blast radius of a leak. Confirm that the workers and triggerer receive only what they need, and that task code cannot read database credentials directly under the Airflow 3 model.
Clock sync across nodes
The production guide warns that unsynchronized clocks can cause forbidden errors when task logs are fetched, and that problem often appears only after the API secret key is set per instance. Confirm that every node in the cluster runs a time synchronization service and that drift is monitored. Include this check in your rotation runbook so that a clock fault is not mistaken for a secret mismatch.
Where the official guidance on Airflow chart secrets is documented
The first place to check is the Helm chart production guide, which covers how the Airflow chart expects required secrets to be supplied. By default the chart keeps required secrets in Kubernetes Secrets, and the guide explains the matching built-in variables you must disable before you can use the _CMD or _SECRET variants. Read it alongside your planned release values so you can confirm, for each secret, whether the chart creates it, reads an existing Secret you manage, or expects an external source. Use the Helm chart production guide as the reference for the exact configuration keys, and confirm them against the chart version you intend to pin, because variable names and defaults can change between chart releases.
The chart documentation also advises against placing sensitive values such as passwords directly in values.yaml. That warning matters for the metadata connection, the Fernet key, the API secret key and any Redis password if you run the built-in Redis with CeleryExecutor. Treat the values file as configuration that will be stored in Git, shared with reviewers and copied into CI logs, and plan for the secret material to be created outside the chart and referenced by name. Ask your platform team which secret store will hold these values and who is allowed to read them before the first install.
Kubernetes stores Secrets unencrypted in etcd by default, so the chart's approach is only as strong as the cluster around it. The Kubernetes good practices for Secrets page covers the controls to confirm: encryption at rest for etcd, least-privilege RBAC that limits who can list or watch Secrets, and the option to pull values from an external store through the Secrets Store CSI Driver. Verify these controls in your own cluster rather than assuming they are on, and record the result as part of your deployment evidence.
How DeployOpen scopes and supports Airflow secret management
DeployOpen plans and implements the secret and configuration layout for your Airflow Helm release inside your own Kubernetes clusters. The work covers which values move into Kubernetes Secrets, how the Fernet key and API secret key are handled per instance, and how changes are rotated without breaking running components.
How we scope
- Inventory every credential the Helm release needs, including the metadata connection, Fernet key and API secret key
- Agree in writing which secrets live in Kubernetes Secrets and which use _CMD or _SECRET variants
- Engagements are fixed-scope, with deliverables and timelines agreed before work starts, and ongoing support available as a retainer
How we build
- Remove sensitive values from values.yaml and wire each one to a named Secret in your chart configuration
- Disable the matching built-in secret environment variables only where a _CMD or _SECRET variant replaces them
- Test a rotation of each key in a non-production namespace and confirm scheduler, API server and workers restart cleanly
How we hand over
- Deliver a secret inventory listing each key, its Kubernetes Secret, its owner and its rotation procedure
- Document the clock synchronization requirement that affects task log retrieval and how to verify it
- Walk your platform and security teams through the values files and the runbook used for future changes
How we support
- Review secret handling again when you upgrade the Airflow chart or move to a newer Airflow release
- Repeat rotation tests on a schedule your team agrees, with results recorded for audit evidence
- Confirm with your cluster administrators how Secrets are encrypted at rest and who holds list and watch access
