How DAG code reaches an Airflow Helm deployment
The Helm chart gives you three ways to get DAG files into a Kubernetes deployment: bake them into the image, run a git-sync sidecar with or without persistence, or mount a PVC that something else populates. The choice decides where DAG code lives at runtime, how quickly a change reaches the scheduler, DAG processor and workers, and how cleanly you can roll back. The official chart runs the official Docker image, so whichever method you pick, the DAG code has to be visible at the same paths in every component that parses or executes it.
The chart documentation generally recommends git-sync with local volumes, which keeps Git as the source of truth and lets each pod pull the revision it needs. Baked images give you immutable, reviewable artifacts, but they tie every DAG change to an image build and deploy. An externally populated PVC suits teams that already sync code through their own pipeline, but it moves responsibility for consistency, access and history onto that pipeline. The chart also requires that DAGs come from a single Git repository, so if your pipelines span several repositories, plan that consolidation or a build step before you choose.
Before you commit, confirm three things in your environment: which repository and branch will be the source, whether your Git server is reachable from the cluster, and how your team will roll back a bad DAG release. Ask your platform team to test one rollback end to end in a non-production cluster, since the answer changes with each option. Then read the Helm chart guidance on managing DAG files for the exact settings that match your choice.
Which DAG delivery option fits which release model
This table compares baked images, git-sync with and without persistence, and externally populated PVCs on release speed, rollback, code history, operational load and code ownership. Read each row against your release cadence: the option that rolls back cleanly for one team can make audit evidence harder for another.
| Factor | Baked DAGs in the image | git-sync with persistence | External PVC populated separately |
|---|---|---|---|
| Release unit | Image build that includes DAG code | Git commit pulled by a sidecar into a volume | Files copied into a shared volume by your own process |
| Release speed | Each DAG change needs a new image build and chart upgrade | DAG changes follow the Git branch without a chart upgrade | Depends on the copy job you build and how often it runs |
| Rollback | Roll back by redeploying an earlier image tag, which requires unique tags per build | Roll back by pointing the sync to an earlier Git revision | Roll back only if your copy process keeps prior versions |
| Code history | Kept only if each build gets a unique tag; reused tags lose history, per the chart docs | Full history lives in the Git repository | Only what your copy process records |
| Operational load | Image registry, scanning and rebuilds on every DAG change | Git credentials, SSH known hosts and sync health to monitor | Storage class, volume access mode and a copy pipeline to maintain |
| Who owns the copy | The image pipeline and its registry | The Git repository owners and the cluster team | Your platform team, which must prove what was copied |
| Single-repository limit | Code can come from any source at build time | The chart docs say DAGs must come from one Git repository | Not limited by the chart, but you define the source |
How git-sync works in the chart and where it breaks
The git-sync sidecar is the pattern the Helm chart documentation generally recommends for getting DAG code into a Kubernetes deployment. A sidecar container runs alongside the Airflow components that parse and execute DAGs, clones the configured repository into a shared volume, and keeps pulling new commits so the files on disk track the branch. Airflow then reads DAGs from that volume as if they were local files. Persistence is the setting that changes how this behaves. Without persistence, the clone lives in ephemeral storage, so each pod start performs a fresh clone and any restart or rescheduling repeats that work. With persistence, the clone survives container restarts within the pod's lifecycle, which reduces repeated fetches but still does not survive a pod being replaced on another node unless the volume itself is shared or durable. Confirm with your platform team which storage class backs the volume and whether it is reachable from every node that may run a scheduler or DAG processor. The chart docs favor local volumes for this reason: they keep the sync path close to the pods that read the files, and they avoid making a shared network filesystem a dependency of DAG parsing.
The most important limit is that the chart docs say DAGs must come from a single Git repository. If your organization keeps pipelines in several repositories, for example one per data domain or one owned by a vendor, you cannot point the chart at all of them at once. Your options are to merge the code sources into one repository, with an agreed folder layout and ownership rules, or to choose another delivery method, such as baked images or an externally populated PVC, which are covered in the other sections of this page. Decide this before you configure git-sync, because moving from several repositories to one later changes branch policy, review ownership and the history that auditors will read. For SSH-based access, set dags.gitSync.knownHosts so the sidecar verifies the server's host key. Leaving it unset exposes the clone step to man-in-the-middle attacks, and the chart docs call this out directly. Record who owns the known hosts value and how it is rotated.
Setting up git-sync with SSH known hosts
This sequence takes DAG code from a Git repository into a Kubernetes deployment through the Helm chart's git-sync sidecar, with SSH host verification in place from the first sync. Each step ends with a check, so the team knows the synced revision and a test DAG before it relies on the pipeline.
Confirm git-sync is the right delivery method
The platform team and the DAG authors agree that DAG code will live in one Git repository, that the Helm chart will pull it with a git-sync sidecar, and that baked images are not used for the same code. Write this decision down with the reason, including who approves changes to the repository and how releases are tagged. Done when the team can name the single repository that feeds the cluster and the person who can merge to its main branch.
Create the repository and read-only credentials
Create the DAG repository on the internal Git server or hosting platform you control, then create a read-only credential for the cluster. For SSH, generate a dedicated key pair and add the public key as a deploy key with read-only access. For HTTPS, use a token scoped to read the repository only. Store the private key or token in a Kubernetes Secret outside the values file. Done when a test clone succeeds from a machine that has the same network path as the cluster, and the credential cannot push.
Collect and verify the SSH host keys
Before you write any values, obtain the host public keys of your Git server from its administrators over a channel you trust, and compare the fingerprints with the values shown by the server's own administration pages. Do not accept a key shown by the first connection without this comparison. Record the exact key lines that will go into the chart. Done when the team has written fingerprints that match the server, and an administrator has confirmed them out of band.
Set dags.gitSync options in values
In the values file, set the git-sync options under dags.gitSync: the repository URL, the branch or revision to track, the subpath that contains DAG files, the SSH key secret reference, and the sync interval. Keep credentials as references to the Kubernetes Secret, not as literal values. Commit the file to the deployment repository so the change is reviewed. Done when a Helm template render shows the git-sync container with the expected repository, branch and secret name, and no key material appears in the rendered output.
Populate dags.gitSync.knownHosts
Set dags.gitSync.knownHosts to the verified host key lines from step three. Without this entry, an SSH sync can accept a key from an impostor server, so this value is required for SSH and not optional. Render the chart again and confirm the known hosts content appears in the git-sync configuration. Done when a sync against a host with a different key fails, which you can test in a non-production namespace by pointing a throwaway configuration at a wrong key.
Deploy the release with Helm
Install or upgrade the Airflow release with Helm using the reviewed values file and the Secret created earlier. Watch the rollout until the scheduler, API server, DAG processor and any other components are running, and read the git-sync container logs for the first clone. Record the Helm revision number so the release can be rolled back. Done when the pods are ready and the git-sync logs show a successful clone of the expected branch without host key warnings.
Verify the synced revision and a test DAG
Compare the commit hash that git-sync reports in its logs or shared volume with the commit at the tip of the tracked branch in the Git server. Then push a minimal test DAG to a branch in the repository, wait for the next sync, and confirm the DAG processor parses it without import errors. Check that the DAG appears in the UI or through the REST API, and that a manual trigger runs. Done when the synced commit matches the repository, the test DAG parses and appears, and the team has removed or disabled the test DAG afterward.
Security risks in DAG delivery paths
Every option for getting DAG code into a Kubernetes deployment moves credentials, code trust and audit evidence to a different place. The cards below cover the risks that follow from that choice, so you can decide which controls to build before the first release.
Repository credentials stored as Secrets
Git-sync needs a token or SSH key to read a private repository, and that credential is usually stored as a Kubernetes Secret. Secrets are stored unencrypted in etcd by default, so anyone who can read etcd backups or the raw datastore can recover the repository credential. Confirm that encryption at rest is enabled for Secrets in your cluster, and use a read-only, repository-scoped token rather than a personal credential.
Least-privilege RBAC for DAG secrets
Any service account or user that can list or watch Secrets in the Airflow namespace can read the git credential, whether or not they ever touch Airflow. Restrict get, list and watch on the git-sync Secret to the service accounts that need it, and keep Airflow operators out of broad namespace-wide read roles. Review these bindings on every chart upgrade, since new components may request additional access.
SSH known_hosts left unset
When git-sync connects over SSH without a pinned host key, the first connection trusts whatever server answers, which opens a path for man-in-the-middle attacks on DAG retrieval. The chart docs direct you to set dags.gitSync.knownHosts so the host key is verified. Compare the key against the one your Git provider publishes, and treat any change to it as an incident until it is explained.
Who can push DAG code
Code merged into the DAG repository runs with the permissions of the Airflow workers, so push and merge rights are effectively deploy rights for every connection and variable those workers can reach. Protect the default branch with required reviews, restrict who can approve merges, and limit which identities hold write access. Ask who can change the branch that git-sync tracks, because that is the change path that matters most.
Pod Security Standards for the sidecar
The git-sync sidecar runs next to the scheduler and workers in the same pod and often needs a writable shared volume. Apply the Pod Security Standards at the namespace level, check which level the chart's pods require, and confirm that privilege escalation and root execution are off where the sidecar allows it. Record the level you enforce so an auditor can see the boundary.
Audit trail from Git versus image tags
Git-sync records every DAG change as a commit with an author, reviewer and timestamp, which gives a direct audit trail for who changed which code. Baked images reuse tags unless you enforce unique tags, and the chart docs warn that a reused tag loses code history. Decide which record is your source of truth, retain it for the period your change management controls require, and confirm that the running revision can be traced back to a commit.
Questions teams ask before choosing a DAG delivery method
Which DAG delivery method is fastest to roll back?
Baked DAGs are usually the slowest to roll back and git-sync is usually the quickest, because a rollback with git-sync means moving the synced repository to an earlier commit rather than rebuilding and redeploying an image. Baked images tie DAG code to an image tag, so rolling back means deploying the previous image. The Helm chart docs warn that reusing the same image tag for baked DAGs loses code history, so every build needs a unique tag if you want a clean rollback path. Confirm the exact behavior of your chosen setup in a staging release before you rely on it during an incident.
Is git-sync required to deliver DAGs with the Helm chart?
No. The Helm chart can deliver DAGs baked into the image, through a git-sync sidecar, or from an externally populated persistent volume claim. The chart docs generally recommend git-sync with local volumes, but baked images and external PVCs are supported choices. Whichever you pick, the docs say DAGs must come from a single Git repository, so teams with several repositories need to combine them into one source or choose another architecture before they start.
How does persistence change behavior after pod restarts?
Persistence determines whether synced DAG files survive a pod restart. With git-sync and no persistence, the sidecar pulls the repository again when a pod starts, so the DAG folder is rebuilt from Git. With persistence, the synced files remain on a volume, which can speed startup but adds a storage object you must back up, size and monitor. An external PVC is populated by a process you run yourself, so its freshness depends on that process. Check how your storage class behaves on node failure before you choose persistence.
What effort should our own team expect compared with a partner?
Effort depends on the number of repositories, how secrets are handled, and whether a CI pipeline and separate environments already exist. A single repository with an existing pipeline and a git-sync setup is the smallest case. Multiple repositories, a new CI pipeline, per-environment branches, and SSH key handling each add build and review work. Your team pays for its own time in every case. A partner is paid for the written scope, which should name the repositories, environments, secrets approach and handover. Ask for the effort breakdown by component in any quote.
Can our team set this up alone?
Yes, if your engineers already run Kubernetes, Helm releases and a CI pipeline, and they can read the Helm chart's DAG management documentation closely. The hard parts are less the chart values than the surrounding decisions: keeping one DAG repository, handling SSH known hosts, planning image tags that preserve history, and proving that a rollback works. Teams without recent Kubernetes or Helm experience usually benefit from a short review of the design before go-live. Decide who owns the repository, the image build and the on-call response before you start.
How do we rotate SSH keys for git-sync without downtime?
Add the new deploy key to the Git hosting service first, then update the Kubernetes Secret the chart reads, then confirm git-sync pulls successfully before you remove the old key. Keep the dags.gitSync.knownHosts value set during the change so the host key is still verified, and do not disable verification to make the rotation easier. Plan the rotation in a staging release first, and keep the old key valid until the new one has synced. Store the key in a Secret rather than in values files, and restrict who can read it through least-privilege RBAC.
Which secrets risks come with git-sync over SSH?
The main risks are a missing or unverified host key, an over-permissioned deploy key, and a private key stored in a values file. The Helm chart docs say git-sync over SSH should set dags.gitSync.knownHosts to avoid man-in-the-middle attacks. Kubernetes stores Secrets unencrypted in etcd by default, so administrators should configure encryption at rest and restrict list and watch access. Use a read-only deploy key scoped to one repository and review who can read the Secret.
How each option affects release speed and rollback
Baked DAGs tie every DAG change to an image build and a Helm upgrade. Each edit to a pipeline means building a new image, pushing it to your registry, and upgrading the release so the scheduler, DAG processor and workers all pick up the same code. That path is slow but predictable, and every change is traceable to an image. Rollback means redeploying an earlier image tag. The Helm chart documentation warns that reusing the same tag for baked DAGs loses code history, so the build pipeline should generate a unique tag per commit or build. Confirm that your registry retains old tags and that your release process records which tag each environment runs.
git-sync decouples DAG changes from image releases. A sidecar pulls the DAG repository on a schedule, so a pipeline fix can reach Airflow without rebuilding the image or running a Helm upgrade. Rollback becomes a Git operation: you revert or check out an earlier revision, and the next sync applies it. Two checks matter before you rely on this. Confirm the sync interval and whether a bad commit can reach production before review, since git-sync will apply whatever the branch contains. Also confirm that the repository is the single source for DAGs, because the chart expects one repository and mixing sources complicates both rollback and audit.
External PVCs depend on how the outside process writes and versions files. The chart can mount a volume that another process populates, such as a CI job or a data platform sync. Airflow does not record which revision a file came from, so release speed and rollback are only as good as that external process. Ask who writes to the volume, whether writes are atomic so a scheduler never reads a half-copied file, and whether the process keeps prior versions you can restore. Without those answers, a rollback may mean copying old files by hand, and the audit trail lives outside Airflow.
Promoting DAG changes from staging to production
This sequence moves a DAG change from a developer's branch to production in ordered, checkable stages, so each environment runs the same code that was tested before it. The exact CI tooling is your choice, but the gates and the record of what ran where should be the same whichever tool you pick.
Agree the branch and tag model
Your platform team and the DAG authors choose how code moves between environments before the first release. A common pattern is a protected main branch that feeds staging, with production pointing at a tagged commit or a release branch. The team writes this down, including who may approve a merge and who may move the production pointer. It is done when the model is documented in the repository and the chosen Git reference for each environment is set in the Helm values for that environment. Confirm that your Git host supports the protections you need, since the chart expects a single repository for DAG code.
Merge changes through review and automated checks
DAG authors open a merge request against the staging branch. Reviewers check the Python code, and the pipeline runs linting, unit tests for custom operators and helpers, and a DAG import check that loads each file with the same Airflow version and providers as the cluster. A failed import is a blocking result, not a warning. The step is complete when the merge request shows passing checks and an approval recorded in Git history.
Sync the DAGs into the staging deployment
With git-sync, the sidecar on the DAG processor pulls the staging reference from the repository on its configured interval, so no manual step is needed beyond the merge. With baked images, the pipeline builds a new image from the merged commit, tags it with a unique value such as the commit hash, and deploys that tag to staging through a Helm upgrade. With an external PVC, a job copies the approved files into the volume. In each case, confirm that the staging DAG processor reports the new revision and that the DAG list reflects the change.
Run parse and smoke checks in staging
Check that every DAG in the changed set parses without errors in the staging deployment, using the Airflow UI or the API server. Then trigger a representative run against non-production data or sandbox connections, and confirm that tasks reach success and that the expected logs and outputs are produced. Record the run identifiers. The step closes when the checks pass and the reviewer signs off on the staging result.
Promote the same revision to production
Promotion should move the exact tested revision, not rebuild from a different branch. For git-sync, the production reference is advanced to the tested commit or tag through a reviewed change to the environment values. For baked images, the production release uses the image tag already validated in staging, so the code that ran in staging is the code that runs in production. Keep the approval for this change in the same audit trail as the staging merge. It is done when the production values show the tested reference and the change is recorded.
Verify production sync and parsing
After the production change, confirm that the DAG processor has picked up the new revision and that each DAG parses without import errors. Check the scheduler and DAG processor logs for parse failures, and confirm that the next scheduled runs are created. If a DAG fails to load, the change is considered failed and the production reference returns to the previous revision. The step is complete when the production UI and logs show the expected revision and no new parse errors.
Record the release and keep the rollback path ready
Record the Git commit or tag, the image tag if one is used, the Helm release revision and the approvers in a release log. Keep the previous known-good reference handy, since rollback depends on it. With git-sync, rollback means repointing to the earlier commit or tag. With baked images, it means redeploying the previous image tag. Confirm the rollback path works in staging before you rely on it in production. The step is done when the log entry exists and the rollback procedure has been written down and tested.
How DeployOpen scopes and delivers this work
The DAG delivery choice is settled in writing before any cluster work starts. DeployOpen's scope for a fixed-scope Airflow deployment inside your own cluster names the delivery pattern (baked image, git-sync with or without persistence, or an externally populated PVC), the single Git repository that holds DAG code, the branch or tag each environment tracks, and who owns the image registry and the Git server. The written scope also records how DAG changes and Airflow image changes are released, because those two paths can be decoupled or tied together depending on the option chosen. Your team should confirm which repository and hosting platform will be used, whether SSH or HTTPS access is required, and who approves changes to the scope.
Configuration and secrets are handled as part of the same plan. The Git credential, any SSH key, and the dags.gitSync.knownHosts value are kept out of the values file and supplied through Kubernetes Secrets in the namespace where Airflow runs. DeployOpen documents which secrets exist, which service accounts can read them, and how they are rotated. Verification covers the delivery path itself: a DAG change is pushed, picked up by the parser, run, and then rolled back by pointing the deployment at a previous image tag or Git revision. The results are written into the handover documentation together with the runbook for upgrades. Optional retainer support can cover chart and Airflow upgrades, checks against the supported Kubernetes versions, and periodic review of the delivery path, under terms agreed separately.
