Which DAG code does an Apache Airflow run use?
The DAG bundle decides which DAG code an Apache Airflow run executes. For a platform team, that choice answers a direct audit question: can you show which commit produced the code behind a run from last Tuesday? In a self-hosted Airflow 3 deployment, DAG files reach Airflow through a bundle, and the bundle type sets whether each run is pinned to a version or picks up whatever is currently available. Three options matter. GitDagBundle pulls DAGs from a Git repository and records the commit for each run, so a run keeps the same code from start to finish. A local bundle reads files from a directory on the Airflow host, and an object storage bundle reads them from a bucket. Both always run the latest code, so a change made mid-run can affect later tasks, and a retry may execute different code from the original attempt.
Teams need this for change control because the bundle is the link between a reviewed change and the code that actually ran. With GitDagBundle, a merged pull request in your Git host can be traced to specific runs, which is the evidence auditors and incident responders usually request. Decide before build which records you must keep: the commit behind each run, the code a retry used, and who approved the merge. Then confirm the exact bundle settings, how versions appear in the UI, and how repository credentials are stored in the documentation for the Airflow version you deploy. The repository also becomes a dependency of your scheduling platform, so access to it and its availability belong in the same review as the rest of the platform.
Bundle choice is part of deployment design. In Airflow 3 the DAG processor runs as its own component, and it is the process that reads bundles and parses DAG files, so it needs a network path to your Git server or bucket and the credentials to reach them. Plan where that component runs, which secrets it holds, how many instances you operate, and how it handles a repository outage. These decisions also shape network rules and secrets handling, so settle them before the first production DAG is deployed. The Airflow 3 upgrade guide describes the component changes that teams moving from Airflow 2 need to plan for.
Git, local and object storage bundles compared
With GitDagBundle, Airflow reads DAG files from a Git repository you configure, and each run is tied to the commit it started with. The run keeps that code for its whole lifetime, so a push made during a run does not change the tasks already executing. Later runs pick up the new commit.
- How code arrives: Airflow fetches DAGs from the repository, so the repository, branch and credentials become part of the platform configuration.
- Version record: Each run records its commit, which gives auditors and on-call engineers a direct link from a failed run to the exact code that executed.
- Mid-run changes: A new commit affects runs that start afterward. Confirm how retries choose a version using the rerun setting for your Airflow version.
- What you build: Repository access for the scheduler and workers, deploy keys or tokens held in your secrets store, and a branch and review process for DAG changes.
- Network dependency: Every component that parses or runs DAGs must reach the Git host, so restricted sites usually need an internal mirror.
Confirm the bundle configuration keys, supported Git hosts and authentication options in the Airflow documentation for your exact version before designing the repository layout.
A local bundle reads DAG files from a directory on the scheduler and worker hosts, or from a directory baked into the container image. Airflow does not tie a run to a version from this source. It always executes the latest files it finds, so the version history lives in your deployment process.
- How code arrives: A deployment tool, configuration management job or image build places files in the directory on each component.
- Version record: Airflow does not record a commit or version identifier for the bundle, so your build or deployment log must show which files were live for each run.
- Mid-run changes: A file that changes on disk can affect a run in progress or a retry that starts later. Schedule deployments with this in mind.
- What you build: A sync or image pipeline that updates every scheduler and worker together, plus a check that all components hold identical files.
- Where it fits: Single-node installs, test environments and sites where no Git host or object store is reachable from the cluster.
Confirm how the bundle directory is set and how your chosen deployment method keeps components in step, using the production deployment guide as the starting point for your version.
An object storage bundle loads DAG files from a bucket in an S3-compatible or similar store. Like a local directory, it always runs the latest code, so version control moves to the process that uploads files to the bucket. Airflow components read whatever the bucket holds when they sync.
- How code arrives: A pipeline uploads DAG files to the bucket, and Airflow components retrieve them according to the sync behavior you configure.
- Version record: Bucket versioning and upload job logs can identify what was published, but Airflow itself does not record a commit for each run.
- Mid-run changes: An upload can land while a run is active, so a run may encounter code that changed after it started. Plan upload timing around your critical schedules.
- What you build: Bucket policies and least-privilege credentials for each component, an upload pipeline that validates DAGs before publishing, and monitoring of sync success.
- Infrastructure dependency: Every scheduler and worker must reach the bucket, which affects multi-site designs and restricted networks where egress is limited.
Confirm the supported object storage providers, sync options and versioning behavior in the Airflow documentation for your version, and test how a failed upload is handled before relying on this source for production.
Bundle and versioning options and when each fits
This table compares how Airflow 3 can source DAG code: GitDagBundle, local bundles and object storage bundles, plus the rerun setting that decides which code version a retry uses. Read each row against your change control, audit and rerun requirements. The main takeaway is that GitDagBundle is the option that records the commit each run used, and every option's settings should be confirmed in the Airflow 3.3.2 configuration reference before you standardize.
| Option | What it controls | Best fit | Trade-off |
|---|---|---|---|
| GitDagBundle | Pulls DAG code from a Git repository and ties each run to a commit | Teams that need a traceable record of the code each run executed | Needs repository access and credentials for the DAG processor and workers, managed as secrets |
| Local bundle | Reads DAG files from a directory on the DAG processor host | Development, test environments and small single-host setups | No built-in commit record, and file changes are picked up at the next parse |
| Object storage bundle | Syncs DAG files from a bucket or object store that CI publishes to | Teams that already publish build artifacts to shared storage | The latest upload runs, so confirm how prior versions are kept and recorded |
| Rerun version setting | Decides whether a retry uses the code from the original run or the current code | Pipelines where a retry must reproduce the first attempt exactly | Confirm the exact setting and its default in the 3.3.2 configuration reference |
| Mid-run code stability | Whether a deployment during a long run changes tasks not yet started | Long-running pipelines that span frequent merges or releases | Check your deployment cadence, since frequent changes can still complicate reruns |
Why the rerun setting decides which version a retry uses
A retry or a cleared task can run against a different code version than the first attempt whenever the bundle has moved on between them. With a Git-backed bundle, the first attempt of a run is tied to the commit it started with, so the rerun setting decides whether a retry uses that original version or the current head of the branch. Before you rely on any behavior, confirm in the Airflow documentation for your release which setting controls this, what its default is, and whether it governs retries, cleared tasks, or both. The answer determines what your team will see when a task fails at 2 a.m. and someone merges a fix.
The consequences fall on three groups. For data consistency, a task that wrote partial output on attempt one and then reruns with changed logic can leave one logical date with output from two code versions. For debugging, the logs from attempt one reflect the old code, so an engineer who reads the current file will draw the wrong conclusion. For incident response, a hotfix merged during an outage may be picked up automatically by retries, which can help or can change results the business has already started to review. Your runbooks should require recording the commit each run used and whether the rerun was pinned or moved forward.
Test the setting in staging before production. Push a deliberate DAG change that alters the output of one task, force a failure on the first attempt, then retry and clear the task. Compare the commit recorded for the run in the UI with the code shown in the task logs for each attempt. Repeat the test on every Airflow release you plan to run, because the behavior depends on the release. Record the confirmed setting, its value and the test result in your change management documentation so auditors and on-call engineers can see which rule applies.
How to set up Git-backed DAG delivery
This sequence takes a self-hosted Airflow 3 deployment from a version check to a DAG bundle that pulls from a Git branch or tag, records the commit on every run, and keeps code stable during a run. Each step ends with a check you can run before moving on. Confirm exact settings against the Airflow documentation for your version.
Confirm the Airflow 3 version and prerequisites
The platform owner confirms the target is Airflow 3 on a supported patch release, with the current patch listed as 3.3.2, and checks that the Python, Kubernetes and metadata database versions match the prerequisites. The metadata database must be PostgreSQL or MySQL, because SQLite is for testing only and MariaDB is not supported. If you are upgrading from Airflow 2, confirm you are on 2.7 or later first, following the upgrade guide. Check: a signed version matrix and a test environment that starts cleanly on the chosen database.
Separate the DAG processor from the API server
Airflow 3 starts the API server with the airflow api-server command and requires the DAG processor to run as a separate component. Deploy the DAG processor as its own process or container, using the Helm chart or official container images if you run on Kubernetes. Confirm in the documentation which component fetches bundle code before you assign the repository credential. Check: both components start, both report healthy in their logs, and the scheduler sees DAGs parsed by the separate DAG processor.
Create a read-only repository credential
The security or platform team creates a deploy key or service account token that can read only the DAG repository, with no write permission and no access to other repositories. Store the credential in your approved secrets backend and keep it out of DAG code and images. Name an owner responsible for rotation and revocation. Check: a test clone with the credential succeeds on the target branch, and a test push with the same credential is refused.
Configure GitDagBundle for the target branch or tag
Before this step, decide whether production follows a branch for continuous delivery or a tag for controlled releases. Add a GitDagBundle entry to the DAG bundle configuration with the repository location, the chosen branch or tag, and a reference to the stored credential. Confirm the configuration key names and value format in the Airflow documentation for your version, and set them through the Helm chart values if you deploy with it. Check: the DAG processor logs a successful fetch, and the DAG list shows the expected DAG IDs from that commit.
Set the rerun behavior for retries
Choose the rerun setting that determines whether a retried task uses the DAG version recorded for the original run or the latest version available. Document the decision with the owners of each pipeline, because it changes which code a retry executes after a hotfix. Confirm the setting name, accepted values and default in the Airflow documentation. Check: in staging, force a failed task to retry after a commit change and confirm the retry ran the version the documented decision predicts.
Run a DAG change through staging
Merge a small, low-risk test DAG change to the staging branch or tag, and let the DAG processor pick it up. Trigger a run of the changed DAG and watch it through the UI or API. Include one deliberate parse error in a second test change to confirm that the failure surfaces in the import errors view rather than silently stopping the pipeline. Check: the staging run completes on the new commit, and the broken test change shows an import error that the team can see.
Verify that a run shows the expected commit
Open the run details in the UI or API and compare the recorded commit with the commit you merged. Then push a second commit to staging while a long-running test DAG is in progress, and confirm the in-flight run keeps the original code while the next run uses the new commit. Keep a short record of both commit hashes as acceptance evidence for the setup. Check: each run records the commit it executed, and the mid-run change applies only to runs started after it.
Security and operational risks of DAG delivery from Git
Delivering DAGs from Git moves pipeline code through a chain of repository access, merge rights, scheduler components and runtime secrets. Each link can fail in a different way, so the controls below are meant to be checked as a set before the first production run.
Repository credentials and scope
The DAG processor needs read access to the repository to fetch the bundle, so the credential it holds can expose every pipeline definition in that repo. Use a read-only deploy key or token limited to the DAG repository and the branch or path it deploys. Confirm the credential rotates on a schedule and that the same token is not reused for write access elsewhere. Check how the bundle configuration stores the credential, and keep it out of DAG files.
Who can merge to the deployed branch
Whatever lands on the deployed branch becomes executable code in the scheduler's environment on its next parse. Protect that branch so merges require review by someone other than the author, and restrict who can push or force-push. Treat DAG changes with the same change control you apply to application code. Record the approver for each merge so a reviewer can trace who allowed a pipeline change to run.
Mid-run code changes and reruns
A GitDagBundle run keeps the commit it started with, but a rerun can pick up a newer version, depending on the rerun setting. A merge during a long run can therefore change what a retry executes. Decide which version reruns should use before go-live, and test a failed task with a rerun after a new commit lands. Confirm the setting in the Airflow documentation for your release, because the behavior is what auditors and on-call engineers will rely on.
DAG processor availability and parse failures
In Airflow 3 the DAG processor runs as its own component, so if it stops or cannot fetch the bundle, new runs stop being created and existing DAGs may show stale or missing state. A single syntax error can also stop a DAG from loading. Monitor the processor's health, alert on parse errors, and run the DAG test suite in CI before merge. Plan who is paged when the processor cannot reach Git.
Secrets that must not live in DAG files
DAG files are readable by everyone with repository access and are often copied into logs, backups and forks, so credentials written into them are exposed. Store connections and variables in a secrets backend or the metadata database with masking enabled, and have DAGs reference them by name. Scan commits for hard-coded keys before merge and check that task logs do not print secret values. Confirm which secrets backend your release supports before you design the bundle layout.
Audit evidence for which commit ran
An audit question such as which code produced a given output needs an answer tied to a specific commit, not to whatever the branch held that day. Record the commit each run used, retain those records for the period your policy requires, and link them to the change ticket and approval. Test the evidence by picking a past run and tracing it back to the merge, the reviewer and the code it executed.
Questions teams ask before setting up DAG bundles
What does our team need to provide before DAG bundle setup starts?
Your team needs to supply access and decisions before work starts, and those should be written into the scope. Expect to provide:
- Read access to the Git repositories that hold DAG code, plus details of your hosting platform and a service account the bundle can use.
- A named owner for each repository who approves branch and release rules.
- The Airflow environments you run, such as development, staging and production, and the teams that deploy to each.
- A contact for identity and secrets, since Git credentials need a secure home.
Your team also decides the rerun policy, because that choice determines which code a retry loads. Plan time for those reviews, since they gate the build.
How long does DAG bundle setup take, and what drives the duration?
The duration depends on the scope agreed in writing, and this page does not give a fixed figure. The main drivers are the number of repositories and how they are organized, the number of environments that each need their own bundle configuration, identity integration for Git access and any SSO on the Airflow UI, and how much testing of rerun behavior you want to see. Ask for a timeline that lists each dependency, including repository access, environment availability and the approval windows your change control requires. Confirm which items are included in the scope and who signs off each step.
Can our engineers configure GitDagBundle themselves without a partner?
Often yes, if they already run Airflow 3 and understand Git authentication, because the configuration is documented for each Airflow version. The harder work is usually the design decisions around the configuration: which commit a run should pin, how Git credentials are rotated, how environments map to branches, and whether the rerun setting fits your change control. Teams without a Git integration background often gain the most from a short review of their design before rollout. Confirm the exact options in the documentation for the Airflow version you run, and test them in a non-production environment first.
What does a partner scope and quote for DAG bundles?
A partner should break the work into written items and price each item against that scope. Expect the scope to cover:
- An inventory of DAG repositories, branches and the current deployment method.
- Bundle design for each environment, including the choice between GitDagBundle and local or object storage bundles.
- Git credential handling and integration with your secrets store.
- Rerun behavior decisions and a test plan that proves them.
- Runbooks, handover and knowledge transfer to your team.
DeployOpen does not publish a price list. A quote follows from the written scope, so ask for each item as a separate line, the assumptions behind it, what your team must provide, and which items recur, such as support retainers.
How do we test that a rerun uses the expected DAG version?
Test it with a planned sequence of steps that produces evidence. As an illustrative test, start a DAG run from a known commit, push a change to the repository while that run is still in progress, and then confirm the run still reports the original commit in its metadata and logs. Next, force a task failure, rerun that task, and check which version it loads under your rerun setting. Repeat the sequence for each environment and each bundle type you use, since behavior can differ between them. Confirm the expected behavior for your Airflow version in the documentation before you write the pass criteria, and keep the test in your pipeline so that each upgrade reruns it.
What changes for DAG bundles when we upgrade from Airflow 2?
The bundle design has to fit Airflow 3 changes, so plan it together with the upgrade. Airflow 3 requires you to be on Airflow 2.7 or later before upgrading, and the upgrade guide recommends moving to the latest 2.x release first. Check DAG code with the Ruff AIR301 and AIR302 rules before the move. The DAG processor runs as a separate component, which affects where bundle code gets parsed. Task code can no longer use Airflow database sessions or models directly, so DAGs that relied on them need changes. Review the Airflow 3 upgrade guide with your repository owners before you choose a bundle type.
Who maintains DAG bundles after go-live, and what does support cover?
Your team owns the repositories and the rules that govern them, and ongoing support can be added through a retainer. Recurring work usually includes Airflow patch upgrades, reviewing changes to bundle configuration, rerunning the rerun tests after each upgrade, rotating Git credentials and answering questions from DAG authors. Ask for the support scope in writing: which environments it covers, how changes get approved, what response expectations apply, and whether retesting after upgrades is included. Support costs recur, so compare them against the one-time build cost in the quote before you decide how to staff the work.
Limits to confirm in your environment
DAG bundles couple your Git hosting, network path and Airflow release together. Each limit below changes whether the Git-backed model works as planned, so confirm them with your Airflow version and your own infrastructure before you choose a bundle type.
Bundle types for your release
Confirm which DAG bundle types your Airflow version supports before you design the delivery path. Git-backed bundles with per-run commit recording are the case this page focuses on, while local and object storage bundles behave differently. Check the bundle options in the Airflow documentation for the exact release you run, and confirm that the version you plan to upgrade to offers the same options.
Git hosting and authentication methods
Confirm which Git hosts your organization uses and which authentication methods the scheduler and parsing components can use to reach them. Service accounts, deploy keys and token rotation all affect who can change the code that runs. Ask your platform team how credentials are stored, rotated and scoped to read-only access on the DAG repository.
Air-gapped and restricted network access
If your Airflow components run in an air-gapped network or behind a strict egress proxy, confirm how they reach the internal Git server. Parsing and task execution both depend on the repository being reachable from every component that needs DAG code. Plan an internal mirror or a controlled sync path, and test that the chosen route works from each network zone before go-live.
Repository size and commit frequency
Confirm how large the DAG repository is and how often teams push to it. Large repositories and frequent commits increase the time the DAG processor spends fetching and parsing bundle versions, and they can make the commit history used for each run harder to scan. Measure clone time and parse time on a representative repository, and decide whether DAG code should live in a separate repository from application code.
Rerun behavior in your release
Confirm which version of a DAG a retry or manual rerun uses in your Airflow release. The answer decides whether a rerun picks up a newer commit or repeats the original code, which matters for audit evidence and for pipelines that changed after a failure. Run a test DAG that fails, is changed in Git, and is retried, then record which commit the retry used.
Parser load on the DAG processor
Confirm that the DAG processor has the CPU and memory needed to parse every bundle version your team pushes. Each new commit can trigger a new parse, so a busy repository multiplies the work. Ask how many DAG files the environment holds, then benchmark the processor on a copy of the repository with realistic commit volume before you size the deployment.
Operating DAG bundles after go-live
Operating DAG bundles starts with watching for parse errors, because new code reaches production only after the DAG processor parses it and the scheduler picks it up. In Airflow 3 the DAG processor runs as a separate component, so a stalled or crashing processor leaves a merged commit invisible to new runs even though the repository looks correct. Check import errors in the Airflow UI after every merge, and alert on the health of both the DAG processor and the scheduler. Treat a parse failure in a bundle as a production incident for that pipeline, since no new run will use the corrected code until the failure is fixed.
Rollout is the decision that matters most. With a Git bundle, a merge to the tracked branch changes what the next run uses, while runs already in flight keep the commit they started with. Decide whether production follows a branch or a release tag, and promote changes through a staging Airflow that uses the same bundle configuration before they reach production. Upgrades need the same discipline: run the Ruff AIR301 and AIR302 rules over your DAG code, and if you are still on 2.x, move to the latest 2.x release first, as the Airflow 3 upgrade guide recommends. Keep the repository change, the Airflow version change and the DAG code change in one reviewed plan so that a single merge never changes two things at once.
Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes, so any deployment still on 2.x should plan its move to Airflow 3 before relying on this pattern. Confirm your current patch level against the version life cycle table. When an incident needs a commit trail, start from the run record, which shows the commit the run used, then check the repository history and any retained tags. Confirm with your own retention policy that run records and repository history outlast your audit window, and rehearse this lookup before you need it in an incident.
How DeployOpen scopes and delivers DAG bundle configuration
DeployOpen plans, builds and documents the DAG delivery path for Apache Airflow in infrastructure you control, including how a Git-backed bundle pins each run to a commit. Engagements are fixed-scope, so the bundle source, rerun behavior and access model are agreed in writing before work starts. Ongoing support is available as a retainer once the deployment is live.
How we scope
- Written scope names the Git repository, branch, and credentials Airflow uses to fetch DAG code.
- Scope records which bundles are Git-backed and which, if any, use local or object storage.
- Rerun and retry behavior is documented as a decision, so teams know which code version a retry uses.
How we build
- Configure the GitDagBundle with a dedicated read-only deploy credential stored in your secrets backend.
- Set up the DAG processor as a separate component, as Airflow 3 requires, and verify it reads each bundle.
- Test a mid-run commit change in staging so the team sees the pinned version hold for active runs.
How we hand over
- Runbook covering bundle sources, version lookup per run, and how to roll back a bad commit.
- Evidence pack showing which commit each sampled run used, for your change and audit records.
- Knowledge transfer sessions with the engineers who will own DAG releases after go-live.
How we support
- Ongoing support covers Airflow upgrades, bundle credential rotation and reviews of DAG delivery changes.
- Retainers can include recurring security testing of the Airflow API server and DAG delivery path.
- Upgrade planning checks bundle behavior against the Airflow version you run, before each change window.
