When a custom Apache Airflow Docker image is the right build
A custom Apache Airflow image is the right build when your containers need providers, Python packages or system libraries that the stock image does not include. In that case you extend the official image and pin every addition, so the result is repeatable and reviewable. You should not fork Airflow or rebuild its runtime from scratch. The project is licensed under the Apache License 2.0 (Apache-2.0), and there is no paid edition from the project itself, so the cost of this work is engineering time, build infrastructure and the operations that follow it.
The base you extend depends on the Airflow major version you intend to run. Airflow 3 was first released on April 22, 2025, and its current patch release is 3.3.2. Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes. If you are still on Airflow 2, a custom image built on it will inherit an unpatched base, so the image design should be paired with a migration plan. Confirm the version table and your upgrade path before you choose the base tag, since the image is the vehicle for every later Airflow patch.
Before you write a Dockerfile, decide who owns each addition and how it will be checked. Every provider, Python package and system library should trace to a DAG or operator that needs it, and each one adds to the attack surface and rebuild frequency. Ask your platform team which registry, scanner and CI system will build and store the image, and who approves a change to the base tag. These decisions shape the rest of the build, so settle them before the first image is produced.
Decisions to make before writing the Dockerfile
Each decision below changes what the image contains, how often it must be rebuilt, and who is accountable when it breaks. Settle them together before the first build, because the base tag, constraints and package sources determine whether a rebuild after an Airflow patch is routine or a project of its own.
Which base tag to pin
The base image tag fixes the Airflow version, the Python version and the OS layer your image inherits. Pinning a specific patch tag, rather than a floating tag, makes each build reproducible, but it means someone must move the pin deliberately on every Airflow patch. Decide who owns that move and how it is tested in staging. Confirm which Python versions and database clients the chosen tag ships, since the tag sets what your custom layers can rely on.
Whether DAGs are baked in
Baking DAG files into the image gives every release an exact, auditable code set, but each DAG change becomes an image build and deployment. Mounting or syncing DAGs from Git keeps code changes fast, at the cost of a second delivery path to secure and monitor. Each choice affects rollback: a baked image rolls back with its code, while synced DAGs may not. Decide this before the tag scheme is designed, because it determines what a tag must represent.
How constraints are applied
Installing packages without constraints lets pip resolve versions that may not work together with Airflow. Applying the constraints file published for your Airflow release and Python version keeps the dependency set known. You can also freeze a constraints file of your own when you add providers. Decide where that file lives, who updates it when a provider changes, and whether your build fails when the resolved set drifts from the file you reviewed.
Which providers are installed
Provider packages add operators, hooks and connections for systems such as warehouses, cloud services and secrets stores. Every provider you install adds code, transitive dependencies and a version you must keep current. Install only the providers your DAGs use, and record each one with its version in the constraints you own. Ask which providers each team needs and who will approve additions, since a later removal is harder than skipping an install now.
Which OS packages are added
Some Python packages need system libraries, such as client libraries for databases or compilers for native extensions. Each apt package you add enlarges the image, widens the patch surface your scanner must cover, and must be rebuilt when the base OS receives fixes. Prefer packages that are needed by a named DAG or provider, and remove build tools in the same layer when they are only needed to compile. Record the reason for each package so that a later audit can explain it.
Whether a private package index is used
A private index, such as an internal mirror of PyPI, lets builds run inside restricted networks and gives you control over which package versions can be fetched. It also becomes a dependency your build cannot run without, and it must be kept in sync with upstream releases. If your build runs with internet access, decide whether a mirror is still worth operating. If it runs in an enclave, confirm that the mirror holds the constraints file and every provider you pin.
Baking DAGs into the image or delivering them separately
With baked DAGs, the DAG files are copied into the image at build time, so every pipeline change produces a new image. The scheduler, workers and API server all read the same files from that artifact, which keeps the DAG code and the Airflow version in one unit that you build, scan and deploy together.
- Release coupling: A one-line schedule change requires a full image build, scan, push and rollout, so pipeline teams move at the same speed as the platform team.
- Rollback behavior: Rolling back means redeploying the previous image, which restores the previous DAG code along with the libraries it was built against.
- History problem: The Helm chart documentation warns that reusing the same image tag for baked DAGs loses code history. A tag such as latest or a fixed environment name points to different code over time, so a past run cannot be traced to the exact file that executed.
- Tag discipline: Give each build an immutable identifier, such as the Git commit of the DAG source plus a build number, and record the mapping in your change system.
- Audit value: The image digest becomes the evidence unit for what ran in a given window, which simplifies change management reviews.
Confirm how your CI pipeline names images, whether digests are recorded at deploy time, and how long your registry retains old images for rollback.
With this model, the image contains only Airflow, providers and libraries, and DAG files arrive after the containers start. Git-sync pulls them from a repository, while a shared volume receives them from a process you control. DAG code then moves on its own schedule, separate from image releases.
- Release coupling: DAG changes ship without an image rebuild, so pipeline teams can merge and see code reach the components on the sync interval you configure.
- Rollback behavior: Rolling back means reverting the commit. Check that the sync reverts files on every component, and check whether queued runs now reference code that has changed.
- History: Git keeps the full commit history, so each DAG version has a traceable identifier. Record the commit a run used in your deploy record.
- SSH known hosts: When git-sync uses SSH, set
dags.gitSync.knownHostsso the pod verifies the Git server's host key and is protected from man-in-the-middle attacks. Obtain and verify that key through your own trusted channel before deployment. The Helm chart DAG file guidance covers the related settings. - Volume option: A shared volume requires storage that supports the access mode your components need, and a bad write reaches every component at once, so the write process needs the same review as code.
Confirm the sync interval, the credentials the sync uses, and whether the repository accepts changes only through reviewed merges.
A hybrid model bakes the slow-changing parts of the environment into the image and delivers pipeline code separately. Shared operators, libraries and common configuration change rarely, while individual DAG files change often. Splitting them by change rate keeps the platform release process strict without making every pipeline edit wait for it.
- Split by change rate: Keep dependencies and shared code in the image, and keep pipeline definitions in the repository.
- Version pairing: Record which image version each DAG revision requires, so a DAG that imports a newly added library waits for the image that contains it.
- Rollback: Roll back the repository and the image independently, but test the exact pair you intend to run together in staging.
- History: Immutable image tags identify the platform build, and repository commits identify the DAG version. Store both identifiers in every deploy record.
- Governance: The image path is usually owned by the platform team and the DAG path by pipeline owners, so each path needs its own review rules and approvers.
Confirm which components read DAGs from Git and which from the image, and that each supported image and DAG pairing passes tests before production.
Image build options and when each fits
This table compares the four realistic ways to build the Airflow image for your environment, covering what each adds, who keeps it current, and when it fits. Read it by first deciding who should own the base image and rebuild cadence, then matching that to the option with the least custom surface.
| Build option | What it adds | Who maintains it | When it fits |
|---|---|---|---|
| Official image with a Dockerfile extension | Extra providers, Python packages and OS packages layered on the official Airflow image using a pinned constraints file | Airflow project publishes the base; your team owns the extension layer and rebuilds | Most teams that need a few providers or libraries and want to track official Airflow patch releases |
| Fully custom image from a base OS | Your own base distribution, installed Airflow, providers and libraries built from scratch | Your team owns everything, including the OS patch cycle and Airflow install steps | Organizations with a mandated hardened base image or strict OS standards that the official image cannot meet |
| Private registry mirror | Copies of the official image and build dependencies held in an internal registry, so builds and pulls do not depend on public networks | Your platform team operates the registry and sync process; the upstream image still comes from the Airflow project | Restricted or air-gapped networks, or where pull rate limits and provenance rules apply; confirm sync and signing steps with your registry team |
| Multi-stage build for compiled dependencies | A build stage with compilers and headers that produces wheels or binaries, copied into a slim runtime stage without build tools | Your team owns the build stage and the compiled packages it produces | Providers or Python packages that need native compilation, where keeping compilers out of the runtime image reduces attack surface |
Building and tagging the image step by step
This sequence turns the official Airflow image into a reproducible, tagged build that pins every version, carries only the packages your DAGs need, and proves it starts all four Airflow 3 components before it moves forward.
Choose the base tag from supported versions
Confirm which Airflow release line your team will run before writing a Dockerfile. Check the version life cycle table to make sure the release you pick is in active or limited maintenance, not end of life, and note the Python version the base image uses. Airflow 2 has reached end of life, so Airflow 3 is the line to target for new builds. Done means you have a written base tag, such as the official image for a specific patch release, and a named owner who will rebuild it when Airflow publishes a patch.
Pin the Airflow version and constraints file
Set the Airflow version as a build argument so it appears in one place and in your image history. Download the constraints file that matches that exact Airflow release and the Python version inside the base image, and store the constraint URL or a copy of the file in your repository. Pinning both values keeps every rebuild resolving the same package set. Done means a build argument and a constraints reference are committed, and anyone can read which versions a given build should resolve to.
Install providers and extra packages against constraints
Add the provider packages and Python libraries your DAGs import, and install them in the same pip step that references the constraints file, so pip cannot pull versions that conflict with Airflow's dependencies. Keep the list short and explicit, and avoid unpinned packages that would change between builds. Done means the build completes without resolver conflicts, and your requirements list matches the packages the DAG code imports.
Add system libraries only where needed
Install OS packages only when a provider or Python library needs a compiled dependency or a client library that cannot come from pip. Document each package with the provider or library that requires it, and remove build tools from the final stage if your build uses multiple stages. Note that Airflow 3.2.0 removed the MySQL client from the container images, so a MySQL-dependent provider may need you to add it back explicitly. Done means every system package has a stated reason, and the list is smaller than what a general-purpose base image would carry.
Decide whether to bake DAGs into the image
Choose one of two paths for DAG code. Baking DAGs into the image gives each build a fixed, testable bundle, but a DAG change then requires a new image and a new tag. Excluding DAGs and delivering them through git-sync or a volume keeps image builds independent of pipeline changes. Whichever path you pick, record the decision with the reason, because it determines how you roll back. Done means a written decision exists, and if DAGs are baked, the Dockerfile copies them in with a deliberate path.
Build and tag each image with its history
Build the image and tag it with a unique value that identifies the Airflow version, the Python version, and your build, such as a version number plus a commit reference. Avoid reusing a tag for different contents, because a reused tag hides which DAG code and packages ran in the past. Push the image to your internal registry. Done means the registry holds an immutable tag for this build, and the tag maps back to a specific Dockerfile commit.
Verify resolved versions and component start
Before promoting the image, run pip check and list the installed Airflow and provider versions, then compare them with the constraints file. Start the image with the api-server, scheduler, dag-processor and triggerer commands in turn, and confirm each process stays up and connects to the metadata database in a test environment. Also confirm the DAG parser loads your DAGs without import errors. Done means you have a saved record of resolved versions and four clean component starts. Check the release notes for version-specific removals before you accept a new Airflow patch, and rerun this verification each time the base tag changes.
Scanning, secrets and supply chain controls for the image
A custom Airflow image bundles your providers, Python packages and system libraries into one artifact that every scheduler, worker and API server runs. Whoever controls how that artifact is built, scanned and stored controls what runs in your pipelines. The controls below connect the build pipeline to the registry so each image can be traced back to its inputs.
Scan every build before release
Run a vulnerability scan on each image the build produces, not only on the base image. Packages you add through pip and apt often carry more exposure than the official Airflow layers, and the scan should cover Python packages as well as operating system libraries. Decide in advance what blocks a release and who reviews findings that have no fix yet. Ask your scanning vendor how it handles findings in the Airflow base layers that you have inherited.
Pin base images by digest
A tag such as a minor version can be repointed to new content without any change in your repository. Pin the FROM line of your Dockerfile to an image digest, and update that digest through a reviewed change so that each build is reproducible. Record the digest alongside the Airflow version in your release notes so you can answer which base image a running pipeline used.
Keep secrets out of build layers
Anything written into an image layer, including a token used to clone a private repository or to reach a private package index, can be recovered by anyone who can pull the image, even if a later layer deletes it. Pass credentials through build-time secret mounts, or install from a source that needs no credential at build time. Confirm with your registry team that build logs do not echo those values.
Verify package sources and versions
Install Python packages only from the index you have approved, and install them against the constraints file for your Airflow release and Python version. Mirror those packages into an internal index if your network requires it. Keeping the constraint file under version control lets reviewers see exactly which provider and library versions changed in each build.
Restrict who can push to the registry
Only the build pipeline should be able to push tags for the Airflow image, and no one should overwrite a tag that already runs in production. Use separate repositories or protected tag rules for release images, and require review for changes to the Dockerfile and constraints file. Confirm which roles can delete images, since deletion removes the rollback point.
Retain build provenance
Store, for each image, the Git commit it was built from, the base image digest, the constraints file used, the scan result and the Airflow version. Keep that record for as long as the image may still be running or may need to be rolled back. Provenance is what lets you show an auditor or incident responder which code and packages a given pipeline executed.
Rebuilding on every Airflow patch release
The patch cycle turns each Airflow release into a repeatable sequence: confirm the version, rebuild the image against fresh constraints, check DAG code, test in staging and promote a new tag. The result is a known-good image for every patch, with the previous tag still available for rollback.
Track the version support table
Your platform owner watches the Airflow version life cycle page for new patch releases and for status changes on the lines you run. Airflow 3 is the current major line, with 3.3.2 as the current patch release at the time of writing. Airflow 2 (current patch 2.11.2) entered limited maintenance on October 22, 2025 and reaches end of life on April 22, 2026, after which it receives no fixes. Done when each base version in use is recorded with its support status and a named owner for the next move.
Trigger a rebuild for the new patch
When a patch lands, the owner opens a change request that names the new Airflow version, the Python version and the image tag scheme. The build pipeline is set to take the new official base image and the matching Python version, and nothing else changes in the Dockerfile. Done when the change request is approved and the build is queued from the same Dockerfile and requirements files used for the last tag.
Rerun constraints resolution
The build fetches the constraints file for the new Airflow version and Python version, then reinstalls your extra packages and providers against it. Any package that no longer resolves cleanly fails the build here, which is the point of the step. Done when the build completes with a resolved dependency set, and the lock output or pip freeze is saved as an artifact with the image tag.
Run DAG checks with Ruff AIR301 and AIR302
Before the image is tested, the DAG code is scanned with Ruff using the AIR301 and AIR302 rules, which flag patterns that the Airflow 3 upgrade guide marks as removed or changed. Findings are fixed in DAG code or recorded as accepted exceptions with an owner. Done when the scan runs in CI with zero open findings on the DAG repository for the target version.
Test the image in staging
The new image is deployed to a staging environment that uses the same metadata database version, executor and DAG delivery method as production. Teams run representative DAGs, check provider connections, confirm that the API server and scheduler start, and review task logs. Done when the staging run completes without new errors and the test results are attached to the change request.
Scan the image before promotion
The image is scanned for known vulnerabilities in the operating system packages and installed Python packages, and the results are compared with the previous tag. Findings that block release are agreed with the security owner before promotion. Done when the scan report is stored with the image digest and the approver records any accepted risk.
Promote a new tag and keep the old one
The approved image is pushed under a new immutable tag that includes the Airflow version and a build identifier, and the production deployment is updated to reference it. The previous tag is kept in the registry so the team can roll back by changing one value. Done when production runs the new tag, the rollback target is documented, and the change record links the build, scan and staging results.
Plan moves off unsupported bases
Images built on a version that has reached end of life or is in limited maintenance get a dated migration plan rather than another patch cycle. Airflow 2 reached end of life on April 22, 2026, so any remaining Airflow 2 image needs a planned move, which the upgrade guide describes as moving to the latest 2.x first and checking DAG code with the Ruff rules. Done when every unsupported image has a migration owner and a target version in the tracking record.
Limits to confirm in your environment
A custom image inherits every version, removal and dependency decision made upstream, so the choices below need checking before the first build. Confirm each one against the release notes for the Airflow version you are targeting, and record the answers in the image build specification.
Removed options in Airflow 3.2.0
Airflow 3.2.0 removed the --conn-id option from airflow connections list, removed methods from TaskInstance and PriorityWeightStrategy, and dropped the MySQL client from the container images. Any wrapper script, DAG utility or plugin that calls these will fail after a rebase onto the new image. Search your repository for each removed item and run the affected commands in a staging build before you change the base tag.
Python and database version support
Airflow 3.3.2 lists Python 3.10 to 3.14 and PostgreSQL 14 to 18 as prerequisites, along with MySQL 8.0, 8.4 and Innovation releases. Your base image Python version, any compiled Python packages and your metadata database version must all fall inside the supported range for the release you pick. Confirm the matrix for your exact target version, since it changes between patch lines.
System libraries for providers
Some providers need OS packages such as compilers, client libraries or certificates that pip cannot install. Add them in your Dockerfile with a pinned base image, and list each package and the provider that needs it. Confirm the requirement for every provider in your constraint set, because a package that installs cleanly on one base image may be missing or named differently on another.
Registry access in air-gapped networks
A rebuild pulls the base image, OS packages and Python packages from registries. In an enclave, confirm that your internal registry mirrors the official Airflow base image, the OS repositories and the PyPI index for every provider you install. Also confirm that a build can reach your scanner's vulnerability database, or that the scanner can run from an offline feed.
Helm chart image settings
The Helm chart must point at your custom image through its image repository, tag and pull secret settings, and the same image must be used by every component that runs Airflow code. Check that the chart version you deploy accepts the image layout you build, and that the chart's DAG delivery settings match your decision on baked DAGs. Confirm these values in the chart documentation for your version before the first upgrade.
Questions teams ask before building a custom Airflow image
Can our team build a custom Airflow image without outside help?
Yes, many teams can, provided someone owns the image as a product. The official image is a sound base, and extending it means adding providers, Python packages and system libraries on top of a pinned Airflow version. The harder part is the routine: rebuilding on every Airflow patch, checking the new constraints file, scanning the result and testing DAGs against the new image. Ask whether your engineers have time for that cycle every month, not just for the first build. DeployOpen can plan and build the image as a fixed-scope piece of work, then document the rebuild steps so your team can run them afterward.
What makes a custom Airflow image take more effort to build?
Effort rises with the number of things you add and how hard each one is to install. The main drivers are:
- The number of provider packages and whether they must match one another and the Airflow version.
- Native dependencies that need compiled libraries or OS packages, such as drivers for specific databases.
- How DAG code reaches the image, since baking DAGs in changes the release process.
- The registry and scanning setup, including whether you already have a build pipeline and a place to store images.
Each extra provider or native library adds testing, not just installation time.
Which costs recur after a custom Airflow image is built?
Most of the cost after launch is recurring upkeep, not the first build. Expect recurring work in three areas: rebuilding and retesting the image on every Airflow patch release, updating provider packages when they change, and fixing findings from image scans. Airflow 2 has reached end of life, so teams still on it face migration work on top of routine patching. Ask any partner to separate the one-time build from a recurring rebuild and test cycle, and to state what each cycle covers.
How often do we need to rebuild the image after Airflow releases?
Rebuild whenever a patch release that matters to your deployment is published, and check the support table for which lines still receive fixes. Limited-maintenance versions receive security and critical bug fixes only, and end-of-life versions receive none, so a stale image can quietly fall out of support. Set a documented rule for how quickly a security patch must reach staging and production, and confirm the current support dates on the official version life cycle page before you commit to a schedule.
What do we need to provide for the image build?
The work needs a few things from your side before it starts:
- A container registry you control, with access for the build pipeline and the cluster that pulls images.
- A build pipeline that can run on a schedule and on demand, plus image scanning integrated into it.
- A staging cluster or environment where new images run real DAGs before production.
- A named owner for DAG code who can confirm which tests and dependencies matter and who approves changes.
Without a staging path and an owner, the image will be built but not safely released.
Does DeployOpen issue certifications for our Airflow image?
No. DeployOpen does not issue audit reports, attestations or certifications. It can prepare systems, controls and evidence for compliance assessments, such as image build records, pinned versions, scan results and change logs, and that evidence can be handed to the independent auditor or certification body you work with. Confirm with your assessor which artifacts they expect before the build starts, since their requirements decide what evidence is useful.
Should we bake DAGs into the image or deliver them separately?
Baking DAGs into the image ties every DAG change to an image build, which simplifies rollback but slows releases for pipeline authors. Delivering DAGs separately, through Git sync or mounted volumes, decouples code releases from image releases. The Helm chart documentation warns that reusing the same image tag for baked DAGs loses code history, so each build needs a unique tag tied to the commit. Decide this early, because it shapes your registry, pipeline and rollback procedure.
How DeployOpen scopes and delivers a custom Airflow image
The scope is agreed in writing before any Dockerfile is written. The scope lists the Airflow release line, the Python version, each provider package and every extra Python package or system library the DAGs need, along with the reason for each. It also records the constraints file the build will use, the base image tag, where DAG code will live (baked in or delivered separately), the registry and tag scheme, and who approves a rebuild. Most engagements are fixed-scope, so scope, deliverables and timelines are settled in that document, and DeployOpen does not publish prices. A quote follows from the written scope, and the factors that move it are the number of providers, the amount of native library work, the registries and scanners to integrate, and the handover documentation the client requires. Confirm with your own platform team which registry, CI system and scanner you will use, since those decide much of the build's shape.
The build runs in your registry and your pipeline. DeployOpen implements the Dockerfile, the constraints wiring and the CI job in the systems you already operate, so the image is produced, tagged and stored where your security team can see it. Each build records the Airflow version, the constraints file used and the image digest, so an auditor or incident responder can reproduce a given image. Test the result in a non-production environment with the same executor and metadata database as production before any tag is promoted. Scanning findings, base image updates and Airflow patch releases each need an owner and a rebuild trigger. Ongoing rebuilds and patch support are available as retainers, which suit teams that want the image refreshed on every Airflow patch without reopening the project each time.
The rebuild runbook is a deliverable, not an afterthought. DeployOpen documents the steps to rebuild on a new Airflow patch, a new provider release or a base image security update: which constraints file to pull, how to run the upgrade tests, how to roll the tag forward and how to roll it back. The runbook names who approves each step and what evidence is kept for change management. Ask any delivery partner to show you this runbook in draft before work starts, and confirm that your team can run it without the partner present.
