Google Cloud Composer vs self-hosted Apache Airflow

Cloud Composer runs Airflow for you on GKE, but it supports Airflow 2 and Airflow 1 rather than Airflow 3. Self-hosting gives you the version, executor and data location choice, at the cost of running the platform yourself.

Book a callTalk to us
On this page

When Cloud Composer fits and when self-hosted Airflow fits

Choose Cloud Composer if you want Google to run the GKE cluster, the managed database and the Cloud Storage bucket that each Composer environment brings, and your team is content to work within Composer's supported Airflow versions. Choose self-hosted Apache Airflow if you need Airflow 3, need to pick executors yourself, or need direct control over where the scheduler, database, workers and data sit inside your own infrastructure. Neither option is better in every case, and the right answer depends on which of those needs is fixed for your team.

The version question comes first. Google's Cloud Composer overview says Composer 3 and Composer 2 support Airflow 2, and that Composer 1 supports Airflow 1 and Airflow 2. The overview does not mention Airflow 3. Airflow 2 reached end of life on April 22, 2026, and the current release is Airflow 3.3.2. If your DAGs must run on Airflow 3, confirm with Google which Composer release supports it before you plan around Composer. Ask for the supported-version statement in writing, with the date it applies, because the Composer documentation is the source for that answer and it may change. Also note that the overview describes Composer as evolving into Managed Service for Apache Airflow, so ask what that transition means for existing environments.

Control is the second question. Each Composer environment is a self-contained Airflow deployment on Google Kubernetes Engine, with schedulers, triggerers and workers running as GKE workloads, plus a managed database and a Cloud Storage bucket for DAGs, logs, plugins and data. Those components are useful when you do not want to operate them, but they also set the boundaries of your design. If you need specific executor behavior, a particular database version, or a fixed location for logs and data under your own controls, check those requirements against the environment setup before you commit. Self-hosting gives you those choices, and it also gives you the upgrade, backup and hardening work. Apache Airflow's production deployment guide is a useful checklist for that side of the decision.

How the two models differ on version, infrastructure and control

This table compares the two models on the factors that shape day-to-day ownership: supported Airflow major versions, the infrastructure each brings, executors, upgrades, authentication and where data and logs sit. Read the Composer column as what the published overview states, and treat any cell marked for confirmation as a question to put to Google before you commit.

FactorGoogle Cloud ComposerSelf-hosted Apache Airflow
Airflow major versionsComposer 3 and Composer 2 support Airflow 2. Composer 1 supports Airflow 1 and Airflow 2. The overview does not mention Airflow 3, so confirm Airflow 3 availability before planning.You choose the version. Airflow 3.3.2 is the current release, and Airflow 2 reached end of life on April 22, 2026, so it gets no fixes.
Underlying infrastructureEach environment is a self-contained Airflow deployment on Google Kubernetes Engine, with schedulers, triggerers and workers as GKE workloads.You build the cluster, scheduler, workers, and metadata database in your own Kubernetes or hosts. Airflow 3.3.2 lists Kubernetes 1.30 to 1.35 as supported.
Metadata databaseA managed database comes with each environment. Confirm its engine, version and backup behavior in the Composer documentation.You run PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or Innovation releases. MariaDB and SQLite are not supported for production use.
Storage for DAGs, logs and pluginsA Cloud Storage bucket holds DAGs, logs, plugins and data for each environment.You choose where DAGs, logs and plugins live, such as a Git-synced volume or your own object storage. Confirm the log backend you configure.
ExecutorsThe overview does not list executor options. Confirm with Google which executors Composer environments run and whether you can change them.You select the executor, such as LocalExecutor, CeleryExecutor or KubernetesExecutor. Since Airflow 2.10.0, several executors can run in one deployment.
Scheduler resilienceScheduler and worker placement is managed as GKE workloads. Confirm how scheduler scaling and failover are configured for your environment size.Airflow supports more than one scheduler at once, using database row-level locks, for performance and resiliency.
Upgrade responsibilityGoogle manages the Composer environment lifecycle. Confirm the upgrade path between Composer generations and the Airflow major version each one offers.Your team plans and runs each Airflow upgrade, including the move from Airflow 2 to Airflow 3, and keeps the database and Kubernetes versions in step.
Authentication and accessConfirm which identity options Composer exposes for the Airflow UI and API, and how they map to Google Cloud IAM.You choose the auth manager. Kubernetes authentication options, such as OpenID Connect, apply to cluster access, so plan both layers.
Data and log locationEnvironment data, logs and DAG files sit in Google Cloud in the bucket and managed services of the environment. Confirm the region and retention settings you need.Everything stays in the infrastructure you choose, including on-premises or air-gapped networks, which is the main reason regulated teams self-host.

Which Airflow major versions each Composer generation supports

The first thing to settle is which Airflow major version a Composer generation runs, because that decides whether a Composer environment can host the Airflow you plan to use. According to the Cloud Composer overview, Composer 3 and Composer 2 support Airflow 2, and Composer 1 supports Airflow 1 and Airflow 2. The same overview does not mention Airflow 3. For a team starting fresh, that gap matters more than any feature in the comparison, since Airflow 2 is no longer a supportable target: according to the Airflow version life cycle table, Airflow 2 reached end of life on April 22, 2026 and gets no fixes after that date. Its last patch release was 2.11.2.

Self-hosting Airflow changes this picture. The current release is Airflow 3.3.2, and its prerequisites list Python 3.10 to 3.14, PostgreSQL 14 to 18, MySQL 8.0, 8.4 and Innovation releases, and Kubernetes 1.30 to 1.35. You choose the Airflow version, the Python runtime and the metadata database yourself, and you can move to a supported Airflow 3 line on your own schedule. A Composer environment fixes those choices for you. Before you commit a DAG estate to Composer, ask Google in writing which Airflow 3 versions Composer will run, on what date, and whether any Composer generation has a published end of support date. Because the overview says Composer is evolving to become Managed Service for Apache Airflow, also confirm which product name and support terms will apply to your environment over its expected life.

If your estate is still on Airflow 2, treat the version question as a migration plan rather than a runtime setting. A Composer 2 or Composer 3 environment will keep you on Airflow 2, which is past end of life, so you will need a dated path to Airflow 3 whichever way you host. Write down the Airflow version each DAG depends on, list the providers and custom plugins in use, and ask Google whether an in-place upgrade to Airflow 3 is available for your Composer environment and what it requires. The answer decides whether the cost of moving to Airflow 3 falls on Google's roadmap or on your own project plan.

How data location and access change compliance work

With Cloud Composer, Google runs the Airflow control plane, database and bucket inside a managed environment, while with self-hosted Airflow your team owns every layer in its own account, data center or air-gapped network. The cards below cover where data sits, who controls access, how secrets are stored, and what evidence each model can produce for an assessor.

Where environment data physically lives

A Cloud Composer environment is a self-contained Airflow deployment on Google Kubernetes Engine, with its schedulers, triggerers and workers running as GKE workloads. DAGs, logs, plugins and data sit in a Cloud Storage bucket that belongs to the environment. Confirm with Google which regions and project settings apply to your environment, and map each of those resources to your data classification rules. With self-hosting, the equivalent resources sit wherever you deploy them, including air-gapped networks, so you decide the location and the boundary.

Shared responsibility for the control plane

Composer hands operation of the Kubernetes cluster and managed database to Google, which changes what your team must document as provider-managed and what it must evidence itself. In a self-hosted cluster, your team owns the kube-apiserver, etcd, node hardening and patching. Ask for the responsibility matrix in writing, including who patches the database and who applies Airflow upgrades. Assessors will ask for this split before they test any individual control.

Access control for users and workers

Kubernetes does not represent normal users as API objects and authenticates them with X.509 client certificates, bearer tokens such as OpenID Connect and JWT, webhook token authentication or an authenticating proxy. In a self-hosted cluster, your identity provider and API server settings define who can reach the cluster. Document how Airflow users, operators and service accounts map to these identities, and confirm how Composer maps your Google identities to environment access. Keep a list of who can change DAGs, which directly changes what runs in production.

Secrets stored in etcd by default

Kubernetes stores Secrets unencrypted in etcd by default, so a self-hosted Airflow deployment that keeps connection credentials as Kubernetes Secrets needs encryption at rest configured on the control plane. Restrict list and watch access with least-privilege RBAC, or source credentials from an external store through the Secrets Store CSI Driver. Ask Composer how it protects environment secrets and whether customer-managed keys are available for the bucket and database. Record which option you chose, since an assessor will check both the setting and the access rules.

Cluster-level access with Kubernetes RBAC

In a self-hosted deployment, Kubernetes RBAC controls which people and service accounts can read pods, Secrets, ConfigMaps and the Airflow namespace. Scope roles to the namespace that runs Airflow, avoid cluster-admin for routine operators, and review bindings on a fixed schedule. Pod Security Standards provide a baseline for workloads that run DAG tasks. Composer limits some of this access for you, so confirm which RBAC roles you can grant inside the environment and which remain with Google.

Evidence each model produces

Both models require the same kind of proof: configuration records, access reviews, backup and restore tests, and change logs for DAGs and cluster upgrades. Self-hosting means your team generates most of that evidence from its own tools, so plan the collection before the audit window opens. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, and you should ask any provider which of their documents an auditor can rely on.

Executors, schedulers and what you can change

The main thing self-hosting changes is the runtime. In a self-hosted deployment you choose the executor that runs your tasks: LocalExecutor, CeleryExecutor, KubernetesExecutor or EdgeExecutor. Airflow has supported several executors in one deployment since 2.10.0, so you can, for example, send most tasks to a long-running Celery pool and isolate heavy jobs in their own pods. Note that the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0, so any estate that relied on them needs a different split before its Airflow 3 upgrade. Confirm which executors Composer exposes to you in Google's documentation before you plan around any of them, because a managed environment may offer a narrower set than the open-source project.

Self-hosting also lets you run more than one scheduler. Airflow's scheduler documentation says multiple schedulers can run at the same time for performance and resiliency, coordinating through database row-level locks. That makes scheduler redundancy a configuration and capacity decision you own: you set the replica count, size the metadata database to handle the lock contention, and test failover. Confirm with Google how many scheduler instances a Composer environment runs and how it handles scheduler failure before comparing resilience.

Version pinning is the third lever. On your own Kubernetes cluster you choose the Airflow image, the Python and provider packages, and the Kubernetes minor version, and you decide when each one moves. Kubernetes maintains only the three most recent minor releases, and the version skew policy requires the API server to be upgraded one minor version at a time, so a self-hosted control plane needs a planned upgrade calendar. The trade-off is that you carry that calendar yourself. Ask Google which Airflow and Composer versions you can pin, how long each is supported, and what forced upgrades look like before you commit.

What your team runs in each model

In Cloud Composer, each environment runs on Google Kubernetes Engine, so the cluster sits inside the service you buy. Self-hosting moves the cluster into your team's calendar. Kubernetes ships about three releases a year on a cycle of roughly 14 weeks, and the project maintains only the three most recent minor releases, so upgrades recur every year.

  • Release tracking: Plan for about three minor releases a year. As of September 2026 the maintained lines were 1.37, 1.36 and 1.35, and each one has a fixed end-of-life date that sets your deadline.
  • Version skew: kube-apiserver moves one minor version at a time and goes first, followed by the controller manager, scheduler and kubelets. Kubelets may trail the API server by up to three minor versions.
  • Control plane layout: A kubeadm high availability build needs at least three control plane nodes and a TCP load balancer on port 6443, either stacked with etcd or with an external etcd cluster.
  • Add-on upgrades: Your team also upgrades the network plugin, ingress, storage drivers and the Airflow Helm release, and tests each one against the target Kubernetes version.

Confirm with Google the notice period for GKE upgrades that affect Composer environments, whether you can set a maintenance window, and how far ahead you can test an upcoming version.

What drives the cost of each option

The cost of Cloud Composer and self-hosted Apache Airflow is spread across different payees, so compare them line by line rather than as one number. With Composer, Google bills for the environment resources it provisions: the GKE workloads for schedulers, triggerers and workers, the managed database and the Cloud Storage bucket that holds DAGs, logs, plugins and data, as described in the Cloud Composer overview. With self-hosting, you pay your cloud provider or data center for compute, the Kubernetes control plane and node capacity, a PostgreSQL or MySQL metadata database, object or block storage, and the networking and load balancers around them. Staff time is the cost that does not appear on a cloud invoice, and it applies to both options.

Setup cost is mostly one-time: designing the environment, building the Helm or manifest deployment, wiring identity and secrets, and migrating existing DAGs. Recurring cost is the environment running around the clock, plus upgrades, patching, backups, monitoring and on-call. Self-hosting adds recurring work that Composer absorbs, including keeping the Kubernetes cluster within its support window and running etcd backups if you operate the control plane. The main levers that push the bill up or down are worker sizing and autoscaling, the number of environments (each one brings its own cluster, database and bucket), log retention, and whether you run a highly available metadata database. Ask for these inputs to be priced explicitly in any estimate.

To compare fairly, request a line-item estimate from Google's pricing pages for the Composer environment size you expect, including the Cloud Storage and database lines, and ask for the same worksheet to cover a self-hosted cluster on the same cloud. Then ask DeployOpen for a fixed-scope quote based on the written scope: the number of environments, the Airflow major version, the executor model, identity integration, backup and upgrade responsibilities, and the on-call hours you expect to need. Confirm which Airflow version each option will run, because an Airflow 2 estate has an end-of-life date that affects both the upgrade effort and the recurring support cost.

Moving between Composer and self-hosted Airflow

This sequence gets a team from its current Airflow estate to a supported target with its DAGs, connections and schedules intact, whichever direction the move runs. Both directions start by leaving Airflow 2, which is past end of life, so the inventory and version match steps carry most of the risk.

  1. Inventory DAGs, plugins and connections

    Your platform or data engineering team lists every DAG, custom plugin, provider package, variable, connection and pool in the current environment. For Composer, this includes exporting the DAGs, plugins and data folders from the Cloud Storage bucket, because the bucket holds them and must be copied out before the environment is retired. Connection secrets are recorded by name and owner, not copied in plain text. The step is done when the inventory is reviewed against the source code repository and every item has a named owner and a decision to move, rewrite or drop.

  2. Choose the target Airflow major version

    The platform owner selects the target version before any code moves. Airflow 2 reached end of life on April 22, 2026, so the target must be Airflow 3 whichever platform hosts it. Composer's overview lists Airflow 2 as the newest line it describes, so confirm with Google which Airflow 3 support, if any, is available for the Composer generation you use. The output is a written version decision naming the Airflow release, the Python version and the provider packages it requires.

  3. Match Airflow version and provider packages

    Engineers pin the target Airflow version and each provider package in a requirements file under version control, then rebuild a local or staging environment from that file. Composer and self-hosted installs can differ in preinstalled packages and environment variables, so compare the two lists line by line. The Airflow 3 upgrade guide lists breaking changes that affect imports and operators, and each one is fixed in the DAG code before testing. The step closes when a clean install of the pinned requirements imports every DAG without errors.

  4. Move the metadata database if applicable

    Moving between environments only carries history if the metadata database moves. A Composer environment uses a managed database that you cannot read directly, so confirm with Google whether and how its metadata can be exported before you rely on it. For self-hosted Airflow, Airflow requires PostgreSQL or MySQL in production and does not support MariaDB or SQLite for this role. Most teams decide to start the new metadata store empty and keep run history in the retired environment's logs. The step closes with a recorded decision and, if data is moved, a row count check on key tables.

  5. Test DAGs against the target version

    Testers run every DAG in a staging environment built on the target version, using copies of the connections with non-production credentials. They compare task outcomes, outputs and logs against the current production runs for the same input dates. Failing DAGs are fixed or retired, and each fix is reviewed in the same repository as the inventory. The step is complete when the staging results match the agreed expectations for each business-critical DAG and the sign-off owner has recorded the result.

  6. Cut over schedules to the new environment

    Operators pause schedules in the old environment and enable them in the new one in a planned window, so no DAG runs twice. For self-hosted targets, confirm that more than one scheduler is intentionally set up only where the design needs it, since Airflow supports multiple schedulers with database row-level locks. Downstream consumers are told the new source of truth. The cutover is done when the first scheduled runs in the new environment succeed and the old environment shows no active schedules.

  7. Retire the old environment and keep evidence

    Once the old environment has run in parallel for the period the team agreed, the platform owner deletes its DAG storage, database and cluster, and keeps the exported code, inventory, test results and change records. For Composer, confirm that the bucket and the managed database are removed so storage and compute charges stop. Auditors and internal reviewers will ask for evidence of this decision, so store it with the migration record. The step closes when the old environment is confirmed deleted and the migration record is filed.

Questions buyers ask before choosing a hosting model

Can Google Cloud Composer run Airflow 3 today?

The Cloud Composer overview does not mention Airflow 3. It says Composer 3 and Composer 2 support Airflow 2, and Composer 1 supports Airflow 1 and Airflow 2. Airflow 3 was first released on April 22, 2025, the current patch is 3.3.2, and Airflow 2 reached end of life on April 22, 2026, so it no longer receives fixes. Before you commit, ask Google in writing which Airflow versions each Composer generation supports and when Airflow 3 will be available on Composer. If your estate still runs Airflow 2, treat the upgrade path as a separate decision from the hosting choice, because it affects both options.

What does self-hosting Apache Airflow on Kubernetes actually require of a team?

It requires people who can run Kubernetes, a production metadata database and a regular upgrade routine, because the Airflow project supplies the software but not the platform around it.

  • Kubernetes: Airflow 3.3.2 lists Kubernetes 1.30 to 1.35. Kubernetes ships about three minor releases a year and maintains only the three most recent, so you plan a cluster upgrade every few months and move one minor version at a time.
  • Control plane: the kubeadm high availability guide describes at least three control plane nodes behind a TCP load balancer, and etcd should run with an odd number of members. Test your etcd snapshot backups and restores before go-live.
  • Database: use PostgreSQL 14 to 18 or MySQL 8.0 or 8.4. MariaDB is not supported, and SQLite is for testing only.
  • Secrets: Kubernetes stores Secrets unencrypted in etcd by default, so you must configure encryption at rest and least-privilege RBAC.
How long does a migration from Cloud Composer to self-hosted Airflow take?

No migration duration is supplied for this comparison, so ask any provider for a timeline tied to a written scope. The length depends on how many DAGs, plugins and connections you run, how many DAGs rely on Composer-specific settings, and how many secrets and variables must be recreated in your own environment. A move that also changes the Airflow major version adds testing for every DAG, because executors, configuration keys and provider packages can behave differently. Ask for the inventory method, the test plan for each environment, the cutover approach and the point at which parallel runs end.

Can our team run self-hosted Airflow alone, or do we need DeployOpen?

Either can work, and the answer depends on the Kubernetes, database and security skills your team already has. DeployOpen plans, implements, customizes, integrates, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. Ongoing support, upgrades and continuous testing are available as retainers. A team that already runs production Kubernetes may need design and hardening help only, while a team without that experience may want a retainer for upgrades and on-call support.

What should a quote for self-hosted Airflow include?

A quote should follow a written scope, because DeployOpen does not publish a price list. Ask that the scope name each environment (development, test and production), the Airflow version and executor model, the Kubernetes distribution and who owns the cluster, and whether the metadata database is self-run or managed. It should also state how Secrets are encrypted and rotated, how backups and restore tests are run, which upgrade and support retainer applies, what documentation and handover you receive, and which cloud accounts and access rights are needed. Separate the labor you pay for from the cloud resources you pay the provider directly, so you can compare the quote with a Composer bill.

How does compliance evidence differ between Cloud Composer and self-hosted Airflow?

With Cloud Composer, Google operates the GKE cluster, managed database and Cloud Storage bucket, so some controls and assurance reports come from Google. Which reports cover the service is not in the material this comparison relies on, so ask Google for its current documentation. With self-hosted Airflow, the evidence is yours: cluster configuration, RBAC settings, encryption at rest for Secrets, backup and restore test records, access logs and change records. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies do that work.

When is self-hosting Airflow on Kubernetes the cheaper option?

Self-hosting is more likely to be cheaper when you already run a Kubernetes platform with spare capacity, have staff who operate it, and can share one cluster across several workloads. Composer costs come from the managed environment and its underlying components, billed by Google. Self-hosting moves cost into cluster capacity, database and storage, plus engineer time for upgrades, backups and on-call work. No prices are supplied here, so build the comparison from your own regional rates. Ask Google for the price components of a Composer environment in your region, then estimate the yearly team hours for Kubernetes upgrades, which arrive about three times a year. Controllability, including executor choice and data location, can justify the cost even when the totals are close.

How DeployOpen scopes and runs self-hosted Airflow

DeployOpen plans, implements, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For Cloud Composer versus self-hosted Airflow, that means starting from your Airflow version, your executor choice and where your data and credentials must stay, then writing those constraints into the scope before any build work starts.

How we scope

  • Written scope fixes the Airflow major version, executors, metadata database and Kubernetes target before work starts.
  • Scope lists the Cloud Composer or self-hosted components in your environment, including any GKE clusters, buckets and identity integrations.
  • Engagements are fixed-scope, with deliverables and timelines agreed in writing; ongoing support is available as a retainer.

How we build

  • Build covers the scheduler, DAG processor, API server, workers and metadata database deployed into your own Kubernetes environment.
  • Secrets handling, encryption at rest and least-privilege access are configured and tested rather than assumed from defaults.
  • Backup and restore procedures for the metadata database and DAG storage are built and exercised before handover.

How we hand over

  • Handover documents the architecture, upgrade path and runbooks your platform team needs to operate the environment.
  • Documentation records the configuration decisions behind the Composer or self-hosted choice so the reasoning stays with your team.
  • Compliance-relevant evidence, such as access controls and backup records, is prepared for your auditors, who issue any formal findings.

How we support

  • Retainers cover ongoing support, Airflow upgrades and continuous security testing of the running environment.
  • Upgrade planning tracks the Airflow support window and Kubernetes version policy so end-of-life versions do not linger.
  • Ask any provider, including DeployOpen, to confirm in writing which upgrades, patch windows and response commitments the quote includes.

Sources and further reading

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com