Moving from Google Cloud Composer to self-hosted Apache Airflow

The number of DAGs sets the bulk of the migration work, and the number of distinct provider packages sets how much compatibility checking the team needs. Each DAG has to be inventoried, assigned an owner, and classified as move as is, rewrite, or retire. Provider packages matter because each Google provider import, operator and hook must resolve in the target image, and provider versions must match the Airflow release you choose. Ask the partner to group DAGs by shared imports and by the Google services they call, since a small number of shared patterns usually covers many DAGs. Count DAGs that are dormant or duplicated separately, because retiring them costs less than porting them.

Often yes, and the rework is driven by which Airflow 2 features the DAGs use. Airflow 3 requires that you be on Airflow 2.7 or later before upgrading, and the upgrade guide recommends moving to the latest 2.x first and checking DAG code with the Ruff AIR301 and AIR302 rules. Task code can no longer use Airflow database sessions or models directly and must use the REST API or Task SDK instead. SubDAGs have been removed in favor of TaskGroups, and the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor are no longer supported as of Airflow 3.0.0. DAGs that read metadata inside tasks or use SubDAGs are the most expensive to change, so flag them early.

The count of Google connections and secrets determines how much configuration work and access review the move involves. Each Airflow connection that points to a Google project, service account or BigQuery dataset must be recreated in the new environment, and its credentials must be stored in a secrets backend. The Google provider includes a Secret Manager backend (CloudSecretManagerBackend), so teams that already keep secrets in Google Secret Manager can often keep them there, which means the new Airflow needs network access and a workload identity that can read them. Inventory connections by service account, note which ones are shared across DAGs, and confirm with your security team which identities may be retained.

They add effort when DAGs, plugins or Python dependencies are delivered through Cloud Storage or baked into a Composer image. Composer environments typically sync the DAG folder from a GCS bucket, so the self-hosted team must choose a delivery method such as git-sync, a shared volume, or a build pipeline that packages DAGs into the image. Custom plugins and requirements installed into a Composer image need to be rebuilt on the official Airflow base image and tested against the same provider versions. Ask for a list of every plugin, custom package and environment variable, because each one becomes a build artifact your team must own. The Airflow container and Kubernetes guidance supports this approach, but the delivery design is a decision you must make.

The Composer generation and image version set the Airflow release you are moving from and therefore the upgrade path. Google documents Managed Service for Apache Airflow (formerly Cloud Composer) versions on its versions page, where Gen 3 environments offer Airflow 3.1, 3.2 and 3.3 builds and newer builds follow a 12-month support period. Record the exact Airflow version, provider versions and Python version of each environment, since the self-hosted target must support the same Python range. If you are on an Airflow 2 image, the upgrade to Airflow 3 is a separate project that should be scheduled before the platform move, not during it.

After cutover your team operates every component that Google previously managed. In Airflow 3 that includes the scheduler, the DAG processor, which runs as a separate component, and the API server, which replaces the webserver. You also run the metadata database, which must be PostgreSQL or MySQL in production, since SQLite is for testing only. Plan for database backups and restore tests, upgrades, log storage, monitoring, and executor capacity such as Kubernetes nodes or Celery workers. Multiple schedulers are supported for resiliency, but each added component adds operational work. Ask for a written runbook covering who patches, who restores and how often.

A capable platform team can do the work, but the question is whether it has the time and skills for the parts that carry risk. The work includes inventorying DAGs, porting code to Airflow 3 rules, rebuilding images, recreating Google connections, designing a delivery method, and running a parallel period before cutover. Each of these tasks has a clear output, which makes them suitable for a fixed-scope engagement with written deliverables and an agreed timeline. Consider whether your team already runs Kubernetes, PostgreSQL backups and secrets management, because those skills determine how much of the ongoing operation you can absorb.

Ask every bidder to price against the same written inventory. Provide the DAG count, provider list, Composer version, connection and secret inventory, plugin and image list, and the target platform you have chosen. Ask the quote to separate one-time work (inventory, porting, image build, connection recreation, parallel run and cutover) from recurring work (upgrades, backups, monitoring and support), and to state what is excluded, such as Kubernetes cluster provisioning or Google-side decommissioning. DeployOpen does not publish a price list. A quote follows from the written scope, which is agreed before work starts.

The cost is the sum of the people and infrastructure needed to run the move and then run Airflow yourself. Migration effort is one-time: inventorying DAGs, rebuilding the Airflow image, replacing Composer-specific settings, moving secrets, testing and cutover. Running costs recur: compute for the scheduler, DAG processor, API server and workers, a PostgreSQL or MySQL metadata database, log storage, monitoring and patching. The main drivers are the number of DAGs, how many Google-specific operators and connections they use, how many environments exist, and whether you already run Kubernetes. Ask any quote to separate one-time migration work from recurring operations and to list which infrastructure is billed to your cloud account.

Duration depends on the size and condition of the DAG estate, not on the Airflow version alone. Teams usually see longer timelines when DAGs depend on Composer-only environment variables, when many tasks use Google operators that need new connections, or when the platform team has not yet built the Kubernetes or VM foundation. A written scope should give phases for inventory, foundation build, DAG porting, parallel run and cutover, with a stated duration for each. No general figure is given here because the supplied sources do not establish one. Ask for a timeline tied to your DAG count and dependency list.

Yes, many teams do, provided they have engineers who can run Airflow in production. The team needs skills in container images, the metadata database, scheduler operations, secrets management and Google service accounts. The hardest parts are usually not the DAG code but the operating model: upgrades, backups, log retention, access control and on-call. If those skills are thin, a fixed-scope engagement can cover the foundation build, the DAG port and the cutover, and leave your team with runbooks. Ask for the handover deliverables in writing before work starts.

Some DAGs need changes, and the Airflow 3 upgrade guide is the place to start. Airflow 2.7 or later is required before upgrading, and the guide recommends moving to the latest 2.x first and checking DAG code with the Ruff AIR301 and AIR302 rules. Airflow 3 task code cannot use Airflow database sessions or models directly and must use the REST API or Task SDK, and SubDAGs were removed in favor of TaskGroups. Composer image versions also matter: Google's versions page lists Gen 3 builds on Airflow 3.1, 3.2 and 3.3, so confirm which build your environments run before choosing the target release.

Secrets move by changing where Airflow reads them, not by copying values into DAG files. The Google provider includes the CloudSecretManagerBackend, so a self-hosted Airflow can keep reading secrets from GCP Secret Manager if the workers use a service account with the right access. Connections defined in the Composer UI or environment variables must be recreated in your deployment, and each one should be checked for scope. Keep the Secret Manager path in the design, then confirm that your workers, not only the scheduler, can reach it. If you prefer a different store, plan that change as a separate step.

Double runs happen when Composer and the new Airflow both schedule the same DAG against the same data. Avoid them by pausing each DAG in Composer before unpausing it in self-hosted Airflow, and by moving one DAG group at a time with a recorded owner. Compare outputs in a parallel run where the new environment writes to a staging location or is read only. Keep the Composer DAG folder in place as a rollback path until the new schedule has produced verified results. Also check the catchup setting, because an unpaused DAG can backfill missed intervals and repeat work.

Airflow can run on Kubernetes, on virtual machines or in other infrastructure you control. The official Helm chart uses the official Docker image, and the production guide recommends container images for Kubernetes. Whichever target you choose, the metadata database should be PostgreSQL or MySQL, since SQLite is for testing only and MariaDB is not supported. Plan for multiple schedulers if you need resiliency, and for a DAG delivery path that replaces the GCS folder. Confirm the Kubernetes version support window before committing to a cluster, because the project maintains only its three most recent minor releases.