Which database should back Airflow 3 in production?
For production, Airflow 3 needs PostgreSQL or MySQL as its metadata database. SQLite is out because the production guide warns that the default backend is for testing only and can lose data, and MariaDB is not a supported metadata database at all. Once that choice is narrowed, sizing follows from three things: how many schedulers, workers and API server replicas will connect, how many DAG runs and task instances the business will generate, and how long run history must stay queryable. Airflow is released under the Apache License 2.0 (Apache-2.0) with no paid edition from the project itself, so the database is one of the main infrastructure decisions you make, and it is your team's to own.
This decision belongs before the first production deployment, and again before any migration from Airflow 2. Airflow 2 reached end of life on April 22, 2026, so teams still running it will need to upgrade, and the upgrade path runs through the same metadata database. Check the Airflow 3.3.2 prerequisites for the exact PostgreSQL range (14 to 18) and MySQL range (8.0, 8.4 and Innovation releases) before you commit, and confirm the version your platform team can actually patch. Also confirm whether your organization already runs a PostgreSQL or MySQL service with backups and high availability that meet your policy, since that usually decides between a managed and self-run database. The Airflow prerequisites page is the starting point for this check.
Write the decision down as a short record: engine, major version, hosting model, owner, backup and restore targets, and the upgrade window. Ask any partner or vendor for the same record in their quote, so connection limits, growth from run history and cleanup policy are scoped with the database rather than discovered after launch. Teams that already run PostgreSQL well often pick it by default, but that is a reason to confirm the match, not a substitute for checking the supported versions and your own operating skills.
PostgreSQL and MySQL options compared for Airflow
This table compares PostgreSQL and MySQL against the Airflow 3.3.2 prerequisites, their support windows, and the high availability options each offers. Read each row as a decision point: the version support dates and the HA tooling you must run yourself usually decide the choice more than raw feature lists.
| Decision point | PostgreSQL 14 to 18 | MySQL 8.0, 8.4 and Innovation |
|---|---|---|
| Airflow 3.3.2 support | Listed as supported: versions 14 to 18 | Listed as supported: 8.0, 8.4 and Innovation releases |
| Support window | 14 ends November 12, 2026; 18 runs to November 14, 2030 (PostgreSQL versioning policy) | 8.4 LTS and 8.0 follow Oracle Lifetime Support; Innovation releases are supported only until the next Innovation release |
| Licence | PostgreSQL License, a permissive open source license similar to BSD or MIT | MySQL Community Edition under the GPL; Enterprise features are a paid, optional add-on |
| Built-in HA | Hot standbys with synchronous or asynchronous replication; no built-in failover detection | InnoDB Cluster and Group Replication with automated failover in Community Edition |
| Failover tooling | External tools required for automatic failover; confirm your choice and test it | MySQL Router and MySQL Shell included in Community Edition |
| Backups | Built-in continuous archiving and point-in-time recovery; you own the tooling and restore tests | Enterprise Backup for hot backups is optional and paid; Community tooling needs your own process |
| Major upgrades | Logical replication, pg_upgrade or dump and restore; logical replication allows very short downtime | Confirm the upgrade path between 8.0, 8.4 and Innovation with your DBA before choosing |
| Best fit | Teams that already run PostgreSQL, want a permissive license and can operate or buy HA tooling | Teams with existing MySQL operations skills or Oracle-based standards who accept LTS and Innovation tracking |
Why SQLite and MariaDB rule out production
The default SQLite backend is for testing only, and the Airflow production deployment guide warns that running it in production can lead to data loss. The metadata database holds DAG runs, task instances, connections and scheduler state, so losing it means losing the record of what ran and what still needs to run. For a production Airflow 3 install, the choice narrows to PostgreSQL or MySQL, and the first step is to remove SQLite from every environment that will carry real schedules, including any shared staging instance that other teams rely on. Confirm this in the first install checklist rather than assuming a later migration will catch it.
MariaDB is not a supported metadata database according to the Airflow prerequisites, so a team with an existing MariaDB estate cannot simply point Airflow at it and expect vendor support for the combination. The practical consequence is that Airflow needs its own supported engine, either a PostgreSQL version in the supported range listed for Airflow 3.3.2 or a MySQL version from the listed 8.0, 8.4 and Innovation releases. That adds a database to build, back up, patch and monitor, along with its own credentials, network rules and recovery procedure. Ask your platform team which engine they can operate well, confirm the version against the prerequisites page before procurement, and include the new instance in your capacity and recovery plans from the start. Apache Airflow prerequisites lists the current supported combinations.
Managed or self-run: which fits a regulated estate?
A managed PostgreSQL or MySQL service moves most routine database operations to the cloud provider. The provider typically handles the underlying host, many patching tasks and backup scheduling, and it may offer a standby that can take over. Your team still owns the schema, the Airflow connection settings, connection counts, parameter choices, user access and the restore procedure you rely on. Your application data remains inside the provider's control plane, so the data location and provider access model need review.
- Patching: Confirm which minor updates are applied automatically, which need a maintenance window, and how major version upgrades are scheduled.
- Backups: Confirm retention length, whether point-in-time recovery is included, and whether backups can be copied to a second region or account.
- Failover: Confirm whether failover is automatic, how long clients are disconnected, and whether Airflow reconnects without manual steps.
- Access: Confirm who at the provider can reach the instance, what logging exists for provider staff access, and whether keys can be customer-managed.
- Supported versions: Confirm the service offers a version within the Airflow 3.3.2 prerequisite ranges.
Before you commit, ask the provider for its written service terms, its audit report availability and its exact recovery objectives so that your evidence matches what you are actually buying.
A self-run database on your own servers or virtual machines gives your team full control over patch timing, storage, network paths and recovery. It also means your team carries every operational task. PostgreSQL does not detect a failed primary or notify a standby by itself, so automatic failover requires external tools that your team selects, configures and tests. Hot standbys and synchronous or asynchronous replication are built into PostgreSQL, but the failover decision is not.
- Patching: Your team applies minor releases by replacing executables and restarting, and plans major upgrades with pg_upgrade, dump and restore, or logical replication.
- Backups: Your team sets up continuous archiving and point-in-time recovery, stores copies off the primary host and runs restore tests on a schedule.
- Failover: Your team selects an external failover tool, defines fencing rules to prevent split brain, and documents how Airflow components are repointed.
- Monitoring: Your team owns alerting on replication lag, disk growth and connection counts.
- Staffing: Confirm who is on call for database incidents and who can approve emergency changes.
Before you start, confirm that your team or a partner can run restore drills and failover tests on a schedule, and that the chosen failover tooling is supported by your own security and change policies.
Running PostgreSQL on Kubernetes can use a database operator that manages replicas, backups and failover, or an external database instance that the cluster connects to. The operator path moves some failover logic into the cluster, but it adds another controller with its own upgrade cycle, permissions and failure modes. Using an external instance keeps the database outside the cluster lifecycle, which suits teams that want database changes separate from cluster upgrades. MySQL on Kubernetes follows the same split, with its own operator options.
- Patching: Confirm whether the operator or your team applies minor versions, and how a rolling restart affects Airflow components.
- Backups: Confirm where the operator writes backups, how restores are triggered, and whether backups are encrypted at rest.
- Failover: Confirm whether the operator promotes a replica automatically, how it fences the old primary, and how Airflow reconnects.
- Permissions: Confirm which service accounts can read database secrets and which roles can change the database custom resource.
- Storage: Confirm the storage class supports the write latency your scheduler and API server need.
Before you choose, confirm the operator's supported Kubernetes versions against the Airflow prerequisite range of Kubernetes 1.30 to 1.35 and get the operator's upgrade and recovery procedures in writing.
How to size and provision the metadata database
This sequence takes the metadata database from a version check to a tested, backed-up instance that Airflow components can reach within their connection limits. Each step closes with a check so the database is verified before any production DAG depends on it.
Confirm engine and version against prerequisites
Your platform or data team checks the Airflow 3.3.2 prerequisites before choosing anything else. The list gives PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases as supported metadata engines, and states that MariaDB is not supported and SQLite should not be used in production. Confirm that the engine version you intend to run is inside that range and that your Python and Kubernetes versions match the same release. The step is done when the target engine version is written into the design record with its support end date from the PostgreSQL versioning policy or the MySQL release track you selected.
Select the engine and release track
Choose between PostgreSQL and MySQL based on what your organization already operates, runs backups for, and can staff. PostgreSQL is released under the PostgreSQL License, and its supported majors are listed with end dates in the versioning policy, so you can pick a major that stays supported through your planned Airflow lifecycle. MySQL offers an LTS track and an Innovation track, and Innovation releases are supported only until the next one, so an Innovation build is a poor fit for a long-lived metadata store. The step closes with a documented decision that names the engine, the major or LTS version, and the owner of upgrades.
Size compute and storage from component counts
Count the schedulers, API server replicas, DAG processor instances, triggerers and workers that will connect to the database, then size the instance from that count. Airflow supports more than one scheduler at a time through row-level locks, so scheduler count directly affects concurrent database load. The prerequisites recommend at least 4 GB of RAM for the Airflow components, which is a floor for the components rather than a sizing rule for the database itself. Ask the database team to size memory, CPU and disk from your expected run history volume and to test under a realistic scheduler load. The step is done when a load test on the chosen size shows scheduler loop and API latency inside the targets you set.
Plan connection limits and pooling
Estimate the maximum number of simultaneous connections from every Airflow component and worker, because connection counts grow with the number of workers and schedulers. For Kubernetes, the Helm chart production guide recommends PgBouncer to reduce connections, and the PgBouncer project documents the pooling modes to evaluate. For VM deployments, decide whether to run PgBouncer or use an external pooler with the same effect. Record the pooled connection limit, the maximum database connections the server allows, and the headroom kept for administrative sessions. The step closes when the connection budget adds up to less than the server limit with margin, and the pooler metrics are assigned to an owner for monitoring.
Configure the external database in the Helm chart
For Kubernetes deployments, point the Helm chart at an external metadata database rather than the embedded Postgres, which the production guide describes as suitable for testing only. Store the connection details in a Kubernetes Secret, not in values.yaml, and confirm the chart is set to use that external connection. The database owner should provision a dedicated role and database for Airflow with only the privileges the chart needs. The step is done when a Helm render or dry run shows the external connection in place and no embedded database resources are created.
Run a test migration and database check
Before the production instance is used, run the Airflow database migration against a non-production copy of the target database and then run airflow db check. Confirm that the migration completes without errors and that the scheduler, API server and workers can connect through the pooler. Keep the migration log with the design record so the change can be reproduced during the next Airflow upgrade. The step closes when airflow db check reports a healthy connection and the migration output shows no failed statements.
Set cleanup and verify backups and restore
Plan run history growth before go-live. The Helm chart supports a databaseCleanup CronJob with a retentionDays setting, and the best practices note that old metadata can be purged with airflow db clean. Set the retention period to match your audit and troubleshooting needs, and confirm that backups cover the metadata database on the schedule your recovery requirements set. The PostgreSQL continuous archiving documentation describes point-in-time recovery, which is one option to evaluate against your recovery targets. The step is done when a restore into a separate instance succeeds and a sample of recent DAG run history is readable in the restored copy.
Document the upgrade and failover path
Record how the database will be upgraded and how failover is handled, since neither is automatic in a default setup. PostgreSQL minor releases replace executables and restart without changing the storage format, while major upgrades use dump and restore, pg_upgrade, or logical replication. PostgreSQL does not detect primary failure on its own, so automatic failover requires an external tool that your team selects and tests. For MySQL, confirm which high availability option you will run before committing to it. The step closes with a runbook that names the owner, the tested procedure, and the date of the last failover or restore drill.
How connection counts and run history drive database load
Connection demand grows with every Airflow component that talks to the metadata database, not only with the number of DAGs. Airflow supports running more than one scheduler at the same time for performance and resiliency, coordinating through database row-level locks. Each additional scheduler adds its own pool of connections, and so do the API server, DAG processor, triggerer and workers. When you plan a scheduler count, add the connections each component opens at its configured pool size, then compare the total with the connection limit of your PostgreSQL or MySQL instance, including any headroom your managed service reserves for administration. Confirm these numbers against your deployment's actual settings and your database provider's documented connection ceiling before you pick an instance size.
Run history is the second driver, and it grows continuously. Every DAG run, task instance, log reference and related record is written to the metadata database and kept until something removes it. Tables that look small at launch can reach millions of rows once schedules run every few minutes across many teams, which slows the scheduler's queries and makes backups and restores longer. The Airflow best practices note that old metadata can be purged with airflow db clean, so decide your retention window before go-live rather than after the first slowdown. Ask your team which audit or recovery needs require history to be kept, and where longer history should live instead of the operational database. For a Helm-based deployment, the chart's databaseCleanup CronJob with retentionDays schedules this purge, and PgBouncer, which the Helm chart production guide recommends for reducing database connections, should be sized alongside it.
Keep parse-time database access out of DAG files, because it multiplies load across the whole fleet. The Airflow best practices say top-level DAG code should not perform database access, heavy computation or network calls, and that Variables should not be read at parse time. The DAG processor parses files repeatedly, so a query placed at module level runs on every parse by every scheduler and processor, turning a small code choice into constant connection churn. Review DAG code for this pattern during code review and in CI, move reads into task execution, and treat any parse-time query as a defect. This check costs little and often explains connection spikes that look like a database sizing problem.
Security and recovery controls for the metadata database
The metadata database holds the state of every DAG run, task instance and connection reference Airflow uses, so its access, encryption and recovery controls matter as much as the database engine you pick. The cards below cover the controls PostgreSQL documents and the optional controls MySQL offers, along with the one secret that must never sit in a values file.
Authentication methods and password hashing
PostgreSQL supports password, GSSAPI, SSPI, LDAP, RADIUS, certificate, PAM and OAuth authentication, and its PostgreSQL 18 release deprecated MD5 password authentication in favor of SCRAM. Check the password_encryption setting and the pg_hba.conf rules on the Airflow database, and confirm that the Airflow service account can authenticate only from the hosts that run the API server, scheduler, DAG processor and triggerer. Ask your identity team which method your directory supports before the first deployment.
Certificates and encryption in transit
Every Airflow component connects to the metadata database, so traffic between them crosses your network. Decide whether connections must use TLS, how server certificates are issued and rotated, and whether clients verify the server certificate or only encrypt the session. Certificate authentication can also replace shared passwords for the Airflow service account, which narrows who can log in and simplifies revocation.
Network placement and connection pooling
Keep the database on a private subnet reachable only from the Airflow components and the administrators who maintain it. A pooler such as PgBouncer sits between the components and the database, so the pooler's own listener, its auth file and its access list become part of the trust boundary. Confirm which hosts can reach the pooler and which can reach PostgreSQL directly.
Backups and point-in-time recovery
Regular base backups alone leave a gap between the last backup and a failure, and run history recorded in that gap is lost. PostgreSQL continuous archiving with WAL files allows recovery to a chosen moment, which is useful after an accidental airflow db clean or a bad migration. Schedule restore tests against a separate instance and record how long a restore takes, because that number, not the backup job's success status, shows whether recovery works.
Synchronous and asynchronous replication
A hot standby serves read-only queries and can take over after a primary failure, but PostgreSQL does not detect failures or promote a standby by itself, so an external tool must do that. Synchronous replication makes each commit wait for a standby, which protects scheduler state at the cost of latency. Asynchronous replication is faster but can lose the most recent commits on failover. Decide which trade-off you accept before choosing the topology.
Optional MySQL Enterprise security features
MySQL Community Edition is free under the GPL license and includes replication, InnoDB Cluster and Group Replication with automated failover. MySQL Enterprise Edition adds Transparent Data Encryption, Enterprise Audit, Enterprise Backup, external authentication modules and 24x7 Oracle support. Choose Enterprise only if your controls require encryption at rest managed by the database or database-level audit trails, and ask Oracle for the license terms in writing before committing.
Questions teams ask before sizing the metadata database
Should we choose PostgreSQL or MySQL for the Airflow metadata database?
For a new deployment, PostgreSQL is the more common choice, but both engines are supported by Airflow 3.3.2, and the decision usually rests on the engines your team already operates. The prerequisites list PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, while MariaDB is not supported and SQLite should not be used in production. Compare the support windows for each engine, the replication and failover tooling your staff knows, and whether your managed-service catalog offers the version you need. Confirm the exact combination against the Airflow prerequisites before you commit.
How do database engine upgrades fit with Airflow release timing?
Treat the database engine and the Airflow release as separate upgrade tracks, and sequence them deliberately. Airflow publishes which engine versions each release supports, so an Airflow upgrade may require a newer engine, and an engine upgrade may require a supported Airflow version. PostgreSQL minor releases do not change the storage format, so they are a routine restart, while major upgrades need a dump and restore, pg_upgrade, or logical replication. Plan the engine major upgrade in a window that does not overlap with an Airflow migration, and rehearse both in a staging copy of the metadata database.
What drives the cost of running the metadata database?
The community engines are free to download. PostgreSQL uses a liberal open source license, and MySQL Community Edition is available under the GPL, so the cost sits in the work around the database. Those costs include high availability design, backups and point-in-time recovery, connection pooling, monitoring, patching, and the staff hours to run them. If you choose a managed database service, the provider charges fees that depend on instance size, storage, replicas and backup retention, so ask for those fees in writing. MySQL Enterprise Edition adds features such as hot backups and Transparent Data Encryption under a commercial subscription, and that cost must be included if you need those features.
What does the database work need from our internal team?
Your team needs someone who owns the database's availability, access and recovery objectives, plus people who can answer for the network path between Airflow components and the database. Airflow schedulers, workers, the API server and the DAG processor all open connections, so your team must agree on connection limits and who changes them. Expect to supply a non-production environment for rehearsals, a backup target you control, identity arrangements for database accounts, and a change window for patching. If your organization runs a central database platform team, decide early whether that team operates the Airflow database or only provides the instance.
Can we size and run the metadata database ourselves?
Yes, many teams do, and the first step is to measure before buying hardware. Sizing depends on how many DAGs and task instances you run, how long you keep run history, how many schedulers you operate, and how many connections each component opens. Airflow documentation says the scheduler can run in multiple instances using database row-level locks, which makes connection counts and lock contention part of the sizing exercise. The Helm chart guide recommends PgBouncer to reduce connections and a cleanup job with a retention setting, so include those in your plan. A partner can help with the design, but the sizing must be validated against your own workload.
What should we ask a partner to include in a written scope?
Ask the partner to state the engine and version, the deployment topology, and whether high availability uses synchronous or asynchronous replication with a named failover mechanism. The scope should cover backup method, restore testing, point-in-time recovery, connection pooling, monitoring, and a written retention and cleanup policy for run history. Ask which parts are one-time work and which recur, what acceptance tests prove the database recovers, and what the handover includes. Compliance evidence should be explicit, since DeployOpen prepares controls and evidence for assessments but does not issue audit reports or certifications. Scopes are fixed in writing before work starts, so get every deliverable and exclusion listed there.
How long should we keep Airflow run history in the metadata database?
Retention is a business and audit decision first, and a technical setting second. Keep enough history to meet the audit and troubleshooting needs your teams have documented, then remove older rows on a schedule. Airflow provides the airflow db clean command for purging old metadata, and the Helm chart supports a database cleanup job with a retention setting. Test cleanup in staging before enabling it in production, because removing records you still need for evidence is hard to reverse. Keep an archive of audit-relevant logs in a location outside the metadata database if your compliance requirements call for longer retention.
Limits to confirm in your environment before go-live
The version and platform limits below change over time and differ between your Airflow patch, your database vendor and your hosting setup. Confirm each one against the current vendor documentation and your own tests before you commit to a database choice.
Database versions for your Airflow patch
Support tables list database versions per Airflow release, and the list can change between patch releases. Check the prerequisites page for the exact Airflow patch you will deploy, then confirm that the PostgreSQL or MySQL version you plan to run appears in that list. Record the Airflow patch and database versions together in your change record, so an upgrade of either one triggers a fresh check.
PostgreSQL end-of-life for your major
Each PostgreSQL major version stops receiving fixes on a published date, and that date may fall within the life of your system. Pick a major version whose support window covers your planned go-live and the period you expect to run it, then schedule the upgrade before the end-of-life date. Confirm the dates on the versioning policy page, since they are the authority on support windows.
MySQL track and support windows
MySQL offers an LTS track and an Innovation track, and their support periods differ. Innovation releases are supported only until the next Innovation release, so a database on that track will need a version change on a short cycle. Confirm which track your Airflow patch supports, then check the Oracle Lifetime Support Policy dates for that series before you commit to it.
Managed service version and extensions
A managed database may lag the community release or omit extensions and settings that Airflow or your monitoring tools expect. Ask the provider which major versions it offers, whether it reaches the end-of-life dates you planned around, and which extensions and parameters it exposes. Check that the managed service accepts the connection options and TLS settings your Airflow deployment will use.
Failover tooling you have tested
PostgreSQL does not detect a failed primary or promote a standby on its own, so automatic failover depends on tools you select and operate. MySQL Group Replication includes automated failover, but its behavior depends on your topology and configuration. Run a failover test against your own cluster, measure how long Airflow takes to reconnect, and confirm that schedulers and API servers recover without manual steps.
Connection limits of your provider
Schedulers, workers, API servers and the DAG processor each open database connections, and their count grows with concurrency. Ask your provider for the maximum connections on your instance size, and whether it offers a pooler. Size the pool against your Airflow concurrency settings, and load test the pool before go-live, since a limit hit at peak load can stop scheduling.
How the scope and sizing decision is documented
DeployOpen plans and implements the metadata database layer inside your own infrastructure, whether that is a cloud account, a data center or an air-gapped network. The database choice, the engine version, the hosting model and the sizing assumptions are recorded in the written scope before work starts, so the decision can be reviewed by your architects, security team and the people who will carry the pager. The record covers the chosen PostgreSQL or MySQL major version, the Airflow version it must support, the connection budget across the API server, scheduler, DAG processor, triggerer and workers, the expected growth from run history, and the retention window agreed for cleanup. Each assumption is stated so it can be revisited when Airflow or the database engine moves to a new release.
Sizing assumptions are only useful if they are tested. Before cutover, DeployOpen runs the restore path against a copy of the database, confirms that point-in-time recovery meets the recovery targets your organization sets, and rehearses the Airflow database migration on a staging copy so that the upgrade steps and rollback steps are known in advance. The results, including measured connection counts and table growth, go back into the scope document. Ask any partner to show the restore test output and the migration log, not only a diagram. Ask your own team to name the owner of the backup policy and the person who approves a change to the cleanup window.
The written scope also records what is excluded. Managed database services, hardware and licensing costs are priced separately, and the quote should list who pays for each line, what is one time and what recurs. Confirm with the database vendor or your cloud provider which features your chosen service supports, such as replicas, automated failover or backup retention, rather than assuming them. Any statement about support windows should be checked against the current vendor lifecycle pages on the day the scope is signed.
How DeployOpen scopes and supports the Airflow database
DeployOpen plans, builds, documents and supports the Airflow metadata database inside your own infrastructure, from the choice between PostgreSQL and MySQL through sizing, connection planning and cleanup. Each engagement starts from a written scope that sets which database engine and version, which hosting model and which operational controls apply to your environment.
How we scope
- Confirm the Airflow release line, Python version and Kubernetes or VM target before choosing the database engine and version.
- Record whether the database will be a managed service or self-run, and who holds backup and patching duties.
- Agree in writing the connection budget, expected run-history volume, retention period and cleanup schedule for the database.
How we build
- Provision PostgreSQL or MySQL in your environment with the supported version range confirmed against the Airflow prerequisites.
- Configure connection pooling, external database settings and scheduler access so the metadata store matches the agreed sizing.
- Test backups, restores, the cleanup job and a major version upgrade path before the database carries production DAG runs.
How we hand over
- Deliver written runbooks covering sizing assumptions, connection limits, cleanup settings and restore steps.
- Document the configuration values and secret locations so your team can review changes without reverse engineering them.
- Walk your operations and database staff through monitoring signals for growth, locks and connection saturation.
How we support
- Engagements are fixed-scope, and ongoing support, upgrades and testing are available as a retainer.
- Review database growth and cleanup results against the agreed scope when Airflow or database versions change.
- Plan supported version moves for Airflow and the database, confirming each step against the vendor life cycle pages.
