Why airflow db clean matters for a self-hosted Airflow database
airflow db clean is the built-in way to purge old metadata from a self-hosted Airflow database, and the retention period you set with it should be decided against your audit needs, not against disk space alone. Teams usually need it when the PostgreSQL or MySQL metadata database keeps growing with every DAG run, task instance, log row and event, and the growth shows up as a slower UI, a scheduler that takes longer per loop and upgrade migrations that run for hours. The Airflow best practices state that old metadata can be purged with airflow db clean, so the command is the vendor-supported starting point rather than a custom script you have to maintain.
The decision starts with what your organization must be able to answer later. Run history in the UI, audit log rows in the log table, and task logs stored in the database are different kinds of evidence, and a cleanup run removes the older rows in whichever tables you target. Before you choose a retention value, ask your compliance owner and internal audit how far back they need to show who triggered a DAG, who changed a connection or variable, and what ran on a given date. Confirm with your auditors whether the audit log is part of the evidence set, and whether an archive of the removed data is required. Those answers, not the current database size, should set the retention window.
Cleanup options and when each fits
This table compares three ways to control metadata database growth: manual runs of airflow db clean, a scheduled databaseCleanup CronJob in the Helm chart, and longer audit retention kept outside routine cleanup. Read each row against your own retention obligations and your installed Airflow version, since table coverage and dry-run behavior vary. The main takeaway is that the cheapest option is rarely the one that answers an audit question.
| Option | When it fits | Who runs it | What to confirm first |
|---|---|---|---|
| Manual airflow db clean | Small or test environments, one-off clean-up before an upgrade, or teams that want each deletion reviewed by hand | An Airflow administrator with access to the scheduler or a maintenance shell | Dry-run support and the list of tables covered in your installed version, and that a fresh database backup exists |
| Scheduled databaseCleanup CronJob with retentionDays | Kubernetes deployments using the official Helm chart that want steady, repeatable pruning of old metadata | The platform team, through Helm values and the cluster's job scheduling and logs | The retentionDays value you set, the cleanup schedule, which tables your chart version prunes, and where job output is kept |
| Longer audit retention in the log table or exported elsewhere | Organizations with audit or investigation needs for user actions beyond the general metadata retention period | Security or compliance owners define the period, and platform engineers set up the export or exclusion | Which audit events the log table stores, how exports are made and protected, and whether the cleanup job leaves the log table untouched |
Setting retention that still answers audit questions
Retention decides how far back the Airflow audit history stays visible, so agree it with compliance and security owners before any automated purge runs. Airflow's audit log table records user actions from the UI, the REST API and the CLI, plus system events such as task state changes. Each entry carries a UTC timestamp, the event, the owner and, where relevant, DAG, task and run IDs. airflow db clean removes rows older than the cutoff you set, and those rows then disappear from the UI, from /eventLogs and from any SQL you run against the log table. A purge is not reversible from the platform itself, so the retention window is effectively the length of your evidence record.
The trade-off is straightforward to state and harder to settle. A shorter window keeps the metadata database smaller and migrations faster, which matters before an upgrade. A longer window keeps more evidence for access reviews, change management and incident reconstruction, but it grows the database and the cost of backups. Regulated teams usually need to answer questions such as who triggered a production DAG last quarter, or when a connection was changed. Ask the compliance owner which obligations apply and how long they require the records, then ask whether those records must live in Airflow at all or can be exported to a log platform you already operate before cleanup removes them.
Before enabling the Helm chart's databaseCleanup CronJob or a scheduled airflow db clean, write down the retention decision, the owner who approved it and the export path for anything that must outlive the window. Run one manual cleanup in a non-production copy and confirm that the audit questions your team actually asks can still be answered from the UI and SQL. Confirm with your auditor or compliance lead whether they accept exported logs as the system of record. Keep this decision under change control, because a later reduction in retention is the change most likely to surprise an auditor.
Setting up a safe cleanup run
This sequence confirms the versions and database you are working with, protects the history you need, and proves the cleanup behaves as intended in a copy before any production run. The result is a scheduled cleanup that your team has tested and can explain to auditors.
Confirm Airflow and metadata database versions
Record the Airflow version, the metadata database engine and its version, and whether the database is PostgreSQL or MySQL. Compare them with the prerequisites for your Airflow release, since Airflow 3.3.2 lists PostgreSQL 14 to 18 and MySQL 8.0, 8.4 and Innovation releases, and confirm that the database is not MariaDB or SQLite. Also confirm that the Airflow version is on a supported line, because a cleanup planned before an upgrade needs the upgrade path settled first. Done when the versions are written down and each one is on the supported list.
Take and verify a metadata database backup
The platform or database owner takes a full backup of the metadata database before any delete runs. The upgrade guide advises a backup first and, where hot backups are not possible, shutting Airflow down so the copy is consistent. Restore the backup into a scratch database and confirm it opens and that key tables such as the DAG run and task instance tables return rows. Done when a restore test has succeeded and the backup location and retention are recorded.
Decide retention for operations and audit needs
Agree with compliance, security and the data owners how long run history, task instances, logs references and audit events must stay queryable. Separate operational retention, which controls UI and troubleshooting history, from audit retention, since the audit log table records user actions and system events with UTC timestamps and may need a longer window. Confirm which period your evidence requirements call for with your auditors rather than assuming one. Done when a written retention value and an owner for the decision exist.
Run cleanup in a non-production copy
Restore the backup to a non-production environment running the same Airflow version, then run airflow db clean in dry-run mode first, if your release supports it, and review the reported tables and row counts. Run the cleanup with the chosen retention and record how many rows each table lost and how long the run took. Check the table sizes before and after to confirm growth is reduced. Done when the dry-run output matches the retention decision and the copy completes without errors.
Review history impact in the UI
Open the cleaned copy in the UI and check that DAG runs, task instances and logs for a recent window still appear as expected, and that runs older than the retention cut-off are gone or marked as removed. Check the audit events for the same window through the UI, the /eventLogs endpoint or SQL to confirm they match the retention decision. Ask the data owners to sign off on the window they need. Done when the recent window is complete and the owners accept the result.
Enable the scheduled cleanup in production
With the copy results accepted, the platform team runs the cleanup once manually in production during an agreed change window, then enables the scheduled job. In the Helm chart, the databaseCleanup CronJob takes a retentionDays value, so set it to the approved figure and confirm the schedule, the service account and the database connection. Keep the backup from step two until the first scheduled run has completed and been reviewed. Done when a manual run and the first scheduled run both complete and the change record is closed.
Add monitoring for database growth and cleanup jobs
Set up alerts for failed cleanup CronJob runs, for database size growth and for the time the metadata database spends on scheduler queries, so a silent failure does not let the database grow unnoticed. Review the first few cycles to confirm the database size trend is flat or falling. Record the owner who receives alerts and the review cadence for retention. Done when alerts fire in a test, the size trend is reviewed after the first cycles, and the retention decision is linked from the runbook.
Running cleanup before an Airflow 3 upgrade
The Airflow 3 upgrade guide tells you to back up the metadata database before anything else, and to trim old data with airflow db clean before the upgrade runs. The reason is practical: schema migrations can be slow on large databases, and the migration rewrites the tables that hold your run history. A database that has accumulated years of task instances, logs and audit events turns the migration window into an unpredictable outage. Before you start, confirm your starting point. The upgrade guide requires you to be on Airflow 2.7 or later, and recommends moving to the latest 2.x release first and checking DAG code with the Ruff AIR301 and AIR302 rules. Read the guide for your exact target version, because the cleanup and migration steps are tied to it.
Decide how you take the backup. The upgrade guide says to shut Airflow down if a hot backup is not possible, so that the copy is consistent. With PostgreSQL, a consistent hot copy depends on how your backup tooling is configured, so confirm with your database team that the backup can be restored and that its consistency has been tested, not just assumed. Keep the backup until you have verified the upgraded instance and the history you expect to see in the UI. Run cleanup against a copy of the database first, record how many rows it removes and how long it takes, and use that measurement to plan the production window. Your retention settings must already cover any audit requirement, since cleanup removes old history that cannot be recovered from the live system.
Security and operational risks of purging metadata
Purging the metadata database removes run history, logs and audit records that other teams may depend on. The cards below cover the risks to decide on before the first cleanup job runs, from audit retention and database privileges to backups, scheduling overlap and access to the CronJob itself.
Deleting audit history
Airflow audit log entries sit in the log table alongside other metadata, so a purge with too short a retention can remove records of who triggered runs, changed state or called the API. Before enabling cleanup, confirm with your compliance owner how long audit evidence must stay in the database, and whether a copy must be exported elsewhere first. Check that the retention you set in the CronJob covers that period, not only the default history your operators want for the UI.
Broad database rights for cleanup
The cleanup process needs write and delete rights on many metadata tables, which is more than most application accounts should hold. A cleanup user that can also read or alter other schemas widens the blast radius of a leaked credential or a misconfigured job. Ask your database team to create a dedicated role limited to the Airflow metadata schema, and confirm which credentials the Helm chart uses for the cleanup pod.
Backups that do not match a cleaned state
A backup taken during or just before a purge may hold a partial picture of history, and a restore may bring back rows the team expected to be gone. The upgrade guide asks for a metadata database backup before trimming data, and shutting Airflow down when hot backups cannot give a consistent copy. Record the time of each purge, take the backup before the job runs, and test a restore to confirm the expected state.
Cleanup overlapping heavy scheduling
A purge that runs while schedulers are creating task instances and writing heartbeats competes with normal work for the same tables. Large deletes during peak scheduling can slow scheduling loops and delay DAG runs that users are waiting on. Schedule the CronJob for a quiet window, check how long a dry run takes on a copy of your database, and watch scheduler delay during the first few runs.
Lock contention and database load
Deletes take row locks, and a long transaction can block scheduler and API queries that touch the same rows, which shows up as slow UI pages and stalled task state changes. Watch for lock waits and connection spikes in the database during cleanup, and confirm that the connection pool sized for Airflow is not exhausted by the cleanup session. Start with a small retention batch and widen it only after you see the impact.
Access to the CronJob and its configuration
Anyone who can edit the databaseCleanup values or the CronJob manifest can change retention to delete more history than intended, or run an extra purge. Restrict edit rights on the release namespace, keep the values file under change control, and review logs from each cleanup pod for the retention used. Confirm who approves a change to retention and whether that approval is recorded in your change management system.
Questions buyers ask about Airflow metadata cleanup
How much effort does Airflow metadata cleanup and retention design take?
The effort depends less on running airflow db clean than on deciding what to keep and proving the decision. The command itself is short to run, but retention design involves agreeing which history the business, operations and auditors need, testing deletions on a copy, scheduling the cleanup and documenting it. For a small installation with no audit requirement, this is a short exercise. For a regulated estate with several teams and long retention periods, it becomes a design task with sign-off. No duration is given here because it depends on the database and the scope you agree. Ask any partner for a written scope that lists each of these steps as a separate deliverable.
What drives the cleanup work up or down?
Four factors shape the work most. Database size and table growth determine how long purges and any follow-up vacuuming take. Audit scope determines how much history must be retained and in what form. The number of DAGs and the run frequency determine how fast the metadata grows and how many retention rules you must test. Upgrade timing matters because Airflow 3 upgrade guidance recommends trimming old data before schema migrations, which can be slow on large databases, so a cleanup run close to an upgrade adds coordination. Ask for each factor to be stated as an assumption in the quote so that a change in scope has a visible effect on the price.
What must our team provide before cleanup work starts?
Your team provides the access and the decisions. Expect to supply:
- Read and write access to a copy of the metadata database, plus a maintenance window for production runs
- Current backup tooling and a tested backup of the metadata database taken before any purge
- Compliance and audit input stating which events, logs and run history must be kept and for how long
- Access to the Helm chart values or deployment configuration, if the cleanup runs as a CronJob
- A named owner who can approve retention rules and accept the impact on UI history
Without the audit input, the design cannot be finished, so ask for it early.
Can our own team run Airflow metadata cleanup without outside help?
Yes, a team that already operates Airflow and its database can run cleanup itself. The command is documented in the Airflow best practices, and the Helm chart supports a databaseCleanup CronJob with retentionDays. The harder part is the decision and its evidence: which records must survive, how audit events are queried after a purge, and how the backup and restore path is tested. Internal teams often handle the mechanics well and benefit from an outside review of the retention rules before they go into production. DeployOpen can take the whole task or review your plan, and the written scope will say which.
Does cleanup remove audit records that compliance teams need?
It can, if the retention settings do not account for them. Airflow audit logs live in the log table and record user actions from the UI, REST API and CLI, along with system events such as task state changes. Before choosing a retention period, confirm with your auditors whether those records must stay in the Airflow database, be exported to a separate store, or be kept for a defined period. The audit log documentation describes how to read the records through the UI, through /eventLogs or by SQL, so you can verify what remains after a run. DeployOpen prepares the evidence for compliance assessments, but independent auditors decide whether your retention meets their requirements.
What is the risk of running cleanup before an upgrade?
The main risk is an inconsistent or unrecoverable database, so the sequence matters. The Airflow 3 upgrade guidance says to back up the metadata database first, to shut Airflow down if a hot backup cannot give a consistent copy, and then to trim old data with airflow db clean, because schema migrations can be slow on large databases. Running a purge and an upgrade in the same window without a verified backup multiplies the failure modes. Plan the cleanup as its own change, verify the backup restores, and only then start the upgrade. Confirm the upgrade path against the current version life cycle, since Airflow 2 reached end of life on April 22, 2026.
How do we check the impact of cleanup on run history in the UI?
Compare what users can see before and after a purge on the same test environment. Record a set of known DAG runs, task instances and audit events, apply the retention rule, then check the grid view, run history, and event log in the UI. Confirm that the records still needed for support and audit remain queryable through /eventLogs or SQL. Note that the UI will show less history for dates outside the retention window, so tell support teams and stakeholders in advance. Keep the test results as change evidence.
How DeployOpen scopes and supports Airflow database cleanup
DeployOpen treats metadata database cleanup as part of the Airflow platform, not a one-off script. We scope retention against your audit and recovery needs, build and test the cleanup in your own infrastructure, and hand over the settings, runbook and evidence your team needs to keep the database under control.
How we scope
- Inventory the metadata database size, the tables that grow fastest, and the history your teams and auditors actually use
- Agree retention periods per data type in the written scope before any cleanup job runs
- Confirm whether cleanup runs through the Helm chart databaseCleanup CronJob or as a planned operator task
- Define the pre-upgrade cleanup step and the backup that must exist before any deletion
How we build
- Run airflow db clean with dry-run output first in a non-production environment that mirrors your database
- Configure the Helm chart CronJob with retentionDays in the values your team controls, under version control
- Verify that the external PostgreSQL or MySQL backend, connection limits and backups behave as expected after cleanup
- Check the UI history, audit log entries and affected DAG runs against the agreed retention before promoting the change
How we hand over
- Provide the written retention policy, the cleanup configuration and the exact commands used in your environment
- Document how to confirm a backup is consistent before running cleanup or an Airflow 3 upgrade
- Walk your platform and compliance owners through where historical evidence now lives and how long it is kept
- Deliver a short runbook for failed cleanup jobs, large-table migrations and restoring from backup
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped in writing
- Review retention settings whenever Airflow, the Helm chart or the database major version changes
- Recheck cleanup results and audit log retention after each planned upgrade of Airflow or PostgreSQL
- Confirm with your auditors which log and history retention periods they need before changing them
