Apache Airflow audit logs for security reviews

What Apache Airflow audit logs record for a security review

A security or compliance team reviewing a self-hosted Airflow needs evidence of who changed DAGs, connections, variables or runs, and when. Airflow's answer to that question is the audit log table, which records user actions from the UI, REST API and CLI, along with system events such as task state changes. Each entry carries a UTC timestamp, an event name and an owner, and can include DAG, task and run IDs when the event relates to them. Teams can read these entries in the UI, through /eventLogs, or with SQL against the metadata database, as described in the Airflow audit logs documentation.

Teams usually need this review at fixed points: before a regulated pipeline goes live, after a change window, during an access recertification, or when an incident requires reconstructing who triggered or cleared a run. Before the review starts, confirm that your Airflow version and configuration write the events you need, and that every path that changes state (UI, API tokens, CLI on the scheduler host) is covered. Check whether a CLI run from a shell on a server is attributed to a named person or to a shared account, because the log can only record the identity Airflow receives.

The audit log shows what Airflow recorded, not what actually happened. It does not prove that a DAG file in the repository matches what the scheduler parsed, that a merged change was reviewed, or that nobody changed the metadata database directly. Pair the log with repository history, deployment records and access reviews. Reviewers also need to confirm who can read the log: the Audit Logs.can_read permission controls access, and the security model says operations users cannot read audit logs or grant permissions, so the reviewer's role must be set up deliberately.

Who can view audit logs and who must be kept away from them

Security reviewers and auditors need to read audit events to answer questions such as who triggered a DAG, who changed a connection, or which API calls a service account made during a given window. Viewing these records requires the Audit Logs.can_read permission, so this is the permission to grant to reviewers and nothing broader. Reviewers should be able to inspect events but should not be able to edit DAGs, connections, variables or roles, because a reviewer who can change what is being reviewed weakens the evidence.

Grant only read access: Map the Audit Logs.can_read permission to a reviewer role and avoid combining it with write permissions on DAGs, connections or variables.
Use named accounts: Give each reviewer an individual identity so that their own views and exports are attributable.
Record access reviews: Keep a dated list of who holds the permission and review it on a schedule your audit program sets.
Export carefully: Reviewers who copy events to a case file should store them in a location the platform team cannot silently alter.
Confirm the read path: Check whether reviewers will use the UI, the /eventLogs endpoint or SQL, since each path needs its own access control.

Before granting access, confirm with your auth manager documentation how Audit Logs.can_read is assigned to roles in your chosen auth manager.

Which review questions the audit log can and cannot answer

Security reviews usually start with a short list of who did what and when. The Airflow audit log answers some of these well and leaves others to controls outside Airflow, so each review question should be checked against what the log actually records and against the systems that hold the rest of the evidence.

Who triggered a manual run

The log records manual runs started from the UI, REST API or CLI, with the owner, UTC timestamp and the DAG and run IDs. Check that the owner value maps to a named person rather than a shared account. The log does not show why the run was triggered or whether the person had approval outside Airflow, so pair it with your change or ticket records.

Who changed a connection

Changes made through the UI, API or CLI appear as user actions, but the log does not hold the secret values themselves. A reviewer can see that a connection was edited and by whom. Whether the new password was correct, or whether it was changed in the external secrets store, needs the secrets backend and its own audit trail.

When a task entered a given state

System events record task state changes with a UTC timestamp and the related DAG, task and run IDs. This is useful for reconstructing a timeline of a failed or delayed run. It does not explain the cause of a failure, and the task logs and scheduler logs remain the place to find the error detail.

Changes to DAG code

The audit log does not record edits to DAG files in the repository or on the deployed file system. A change to a pipeline can alter what runs without any entry for the author. Review merge history, branch protection and deployment records to answer who changed the code and what was approved.

Actions outside Airflow

Anything done on the database directly, on the host, or in the cloud account that runs the workers falls outside the log. Direct SQL changes, container restarts and infrastructure edits need the host, database and cloud audit trails. Confirm which of these you collect and whether they cover the same time window as the Airflow events.

Who can read the log

Viewing audit logs requires the Audit Logs.can_read permission, and operations users cannot read them or grant permissions. A review that starts from the log must therefore be run by a role that holds this permission. Confirm who holds it, since the log itself is a sensitive record of administrative activity.

Audit log read and export options and when each fits

This table compares four ways to read Airflow audit events: the UI, the /eventLogs endpoint, direct SQL on the log table, and an exported copy in a SIEM. Read each row for the access it needs, where it fits a review, and the risk to plan for, then choose the combination your evidence process can rely on.

Read option	Access needed	Best use in a review	Main risk
Airflow UI audit log view	Audit Logs.can_read permission in the auth manager	Quick, filtered checks of who triggered or changed a DAG or run	Results depend on what the UI shows and on retained history, so exports are needed for evidence
/eventLogs REST endpoint	Authenticated API access with the same read permission	Scripted pulls for a fixed review period, with output saved alongside the review	Token scope and pagination mistakes can cause incomplete extracts; confirm the filters you rely on
Direct SQL on the log table	Read-only database account on the metadata database	Ad hoc joins across users, DAGs and runs for investigations	Grants direct database access that bypasses API permissions; confirm the account is read-only and logged
Exported copy shipped to a SIEM	Pipeline configuration by the platform team and read access in the SIEM	Long-term retention, correlation with identity and host logs, and alerting on sensitive actions	Export gaps during outages or schema changes; confirm delivery monitoring and the event fields shipped

Keeping audit evidence longer than airflow db clean keeps metadata

Audit records in Airflow live in the metadata database, so they are subject to the same cleanup as other old metadata. Airflow's best practices describe purging old metadata with airflow db clean. If your deployment runs that cleanup on a schedule and you have no separate retention for audit entries, the records that answer "who triggered this run, who changed this connection" can disappear on that schedule. For a security review, the question to settle first is not how long Airflow keeps the data, but how long your policy requires you to be able to produce it.

Set the retention period from your organization's log retention policy and the review cycles it implies, then configure the cleanup to match or to leave audit tables alone. Before any cleanup job runs, export the entries you must keep to storage you control, either through the /eventLogs endpoint or by querying the log table directly with SQL. Confirm which tables airflow db clean touches in the exact Airflow version you run, since table coverage and options change between releases. Check the Airflow best practices and your version's documentation, and test the cleanup against a copy of the database before you rely on it.

Treat the export as an operational control, not a one-off task. Record who owns the export job, where the files are stored, how they are protected from edits, and how a reviewer would find a given event months later. If you ship audit events to a SIEM, the SIEM's retention can serve as the long-term record, but confirm that the events arrive complete and that the timestamps are preserved in UTC before you let the metadata cleanup run on the shorter schedule.

How to set up audit log review and SIEM shipping for Airflow

This sequence confirms your Airflow version, access model and database, proves that audit events are recorded for each kind of action, sets up an export to your SIEM, and ties database cleanup to your retention decision so that the evidence you need is kept.

Confirm the Airflow version and metadata database
The platform team records the Airflow version, the auth manager in use and the metadata database backend, then checks that the backend is PostgreSQL or MySQL and not SQLite. Airflow 3 is the line covered here, and the current patch release is 3.3.2, so confirm your deployment is on a supported release line in the version support table. Check that the database is the one that holds the log table you will query. The step is done when you have a written record of version, database and auth manager, and when that record is attached to your review evidence.
Assign the audit log read permission to a reviewer role
An administrator confirms which auth manager is active, because only one can run at a time, and then grants the Audit Logs.can_read permission to a dedicated reviewer role. Operations users should not hold this permission if they also administer the system, since the security model notes that operations users cannot read audit logs or grant permissions. Check the result by signing in as the reviewer and opening the audit log view, then signing in as a user without the role and confirming the view is refused. Record both results.
Verify events for a UI, API and CLI action
Run one test action in each channel against a non-production DAG: pause or unpause it in the UI, make a call to the REST API with a service credential, and run a CLI command such as a DAG state query. Then open the audit log in the UI, or query the /eventLogs endpoint, and confirm that each action appears with a UTC timestamp, the event name, the owner and the DAG identifier where it applies. Also confirm that task state changes from a test run are recorded. Keep the query output as the baseline for your later checks. If an action is missing, do not move on until you know which channel failed.
Set the export path and format
Decide how events leave Airflow: through the UI or endpoint for manual review, through a scheduled query of the log table, or both. The platform team writes a read-only query against the log table that selects the timestamp, event, owner, DAG, task and run identifiers, and stores its output in a defined format such as JSON lines or CSV. Confirm the query runs under a database account limited to reading that table. The check is a file produced from a test run that contains the events you created in the previous step, with no credentials or unrelated data in it.
Forward audit events to the SIEM
The security team configures a collector or ingestion job that reads the exported file on a schedule and sends it to the SIEM, such as Wazuh or your existing platform, with the source field set to identify the Airflow instance. Confirm the SIEM supports your chosen format and the ingestion path inside your network, and check the vendor documentation for field mapping. The check is that the test events from the baseline appear in the SIEM with the same timestamps and identifiers as in Airflow, and that a deliberate test action produces a new SIEM event within the schedule you set.
Tie database cleanup to the retention decision
Airflow's airflow db clean command purges old metadata, and audit log rows are part of that metadata. Before scheduling any cleanup, agree on a retention period with compliance and legal owners, and confirm that every event older than that period has already been exported to the SIEM. Then set the cleanup to a window that is longer than your exports need, or exclude the log table if your policy requires it. Check the result by confirming that the earliest event in Airflow matches your retention boundary and that the SIEM still holds the older events you need.
Document the review procedure and run it once
The security lead writes a short procedure that names the reviewer role, the query or endpoint used, the export schedule, the SIEM dashboard or search, the review frequency and the escalation path for unexpected actions. Run one full review cycle using that procedure, including a sample check that a known test action appears in both Airflow and the SIEM. The step is done when the signed procedure and the first review record are stored with your change and access evidence.

Security considerations for protecting the audit trail

An audit log is only useful if the people it is meant to check cannot quietly change it, read more than they should, or misstate when events happened. The cards below cover the risks that sit around the log table itself and the controls that keep the trail credible in a security review.

Operators who can alter the metadata database

The audit log is stored in the Airflow metadata database, so anyone with direct write access to that database can insert, edit or delete rows without going through the UI, API or CLI. Review the database roles that hold write or DDL rights, separate the Airflow runtime account from administrative accounts, and confirm who can run maintenance commands. The security model describes which users Airflow trusts with these operations, and the production guidance on the metadata database explains why a server-grade backend matters here. Ask who has shell or database access to the host, and record that answer as part of the review evidence.

Sensitive values inside log entries

Event records can carry parameters, DAG run configuration or other fields that a user typed into a request, and those values may include passwords, tokens or personal data. Once written, they are copied into every export and backup. Decide which fields your team considers sensitive, test a few representative actions to see what is stored, and restrict who can read the raw table. Masking in task logs does not automatically cover every audit field, so confirm the behavior in your own version before relying on it.

Clock and timestamp consistency

Airflow records event timestamps in UTC, which makes cross-system timelines easier to reconcile only if every host and external system agrees on time. Check that the scheduler, API server, workers and the database use synchronized clocks, and that the SIEM stores the original UTC value rather than a local-time rewrite. Mismatched clocks can reorder a sequence of permission changes and a failed login, which weakens any conclusion drawn in an investigation. Document the time source for each component.

Restricting who can read logs

Reading audit events requires the Audit Logs.can_read permission, and the security model states that operations users cannot access audit logs or grant permissions. Confirm which roles in your auth manager hold this permission, and make sure reviewers receive it without also receiving rights to change DAGs, connections or users. Because the same permission model governs the UI and the /eventLogs endpoint, test both paths with a reviewer account and record the results.

Integrity of exported evidence

Exports to files, spreadsheets or a SIEM are copies, and a copy can be edited without anyone noticing. Store exports in write-once or access-controlled locations, record who exported them and when, and keep hashes or checksums so a reviewer can confirm later that a file was not changed. Reconcile a sample of exported events against the source table to show the export is complete. Ask your SIEM team how it protects ingested records and how long they remain searchable.

Separation of duties between admins and reviewers

The person who administers Airflow should not be the only person who reviews what that administrator did. Assign reviewers a read-only role that includes the audit permission, keep user and role administration with a separate group, and require a second person to approve changes to retention settings or database cleanup. Without this split, an administrator could change a setting and then review the trail that would have shown the change. Write the split into your access review procedure so it is checked each cycle.

Questions buyers ask about Airflow audit logs and security reviews

How much effort does shipping Airflow audit logs to a SIEM take?

The effort depends less on Airflow than on the SIEM and the path the events take to reach it. Airflow audit log entries sit in the metadata database, so the first job is choosing how they leave it: a scheduled export, a query against the log table, or a log pipeline that reads from the database. The main drivers are how many Airflow environments feed the SIEM, how the SIEM parses and normalizes the fields (event, owner, DAG, task and run IDs, UTC timestamp), whether the SIEM already has a parser for this shape, and how alerts are tuned so reviewers are not buried in routine task state changes. Ask any provider for a written list of the event types in scope, the fields mapped into the SIEM, and the test cases used to prove that a sample action appears correctly.

Can a team set up Airflow audit log review without outside help?

Yes, for many teams the basics are within reach of an experienced Airflow administrator working with the SIEM owner. Reading events in the UI, calling the /eventLogs endpoint, and running a SQL query against the log table do not need a partner. Outside help becomes more useful when the work spans several environments, when the export needs to survive upgrades and database maintenance, or when the evidence must be documented for an assessment. Be honest about internal capacity: someone must own the export job, watch it fail, and know who may read the results. A fixed-scope engagement can cover the design, build and documentation, while your team keeps operating it.

Who needs to be involved in an Airflow audit log review?

Three roles usually carry the work. An Airflow administrator owns the metadata database, the export or query, the permissions that allow viewing audit logs, and the cleanup schedule. A SIEM owner owns ingestion, parsing, alert rules, access to the stored events and their retention in the SIEM. A reviewer, often from security or internal audit, decides which actions matter, checks samples on a fixed schedule, and signs off on exceptions. In a small team one person may hold two roles, but the reviewer should not be the same person who administers the logs they review. Agree these roles in writing before the build starts, because gaps appear most often at hand-offs.

How does retention in Airflow interact with database cleanup?

Audit log entries live in the Airflow metadata database, and Airflow's best practices note that old metadata can be purged with airflow db clean. If cleanup runs before events are exported, those records are gone from Airflow and cannot be recovered from it. The safe pattern is to export or ship events to a store with its own retention before any cleanup job touches the log table, then set cleanup to keep the window your own policy requires in the database. Confirm two things with your team: the retention period your policy or regulator expects, and whether the cleanup configuration in your version covers the audit log table as you expect. Test the cleanup in a non-production copy first.

Does an Airflow audit log prove we are compliant?

No. An audit log is evidence that some actions were recorded, not proof that your controls work or that you meet a framework. It can support access review, change tracking and incident investigation, and it can show that the logging control exists and runs. Independent auditors and certification bodies decide whether controls satisfy a framework, and DeployOpen prepares systems, controls and evidence for those assessments but does not issue audit reports, attestations or certifications. When you plan evidence, define the control, the owner, the sampling method and the retention, then ask your auditor which artifacts they expect. The logs are one input among access lists, change records and review sign-offs.

Who can read Airflow audit logs, and why does it matter for reviews?

Viewing Airflow audit logs requires the Audit Logs.can_read permission, and Airflow's security model says operations users cannot access audit logs or grant permissions. This separation matters for reviews: the people who run pipelines should not be able to edit their own trail, and reviewers need read access without broader rights. Confirm which roles in your auth manager carry this permission, check whether service accounts used by automation hold it unnecessarily, and record the role mapping as part of the review evidence. Exports to a SIEM add a second access path, so limit who can read the stored events there as well.

What should we check during a periodic Airflow audit log review?

A periodic review should confirm that events are arriving, that the fields a reviewer depends on are present, and that a known test action appears in the SIEM within the window your team agreed. Check that changes to DAGs, connections, variables and user permissions are recorded with an owner and a UTC timestamp, that API and CLI actions are attributed rather than anonymous, and that cleanup has not removed events the export still needs. Sample a set of actions against change tickets and access approvals, and record exceptions with an owner and a date. Keep the review as a repeatable checklist so each cycle produces the same evidence.

How DeployOpen scopes audit log reviews and SIEM shipping

DeployOpen scopes audit log work as a fixed-scope engagement, with the deliverables agreed in writing before any configuration starts. DeployOpen, a Chainscore Labs company, plans, implements, hardens and documents self-hosted Airflow inside your own cloud accounts, data centers or air-gapped networks. For audit logging, that usually means confirming who can read the log table, setting a retention rule that accounts for airflow db clean, deciding how records leave the metadata database, and writing down the review procedure your security team will follow. The written scope names which of these you need, which Airflow deployments are covered and who on your side signs off on each item.

Shipping to a SIEM is a separate line item because it depends on the SIEM you run. DeployOpen can scope the export path, the field mapping and the access controls on the destination, but the SIEM platform itself, its licensing and its sizing belong to your existing contracts or to a separate scope. Ask for the scope to list the source system, the transport, the owner of the destination and the test you will use to confirm that a sample UI or API action arrives with its timestamp and owner intact. Retainers are available if you want the export checked after each Airflow upgrade.

DeployOpen prepares systems, configuration and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, so the deliverable is a set of documented controls, a record of how each was tested and a list of open questions for your assessor. Where your assessor needs a standard reference, confirm the control mapping with them directly. Once the initial work is accepted, ongoing support, upgrades and periodic log review checks can be placed on a retainer so that log coverage keeps pace with your Airflow version and team changes.

Using Apache Airflow audit logs for security reviews