Airflow for SOX ITGC job scheduling

How Apache Airflow supports SOX ITGC job scheduling evidence

Apache Airflow can serve as the controlled batch scheduler for month-end and quarter-end financial reporting jobs, but it does not make a company compliant with SOX. Management's annual ICFR assessment under Section 404 and any auditor testing remain with the company and its external auditors. For the finance or IT operations lead, the practical question is which general control areas the scheduler touches and what evidence it can produce for each. Under the SEC's guidance in Release 33-8810, the areas management commonly considers are program development, program changes, computer operations and access to programs and data. Airflow touches computer operations (run history, failure alerts, retries and follow-up on failed reporting jobs), program changes (DAG code reviewed and promoted through environments) and access (who can trigger, clear or edit runs). It does not cover program development on its own, HR-driven access provisioning, or the broader IT environment such as the operating systems, databases and networks beneath it.

Scope is set by the company's own risk assessment, not by the platform. SEC guidance says management need only evaluate the IT general controls necessary for other controls over financial reporting to operate properly and consistently, and need not evaluate those that only affect operational efficiency. Before building anything, the finance and IT leads should list which reports depend on which DAGs, which of those feed controls that auditors plan to rely on, and which general controls those controls depend on. DeployOpen can prepare the Airflow configuration, the change workflow, access design and evidence collection to match that list. DeployOpen does not issue audit reports, attestations or certifications. Confirm with your external auditor, before design starts, which evidence they will sample and in what form.

Mapping SOX ITGC requirements to Airflow capabilities and configuration

This table maps the SOX ITGC areas that auditors test to the Airflow capabilities and configuration that support them, with the company's own process named wherever a row depends on it. Read each row as a pair: the platform provides the mechanism, and the company provides the approval, review and follow-up that make the evidence meaningful. The main takeaway is that Airflow supplies run history and logs, but the control itself is only as strong as the change, access and monitoring process around it.

Requirement area	Airflow capability	Configuration needed
Computer operations (AS 2201 B29, Release 33-8810)	Run history, task states and retry records for each scheduled financial job	External PostgreSQL or MySQL metadata database, not SQLite; retention settings agreed with the audit team
Scheduler availability and resiliency (AS 2110 B4)	Multiple schedulers can run at once using database row-level locks	At least two schedulers, a supported PostgreSQL or MySQL backend, and monitoring of scheduler health
Inaccurate processing and manual intervention (AS 2110 B4)	Logged clears, reruns and manual triggers tied to a user identity	Auth manager chosen and role mapping set so only named roles can trigger or clear runs; reviewed by the company
Unauthorized program changes (AS 2110 B4, COSO Principle 11)	DAG bundles can be versioned so each run records the code version it used	Versioned DAG delivery from a controlled repository; pull request approval and separated environments are company process
Program development and change control (Release 33-8810)	Run records show which DAG version executed, supporting change traceability	Change tickets linked to commits and deployments; the link itself is set up by the company, not by Airflow
Access to programs and data (Release 33-8810)	Auth manager controls who can view, trigger and edit DAGs, with API access	Separate DAG processor and API server; access reviews run periodically by the company, and the review is evidence it did happen
Failure monitoring and follow-up (AS 2201 para 45)	Failed task states, alerts and callbacks can feed a monitoring system	Alert routing to a named on-call group; tickets for each failure are a company process, not a platform feature
Audit trail of administrative actions	Audit log records for supported actions, to be confirmed for the exact version in use	Log shipping to retained storage outside the scheduler host; confirm which events are captured in the target release
Operational efficiency only (Release 33-8810)	Scheduling features that do not touch financial reporting risks	Scope the ITGC review to jobs that feed ICFR; the company decides which pipelines are in scope

What evidence Airflow produces for auditors and how to keep it

An auditor testing a scheduled financial job will ask what ran, what it logged, who changed it and which code version produced the output. Airflow can hold most of that record, but the retention, export and inspection steps sit with your team, so plan them before the audit period starts.

Run history as the operating record

The metadata database records each DAG run, task instance, state, start and end time, and retries. This is the primary evidence for whether a job ran, on time and with what outcome. Auditors commonly use it for inspection and to select samples for re-performance. Confirm how long your metadata database keeps run records, because the database cleanup process can remove them, and export the period you must cover to storage you control before cleanup runs.

Task logs for observation

Task logs show what a specific job printed or raised during a run, which supports observation of operations and follow-up on failures. Where logs are stored depends on your deployment, whether on local disk, a shared volume or object storage. Confirm who owns log retention and whether logs from short-lived Kubernetes pods are shipped off the cluster before the pod disappears, since missing logs are a common gap in audit requests.

Audit logs for user actions

Audit logs record who triggered, cleared, paused or edited runs and when. They answer the access question: which named person or service account changed the state of a financial job. Export them regularly to a central log platform with restricted write access, and confirm that the retention period in your environment covers the full audit period and any review window your auditor requests.

DAG source in version control

The DAG code in your Git repository is the authorized definition of each job, and its commit history is the change record. Inspection of this history shows who approved each change and whether it passed review before deployment. Confirm that protected branches, required reviews and retention of merged pull requests are configured in your Git platform, because Airflow itself does not hold the approval record.

Version per run

The link between a run and the exact code that produced it is what lets you re-perform a job. If your DAG bundles are versioned, each run records the bundle version it used. Ask your team to confirm in writing which identifier is stored per run, how it maps to a Git commit, and whether it survives later deployments. Without this link, the auditor cannot prove that the code they reviewed is the code that ran.

Re-performance and inquiry support

Re-performance means rerunning a control to see if it gives the same result, and inquiry means explaining the control to staff. Plan a documented walkthrough for each financial DAG, with the expected inputs, outputs and failure handling, and test one re-run in a non-production environment against archived inputs. Keep the walkthrough and test results with the evidence so they are available when requested.

Failure monitoring and follow-up evidence for batch runs

For scheduled financial reporting DAGs, the evidence question is whether a failure was seen, by whom, and what happened next. The control works only if alerts reach a defined group, each alert can be traced to a specific run, and each failed run is followed up in a ticket that is closed with a documented outcome.

Alert routing: Document which DAGs notify which group, and keep the distribution list or on-call rotation definition as part of the control record.
Notification record: Keep the alert message or delivery log showing the DAG ID, run ID, failed task, and the time the notification was sent.
Follow-up ticket: Open a ticket for each failed reporting run that links the run ID, records acknowledgment, the cause and the resolution.
Coverage: Confirm that monitoring covers task failures along with upstream sensor timeouts and missed schedules, since a run that never starts can go unnoticed.

Check that a sample of failed runs from the period under review can be traced from alert to ticket to a successful rerun, using the history your own monitoring tools hold.

What Airflow does not cover for SOX ITGC

Airflow can enforce and record scheduling controls, but several SOX obligations sit outside the scheduler. The cards below set out what remains with management, the identity team and your auditors, and which platform settings must be planned before the first financial job runs.

ICFR scope stays with management

Airflow does not decide which processes, locations or accounts fall inside your internal control over financial reporting. SEC guidance says the IT general control areas that matter vary with each company's facts and circumstances, so the scope is a management decision documented before the scheduler is configured. Check that the scope statement names the DAGs and connections it covers, and that someone owns it.

Financial reporting job classification

Airflow runs every DAG the same way, so it does not know which jobs feed a financial statement line. Your finance and controls owners must tag which pipelines affect reported numbers and which only support operations. Without that list, change control and monitoring tend to be applied unevenly, and auditors will ask how the list was built and reviewed.

Access reviews stay in the identity provider

Airflow can map roles from an identity provider, but the periodic review of who can trigger, clear or edit runs is an identity governance activity. Access certifications, leaver removal and segregation checks should run in the system of record, with Airflow roles reconciled against that evidence. Confirm which auth manager you will use and where role assignments are approved.

Management assessment and auditor attestation

Management performs the annual ICFR assessment under Section 404, and auditors issue any attestation. Airflow produces run history, logs and change records that feed that work, but it does not produce an assessment, a conclusion or an attestation. Ask your auditor which samples and populations they will request from the scheduler before you define retention.

Airflow 2 end of life

Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes. A financial scheduler on an unsupported release is hard to defend under change management and IT risk assessment. Plan the move to Airflow 3 as a controlled change with its own testing and approval, and record the version in use for each period under review.

Separate Airflow 3 components to configure

Airflow 3 replaces the webserver with an API server started by airflow api-server, and the DAG processor must run as its own component. Each needs its own deployment, health checks, logs and access settings, and the auth manager is set through a single option. Document these components in the configuration baseline so that a change to one is traceable.

Configuring Airflow 3 for a controlled financial reporting schedule

This sequence builds a self-hosted Airflow 3 scheduler in which every component, access path and DAG version can be shown to your auditors. Confirm each step with your own control owners and auditors before relying on it as evidence.

Confirm the supported version and prerequisites
The platform owner checks the Airflow version life cycle and the prerequisites for the chosen release, including Python, database and Kubernetes versions where used. Confirm that the version you run is in active support, since limited-maintenance and end-of-life versions receive only security fixes or nothing at all. The output is a dated record of the version, its support status and the prerequisite matrix, filed with the change ticket that approves the build. Ask your auditors which version record they expect to see.
Run the metadata database on PostgreSQL or MySQL
The database owner provisions a production metadata database on PostgreSQL or MySQL in infrastructure you control, and does not use the default SQLite backend. Document the database engine version, backup schedule, access accounts and who holds the administrator role. The output is a database configuration record and a successful connectivity check from the scheduler, API server and DAG processor. MariaDB is not supported as a metadata database, so confirm this with the vendor before any engine change.
Deploy the API server and DAG processor separately
The platform engineer deploys the API server with the airflow api-server command and runs the DAG processor as its own component, because Airflow 3 no longer uses the webserver model. Record each component's host or pod, its service account, its configuration source and its restart policy. The output is a component inventory showing that the scheduler, API server, DAG processor and workers are separate and individually controlled. Auditors often ask how each of these is started, so keep the startup configuration under version control.
Enable multiple schedulers where resiliency is required
Where a missed financial run would delay reporting, the platform team runs more than one scheduler, which Airflow supports through database row-level locks. Test a scheduler failover in a non-production environment, stopping one instance while a scheduled run is in progress and confirming that the run completes or fails visibly. Record the scheduler count, the failover test result and the alert that fires when a scheduler stops. Confirm the expected number of schedulers and the failover behavior with your operations and audit teams.
Select the auth manager and role model
The security administrator chooses one auth manager, set through the auth_manager option, since Airflow 3 allows only one at a time. The default is Simple auth manager, which is generally unsuited to a controlled financial environment, so review the alternatives in the authentication documentation and confirm the choice with your identity team. Define roles that separate people who author DAGs, people who trigger or clear runs, and people who only view logs. The output is a role matrix signed by the control owner and a configuration file that matches it.
Set up DAG delivery from version control
The platform team connects DAG delivery to a version control repository so that only reviewed commits reach the production DAG folder or bundle. Require pull request approval before merge, and restrict who can push to the production branch. Record which commit each deployed DAG version came from. The output is a documented delivery pipeline and a sample test showing that an unreviewed change cannot reach production. Ask your auditors how they want commit history linked to scheduled runs.
Document the scheduling and change procedures
The process owner writes the scheduling procedure covering job calendars, dependencies, failure notification, follow-up ownership and how a failed financial run is re-run or cleared. The change procedure covers DAG changes, Airflow upgrades, emergency fixes and approvals. Both documents name the control owners and the evidence each step produces. The output is approved procedures stored with the control matrix, reviewed by your internal audit and external auditors before the first period-end cycle.

What do SOX teams ask before scheduling jobs in Airflow?

Does Apache Airflow itself cost anything to run?

The Airflow software is free to use: it is an Apache Software Foundation project released under the Apache License 2.0 (SPDX: Apache-2.0), and the project itself offers no paid edition. Your spend sits elsewhere: the servers or Kubernetes capacity that run the scheduler, API server, DAG processor and workers, the metadata database, storage for logs and DAG bundles, backups, monitoring, and the people who operate and change the platform. For a SOX environment, add the cost of independent testing of controls and of any external services you choose. Ask any partner for a written scope that lists each recurring infrastructure and licensing item separately from one-time implementation work, so you can compare it with your own run cost.

How much effort does a SOX-ready Airflow setup take?

No supplied source gives a duration or effort figure for this work, so treat any number you hear as unverified until it appears in a written scope. Effort is driven by how many financial reporting jobs move onto Airflow, how many systems they touch, whether the production metadata database and secrets store are already in place, how much change control and access review your organization already runs, and how many environments need to be separated. Ask for the effort split by workstream: platform build, job migration or onboarding, control design, evidence collection and handover to your operations team.

What does the company need to provide before work starts?

Expect to supply access to the cloud accounts, data centers or private networks where Airflow will run, named owners for each financial reporting job and its upstream and downstream systems, and the existing control narrative for batch processing if one exists. Your security team will need to approve identity integration, secrets handling and log retention, and your change advisory process must be defined before DAGs go to production. Confirm in the written scope who approves each of these inputs, and what happens if a dependency, such as a database or identity provider, is not ready on the planned date.

Can our internal team run Airflow for financial reporting jobs?

Often yes, but it depends on skills and capacity that you should check honestly. Operating a SOX-relevant scheduler means keeping Airflow on a supported version (Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes), running a production-grade metadata database such as PostgreSQL or MySQL instead of SQLite, handling DAG change control, and producing evidence for every run. Compare your team's current capacity with that list. A partner can build and document the platform, then hand over runbooks and evidence procedures, or support it under a retainer if your team wants ongoing help.

What will the external auditor ask to see for scheduled jobs?

Auditors test the general controls over computer operations and change management that support automated application controls. Under PCAOB AS 2201, operating effectiveness is tested by inquiry, observation, inspection of documentation and re-performance, so expect requests for the job schedule and failure monitoring records, evidence that failed or late runs were investigated and resolved, approvals for DAG changes and upgrades, and a list of who can trigger, clear or edit runs with their access reviews. Your auditor decides the sample and the procedures. DeployOpen prepares systems, controls and evidence for that assessment but does not issue audit reports or attestations.

Which Airflow controls are most likely to be questioned?

Auditors commonly focus on three areas: whether unauthorized changes to DAGs or the platform can reach production, whether failed or inaccurate runs are detected and followed up, and whether manual intervention such as clearing or re-running a financial job is authorized and logged. Each area needs a documented control design, a named owner and retained evidence. Ask your partner how the design will show who can trigger and clear runs, how versioned DAG bundles tie each run to a reviewed commit, and how monitoring alerts are routed and closed. Confirm these points against your own control matrix before relying on them.

How do we get an accurate quote for this work?

Ask for a written, fixed scope before any work starts, because DeployOpen does not publish a price list and quotes follow from the scope. The scope should name the jobs in scope, the environments, the identity and secrets integrations, the control documentation to be prepared, the evidence to be collected, the handover deliverables, and the timeline agreed in writing. Ask the vendor to separate one-time build work from recurring retainers for support, upgrades and testing, and to state which items depend on your decisions. A scope that cannot be tied to specific jobs and systems is not yet ready to price.

Who owns which SOX control decisions

Management decides what is in scope and how each control is designed, and the company operates those controls every day. SEC guidance in Release 33-8810 says the IT general control areas relevant to an ICFR evaluation vary with each company's facts and circumstances, and that management only needs to evaluate the IT general controls necessary for other controls that address financial reporting risks to operate properly and consistently. For a scheduler, that means finance and IT leadership must decide which DAGs feed financial reporting, which run-level outcomes count as controlled events, who may trigger or clear runs, and who reviews failures and their follow-up. These are company decisions. DeployOpen can document the options and configure the platform to match, but it cannot decide what your ICFR scope is.

Your staff run the controls, and DeployOpen configures the platform and assembles the evidence. In practice, the operating team owns the access model for triggering and clearing runs, the review of failed runs and the sign-off on DAG changes that reach production. DeployOpen's part is to configure the platform components, such as the metadata database, the scheduler, the API server and the auth manager, to support those controls, and to organize the resulting records: change history for DAG code, run history, and audit logs. Confirm with your own team which records your environment retains and for how long, because retention depends on your database maintenance and log shipping choices, not on the scheduler by default.

Independent auditors test the controls and issue the opinion, and no part of this implementation carries a certification. Under PCAOB AS 2201, auditors test operating effectiveness by combining inquiry, observation, inspection of documentation and re-performance, and they assess IT risks under AS 2110. Section 404 requires management's annual ICFR assessment and, for applicable filers, an auditor attestation under 15 U.S.C. 7262. DeployOpen does not issue audit reports, attestations or certifications. Ask your auditor early which samples and populations they expect, so the evidence you collect is the evidence they will request. Clarify these roles in the written scope before work starts, so that each control has a named owner, a named evidence source and a named reviewer.

How DeployOpen works on this

DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure, so the Airflow scheduler, metadata database and DAG delivery path stay under your control. For SOX work, we start from the financial reporting jobs your auditors care about and design the scheduling, monitoring and change evidence around them.

How we scope

We agree in writing which financial reporting jobs, schedules and environments fall inside the Airflow scope before work starts.
We map which controls your team relies on from Airflow, such as failure alerts, reruns and DAG change approvals, and record the evidence each one should produce.
Engagements are fixed-scope, so deliverables and timelines are set in writing, and you receive a quote that follows directly from that scope.

How we build

We deploy Airflow 3 on infrastructure you operate, with the API server and DAG processor as separate components and PostgreSQL or MySQL as the metadata database.
We configure role-based access so that only named groups can trigger, clear or edit runs, and we document who holds each permission.
We wire failure notifications and run history retention so that every failed financial job leaves a trace that your team can follow up and close.

How we hand over

We deliver runbooks for monitoring, failure triage, manual reruns and emergency DAG changes, written so your operations staff can perform them.
We document the DAG change path from pull request review through staging to production, including how each run is tied to the code version that executed it.
We provide a control evidence pack that maps each configured control to the artifacts it produces, which your internal audit and external auditors can then test.

How we support

Ongoing support, Airflow upgrades and continuous testing are available as a retainer, so the scheduling controls stay maintained after handover.
We plan upgrades around Airflow's support life cycle, since Airflow 2 reached end of life on April 22, 2026, and version changes can affect controls.
We confirm with your auditors which evidence they expect before go-live, because the sufficiency of IT general controls depends on your own facts and circumstances.

Apache Airflow for SOX ITGC job scheduling and monitoring