Where Apache Airflow helps with SOC 2 processing integrity
If your team runs scheduled pipelines that feed reports or customer-facing outputs, your SOC 2 examination will likely ask how you show that processing is complete, accurate and timely. The relevant criterion is PI1.3, which requires policies and procedures over system processing so that products, services and reporting meet the organization's objectives. Apache Airflow does not satisfy PI1.3 on its own. It is a scheduler and execution engine, and the policies, specifications and review routines around it are still your responsibility. What Airflow does provide is a platform where several of the points of focus under PI1.3 can be built into the workflow: recording processing activity, detecting processing errors through task failures and data checks, and correcting them through retries and controlled reruns.
Airflow helps most with the recording and error-handling side of that picture. Each DAG run and task instance is kept in the metadata database, so run history can show what ran, when, and with what outcome. Retries, failure callbacks and data quality tasks give you a documented mechanism for catching and handling errors before outputs reach users. Airflow also supports CC7.2, which asks you to monitor system components for anomalies and analyze them. Failed or late runs can be routed to alerting and reviewed as events. The gaps are just as important. Airflow does not decide which processing specifications are correct, does not validate the business logic in your DAGs, and does not record who approved a change unless your repository and review process do. Those controls sit in Git, your ticketing system and your operating procedures, and they need to be designed alongside the scheduler.
Before relying on Airflow run history as evidence, confirm with your auditor which populations and time periods they will sample, and check how your metadata database is backed up and retained, because evidence that is pruned on a schedule may not cover the examination period. Confirm the Airflow version you run is still supported, since the version life cycle determines whether security fixes are available. DeployOpen can plan the Airflow configuration, retention settings, alerting and evidence collection for your processing controls and prepare the systems and records for your assessment. DeployOpen does not issue audit reports or attestations. Your CPA firm performs the SOC 2 examination and forms the opinion.
Mapping PI1.3 and CC7.2 to Airflow capabilities and configuration
This table maps each PI1.3 point of focus and the CC7.2 anomaly monitoring criterion to the Airflow feature, configuration and evidence source that supports it. Read each row as a question to answer in your own environment, and confirm specific settings against the Airflow documentation for your version before relying on them.
| Requirement area | Airflow capability or configuration | Evidence an assessor is likely to ask for |
|---|---|---|
| Defining processing specifications (PI1.3) | DAG definitions in version control with documented schedules, task dependencies, input and output datasets, and owner tags on each DAG | Repository history of DAG files, the deployed commit or bundle per run, and the written specification each DAG implements |
| Recording processing activities (PI1.3) | Run history in the metadata database covering DAG runs, task instances, states, start and end times, and manual triggers or clears | Exported run history for a sample period, showing who triggered or cleared runs and the outcome of each |
| Recording processing inputs (PI1.3) | Task logging of input parameters, source identifiers and row counts, plus run configuration passed at trigger time | Task logs and run configuration for sampled runs, matched to source system extracts or file manifests |
| Detecting processing errors (PI1.3) | Task failure states, failure callbacks and alerting to an on-call channel, configured per DAG or as a default | Alert records, failure notifications and the ticket or follow-up linked to each failed run |
| Correcting processing errors (PI1.3) | Task retry settings (retries and retry delay) for transient errors, and clearing or rerunning failed tasks after a root cause is fixed | Retry configuration in DAG code, clear or rerun history showing the operator and reason, and the corrected output |
| Data checks on outputs (PI1.3) | Validation tasks such as row count, null, range and reconciliation checks that fail the run when thresholds are not met, confirmed against the provider or library you choose | Check definitions in code, check results per run, and records of how failed checks were resolved |
| Anomaly monitoring of scheduler and workers (CC7.2) | Scheduler health and heartbeat monitoring, worker and API server metrics exported to your monitoring stack, with alerts when components stop responding | Monitoring dashboards, alert rules, and records showing anomalies were reviewed and classified as security events or operational errors |
| Audit logging and log retention (CC7.2) | Audit log of user and API actions, and task log retention set to a period your policy defines, shipped to central storage if required | Audit log extracts for sampled privileged actions, and retention settings matched to your documented policy |
| Metadata database and scheduler availability | PostgreSQL or MySQL as the metadata database (SQLite is for testing only), backups, and more than one scheduler where resiliency is required | Database backup records, restore test results, and configuration showing the production backend and scheduler count |
Which run records auditors can sample and how to keep them
A SOC 2 Type 2 examination tests whether controls operated over a period, so the auditor samples records from across that window rather than a single day. The cards below cover the Airflow records that show processing ran, failed, was corrected and was authorized, and the retention and export choices that decide whether those records still exist when the auditor asks.
Run and task history in the metadata database
Every DAG run and task instance, with its state, start and end times, try number and operator, is stored in the metadata database. This is the primary record that processing completed, failed or was retried. Check that your metadata database is PostgreSQL or MySQL rather than SQLite, and confirm with your team how database cleanup is configured, because cleanup removes the history an auditor may sample.
Task logs for each attempt
Task logs show what a processing step actually did on each try, including data checks, row counts and error messages that explain a failure and its correction. Logs are written per attempt, so a retry leaves its own trail. Confirm where logs are stored, whether they are shipped to a central system, and how long they are kept, since they are often deleted sooner than run history.
Audit logs of user and API actions
Audit logs record who triggered, cleared, paused or edited runs and through which interface. For processing integrity, they show that a rerun or manual correction was made by an authorized person. Confirm which actions your Airflow version records, how the logs are exported, and whether they capture actions taken through the REST API as well as the user interface.
DAG code versioned in Git
The DAG code that defined each processing specification should live in Git, so you can show which version ran on a given date. Pair this with the commit or bundle version recorded for each run where your deployment supports it. Check that the repository keeps its history, protected branches and review records, because the run history alone does not show the logic that was executed.
Backups and retention of metadata
A metadata database backup and a written retention policy determine whether history from earlier in the audit period still exists. Set retention for run history, logs and audit records in writing, align it with the period your auditor will test, and test a restore so you can produce records after an incident. Ask your auditor which period and sample sizes they expect before you set the policy.
How retries, data checks and alerts catch processing errors
A scheduler reporting success does not prove that processing was correct. Airflow distinguishes a task that fails, a task that goes up for retry, and a task that succeeds while producing a wrong or incomplete output. The first two are visible in run history. The third is invisible unless the DAG checks its own results. For SOC 2 processing integrity, which asks whether processing is complete, valid, accurate, timely and authorized against defined specifications, retries handle transient errors such as a dropped connection, but only data validation inside the DAG can catch a load that ran cleanly and delivered too few rows. Confirm which retry settings, such as retries and retry_delay, your team will standardize on, and record the reasoning for each critical pipeline.
Validation belongs in the pipeline itself, as tasks that compare row counts, check required fields, reconcile control totals or test referential rules against the agreed processing specification. Failed checks should stop downstream tasks and raise an alert rather than let bad data flow forward. Which operators, sensor or check libraries and alerting callbacks fit your stack varies by provider version, so confirm them against the provider documentation for your Airflow release before you standardize. Each check result should be written to a table or file your team owns, along with the run identifier, so the evidence survives log rotation.
Airflow 3 changes how those validation records are written. Task code can no longer open Airflow database sessions or use Airflow models directly, so a check that writes its outcome to the metadata database must go through the REST API or the Task SDK. Decide early where check results live, since writing them to an external audit table through a supported client is a design choice that affects retries and idempotency. Test that a retried task does not record duplicate or contradictory results, and confirm alert routing with the people who will respond to it.
Configuring Airflow to produce processing integrity evidence
This sequence sets up a self-hosted Airflow 3 so that every scheduled run has a recorded history, standard retry and data check behavior, retained logs and a documented review of failures. Each step produces an artifact an examiner can sample against PI1.3.
Select a supported Airflow 3 release and database
The platform owner chooses a current Airflow 3 patch release from the version life cycle page and checks it against the prerequisites for Python, Kubernetes and the metadata database. Use PostgreSQL 14 to 18 or MySQL 8.0, 8.4 or an Innovation release, because MariaDB is not supported as a metadata database and SQLite should not be used in production. Record the chosen versions in the change record so the evidence trail starts with a known platform baseline. Done when a version record names the Airflow release, Python version, database engine and version, and the database is reachable from every scheduler and API component.
Deploy the DAG processor as a separate component
The platform team runs the DAG processor as its own process or container, separate from the API server and scheduler, as Airflow 3 requires. Confirm that each component reads DAG code from the same approved source, so the DAG version that parsed is the version that runs. Check the parser logs after deployment to confirm that DAGs load without import errors. Done when a test DAG appears in the UI from the approved source and the parser logs show a successful load with no unresolved errors.
Start the API server and set the auth manager
Start the API server with the airflow api-server command and set one auth manager through the auth_manager option, since Airflow 3 allows only one at a time. The default Simple auth manager is suitable for testing, so confirm with your identity team which production manager and role mapping will be used before go-live. Limit who can trigger, clear and mark runs, because those actions change the processing record. Done when a non-administrator test account can view runs but cannot perform restricted actions, and the chosen auth manager is documented in the configuration record.
Standardize retries and data checks in DAG templates
Engineering leads publish a shared DAG template that sets retry counts, retry delays, failure callbacks and at least one data validation task, such as a row count or schema check, before loading to a downstream system. Teams build new DAGs from this template so that error detection is applied consistently rather than per author. Test the template in a non-production environment by forcing a failing check and confirming that the retry, alert and downstream block behave as designed. Done when a forced failure produces a logged retry sequence, a failure notification and no downstream load.
Configure log retention and export
The platform team sets how long task and scheduler logs are kept locally and exports them to a central log store that the security or audit function controls. Confirm the retention period with the audit owner, since it depends on the evidence period the SOC 2 examination covers and on internal policy. Check that log export keeps run identifiers, timestamps and task outcomes so that a single run can be traced end to end. Done when a sample failed run can be followed from the UI to the exported log entries using its run identifier.
Schedule metadata database backups
The database owner schedules regular backups of the metadata database, which holds run history and task state, and stores them outside the Airflow host. Ask the team that operates the database to confirm backup frequency, encryption and access, because these depend on your infrastructure and recovery objectives. Restore a backup into a separate environment at least once and verify that run history is complete. Done when a restore test shows that recent run records and their states are present and match the source.
Hold a written review of failed runs
The operations owner runs a recurring review of failed and retried runs, records each failure's cause, the correction made and the reprocessing decision, and signs the review. This is the step that turns detection into evidence of correction, which PI1.3 and CC7.2 both expect. Agree the review cadence and sign-off roles with the control owner before the first review, and keep the written record in the same system as other control evidence. Done when a signed review record exists for a period, linking each failed run to its cause, correction and reprocessing outcome.
What Airflow does not cover for processing integrity
Airflow runs and records your pipelines, but several processing integrity responsibilities stay with your organization and your auditor. The cards below separate what the platform cannot decide or prove on its own from the controls you must still design, operate and evidence.
Your processing specifications
Airflow executes whatever DAGs define, but it does not know what correct processing means for your business. Processing specifications, such as expected inputs, transformations and output completeness, must be written by your team and tied to each pipeline. Check that every production DAG maps to a documented specification that a reviewer can compare against the code.
Security event decisions
Airflow records task failures and access events, but it does not decide whether an anomaly reflects malicious activity. Under CC7.2, your security team must analyze anomalies and classify them as security events or ordinary processing errors. Confirm who receives failed-run alerts, who triages them, and how that decision is recorded.
Policies and procedures
Processing integrity under PI1.3 requires policies and procedures over system processing, and Airflow cannot write or approve them. Your organization must define who may trigger, clear and backfill runs, how reruns are approved, and when data is reprocessed. Auditors look for these documents to match what the scheduler actually does.
Business logic correctness
A DAG that runs successfully can still calculate the wrong result. Airflow checks task success and can run data quality checks you write, but it does not validate that your transformations, joins or thresholds match the business rules. Plan independent reconciliation, test coverage and reviewer sign-off for logic changes.
Unsupported Airflow versions
Limited-maintenance versions receive only security and critical bug fixes, and end-of-life versions receive no fixes or support. Airflow 2 reached end of life on April 22, 2026, so a platform still on it has no upstream fixes for processing defects. Confirm your running version against the support table before you rely on it for controlled processing.
The SOC 2 report itself
Airflow and this implementation work do not produce a SOC 2 report. A licensed CPA firm performs the examination, and a Type 2 report covers operating effectiveness over a period you select. Ask the auditor which evidence samples they will request from run history and audit logs, and agree on the period and scope early.
Questions buyers ask before hiring help for SOC 2 processing integrity
Does Apache Airflow alone satisfy PI1.3 for SOC 2?
No. Airflow supports PI1.3 but does not satisfy it on its own. PI1.3 asks the organization to implement policies and procedures over system processing, so the evidence has to come from your own processing specifications, error handling rules, reconciliation checks and review routines. Airflow contributes recorded run history, task logs, retries, data quality checks and audit logs that you can point to. Your team still defines what correct processing looks like for each pipeline, who reviews failures, and how corrections are authorized and tracked. Auditors test whether those procedures operate over the examination period, not whether a tool is installed.
What drives the effort and cost of SOC 2 processing integrity work?
The main drivers are the number of pipelines in scope, how complex the data checks must be, how log export is designed, how backups and access controls are set up, and when the audit window starts. Each pipeline that feeds a reported figure needs its own success criteria, retry rules and validation logic. Log export to your SIEM or log store adds integration work and retention decisions. Access setup for who can trigger, clear or edit runs adds review effort. Costs split into one-time build work (design, configuration, documentation) and recurring work (reviewing failures, maintaining checks, producing evidence each period). Ask any partner for a quote that lists these items separately.
How long does a processing integrity engagement take?
Duration depends on the scope you agree, and DeployOpen confirms the timeline in writing before work starts. The largest variables are how many pipelines are included, how many data checks need to be designed from scratch, how much existing evidence is already collected, and how quickly your owners can review and approve decisions. The examination period matters as well. A Type 2 report covers operating effectiveness over a period, so controls need to run consistently for that window before the auditor can test them. Ask for a schedule that separates design, build, evidence collection and handover.
What does our team need to provide for this work?
The work needs named owners on your side, reliable access to the environments being assessed, and your existing policies and procedures. You should name an owner for each pipeline in scope who can confirm expected outputs and accept error handling rules. Someone from security or compliance should own the control narrative and the evidence list. Engineers need access to the Airflow deployment, its metadata database, task logs, audit logs and the systems that receive logs. Existing policies on change management, incident handling and access reviews are needed so new procedures match what you already do. Confirm the exact access model and approval path with your team before the scope is signed.
Can an internal team handle SOC 2 processing integrity without a partner?
Yes, an internal team can do this work if it has people with SOC 2 knowledge and Airflow skills. The SOC 2 side requires understanding of the Trust Services Criteria, how points of focus translate into controls, and how evidence is sampled. The Airflow side requires knowing how run history, task retries, data checks, logging and access controls behave in a production deployment. Many teams find the hardest part is building and keeping consistent evidence over the full period, not the initial setup. A partner can help with design, implementation and documentation, while your team keeps ownership of decisions and daily operation.
How does DeployOpen differ from a SOC 2 auditor?
DeployOpen prepares systems, controls and evidence for assessment. It does not issue audit reports, attestations or certifications. Only an independent CPA firm performing a SOC 2 examination can conclude on your controls and issue a report. DeployOpen's role is to help you design the processing controls, configure Airflow and its surrounding logging and access settings, and organize the evidence an auditor will request. The auditor then tests those controls independently. Keeping these roles separate also means the people who built a control are not the people who judge it. Ask your auditor early which evidence formats and sampling approach they expect.
Which SOC 2 report type should we plan for before we start?
Plan for the report type your customers or contracts require, and confirm the scope with your CPA firm. A SOC 2 Type 1 report covers the system description and the suitability of control design as of a date. A Type 2 report adds an opinion on operating effectiveness over a period, which means your processing controls must run and leave evidence throughout that window. For processing integrity, the Type 2 path needs run history, correction records and review sign-offs to exist for the whole period. Note that SOC 3 reports cover the same categories in less detail and can be freely distributed, while a SOC 2 report's use is generally restricted. Decide this early, because it changes how long evidence must be collected.
What should we confirm about Airflow versions before the audit period starts?
Confirm that the Airflow version in use is supported for the whole period, because controls that depend on a platform going unpatched are hard to defend. The Airflow version support table lists Airflow 3 with a first release date of April 22, 2025, and Airflow 2 reached end of life on April 22, 2026, after which it gets no fixes. Limited maintenance versions receive only security and critical bug fixes. Check the current patch level against the Apache Airflow version life cycle page before you define the scope, and plan any upgrade so it does not fall in the middle of the examination window without documented change management. Your partner should state which version they will assume in the written scope.
How DeployOpen scopes and delivers SOC 2 processing integrity work
DeployOpen is a Chainscore Labs company that plans, implements, documents and supports self-hosted open-source platforms in infrastructure you control. For Apache Airflow and SOC 2 processing integrity, we start from the pipelines and controls your auditor will sample, then build the evidence trail around them. We do not issue SOC 2 reports; your CPA firm does.
How we scope
- Inventory the DAGs and pipelines in scope, with their input sources, outputs and the processing objectives each one supports.
- Agree which processing integrity points of focus the written scope addresses, and which remain with your team.
- Confirm with your auditor the period, system description and evidence formats before any build begins.
How we build
- Configure retries, failure callbacks and data validation checks in DAGs so processing errors are detected and corrected.
- Set up the metadata database, run history retention and audit logging so processing activity is recorded and retrievable.
- Test failure paths in a non-production environment and document expected behavior for each critical pipeline.
How we hand over
- Deliver runbooks for reviewing run history, investigating failed tasks and documenting corrective actions.
- Provide a control mapping that ties each processing integrity point of focus to a configuration, procedure and evidence source.
- Walk your operations and audit teams through sample evidence extraction so they can answer auditor requests without us.
How we support
- Support is available as a retainer for upgrades, log and retention reviews, and changes to pipelines in scope.
- Re-run failure path and control tests after Airflow upgrades or material DAG changes, and record the results.
- Ask for a written quote based on your scope, including who is responsible for each deliverable and what recurs after handover.
