Open source data pipelines for SOX ITGCs

Open source data pipelines for SOX ITGCs: what the framework asks of your pipelines

When Airflow, Dagster, NiFi or Kafka jobs produce the numbers that land in a general ledger, a reconciliation or a disclosure, the pipeline becomes part of internal control over financial reporting. Management must evaluate ICFR as of each fiscal year end, and the Sarbanes-Oxley Act of 2002 (Section 404) makes that assessment an annual duty. The short answer is that SOX does not prescribe tools. It does not name Airflow or Kafka, and no SEC rule or PCAOB standard publishes a catalog of IT general controls. What it expects is that the systems feeding reported figures are controlled, that the controls work, and that you can prove both. In practice, the areas that matter for a pipeline are computer operations (job scheduling, monitoring and failure handling), program changes (how DAGs, flows and connector configurations get approved and promoted) and access to programs and data (who can edit, rerun or backfill a job).

SEC Release 33-8810 says the IT general controls relevant to an ICFR evaluation vary with each company's facts and circumstances, and that management only needs to evaluate the controls necessary for other controls over financial reporting to operate properly and consistently. Your first decision is therefore scoping: which pipelines sit on a path to a financial assertion, and which only serve operational analytics. External auditors work from a different direction. Under PCAOB AS 2201 they test internal control, and under AS 2110 they assess IT risks, including unauthorized changes, inaccurate data processing and inappropriate manual intervention. Their work depends on understanding the general controls that support automated controls. Ask your auditors early which pipeline controls they plan to rely on, because that answer determines the evidence you must keep.

DeployOpen prepares the systems, controls and evidence for these assessments, including the configuration baselines, change records and job monitoring that a pipeline environment needs. DeployOpen does not issue audit reports, attestations or opinions. Those come from your independent auditor and, where applicable, your own management assessment. Confirm with your external auditor and your internal audit team which ITGC areas they expect for each in-scope pipeline before the design is fixed, so that the scheduling, change approval and access controls described in the rest of this page are built to the evidence they will request.

Which SOX requirements map to which pipeline controls

This table maps the IT risks and general control areas that auditors reference to specific pipeline configuration choices in Airflow, NiFi, Dagster and Kafka. Read each row as a question to answer in your own environment: what the control must prove, and which setting or process supplies that proof. The main takeaway is that most evidence comes from configuration baselines, change records and reconciliation outputs, so design them before the first production run.

Requirement or risk	What the pipeline must do	Configuration and evidence to prepare
AS 2110 risk: unauthorized changes to programs	Only reviewed DAG and flow changes reach production, and emergency edits leave a record.	Git pull requests with named reviewers, protected main branch, promotion from dev to test to prod by pipeline, and a log of any direct NiFi flow edits.
AS 2110 risk: inaccurate or incomplete processing	Each transformation step produces the same result for the same input and fails loudly when it cannot.	Versioned DAG code, pinned dependencies, idempotent task design, and row count or control total checks written to a run log.
AS 2110 risk: potential loss of data	Failed or partial loads are retried, held, or rerun from a known point without silent gaps or duplicates.	Airflow retries and backfill procedures documented, PostgreSQL or MySQL metadata database with backups, and Kafka replication settings and retention confirmed with the platform owner.
AS 2110 risk: reliance on inaccurate systems and inappropriate manual intervention	Manual reruns, parameter overrides and data fixes are limited to named roles and recorded.	Restricted Airflow roles for triggering and clearing tasks, a ticket reference required for manual reruns, and audit logs exported to the central log store.
AS 2201 B29: effective general controls over computer operations	Scheduled jobs run on time, failures are visible, and someone owns each alert.	Schedules and SLA or timeout settings per DAG, alerting on failed runs to an on-call channel, and a daily record of failed and rerun jobs for review.
AS 2201 B29 and SEC 33-8810: program changes	Changes are tested, approved and traceable to the job that ran in production.	Pull request approvals, CI test results attached to each change, deployment tags tied to Airflow or Dagster code versions, and a change log for NiFi process group versions.
SEC 33-8810 and COSO Principle 11: access to programs and data	Only authorized people and services can read, change or trigger pipelines and topics.	Airflow auth manager roles mapped to groups from the identity provider, Kafka SASL authentication with SCRAM or Kerberos, and StandardAuthorizer ACLs with allow.everyone.if.no.acl.found left at false.
Conversion and reconciliation controls	Source-to-target totals agree after each conversion step, and differences stop downstream publishing.	Reconciliation tasks comparing counts and control totals, documented tolerance rules agreed with finance, and retained reconciliation reports for each reporting cycle.

Which open-source platforms fit a SOX-relevant pipeline

A pipeline that feeds financial reporting needs an orchestrator, a streaming layer, a transfer path and an ingestion tool, each with its own licensing and operating model. The platforms below are grouped by the job they do, so finance, IT and audit teams can match a platform to the workload and to the skills that will run it.

Apache Airflow for scheduled reporting jobs

Airflow (Apache-2.0) schedules, retries and monitors batch jobs that build the reporting tables, and its run history supports the evidence auditors ask for on job completion and failure handling. The production database matters: Airflow 3.3.2 expects PostgreSQL or MySQL as the metadata database, and SQLite is for testing only. Confirm that every Airflow 2 deployment has been moved off the line that reached end of life on April 22, 2026, and that multiple schedulers are configured only on a database that supports row-level locking.

Dagster for asset-based pipelines

Dagster (Apache-2.0) models pipelines as software-defined assets, which makes lineage and materialization history visible per table and suits teams that want code review on every data asset. Its current release is 1.13.26. Confirm how run and materialization history is retained, exported and protected from edits, because that history is part of the evidence for transformation controls. Check the team's Python skills before choosing it over Airflow.

Apache Kafka for streamed financial events

Kafka (Apache-2.0) carries transaction and ledger events between systems when reporting needs near-real-time feeds. The 4.x line runs in KRaft mode only, so ZooKeeper-based clusters must migrate first. Access should be restricted with SASL authentication and ACLs, and the StandardAuthorizer's default of denying resources without ACLs should be kept in place. Confirm the supported release line and upgrade path with the Kafka project before committing to a version.

Apache NiFi for controlled transfers

NiFi moves files and records between network zones or from legacy systems with per-flow provenance, which records where each record came from and where it went. It fits transfers where the team needs visual flow control and a record of each hop. Confirm the NiFi version, the authentication model and how flow definitions are versioned and promoted, since uncontrolled edits to a flow are a change-management finding waiting to happen.

Airbyte for source ingestion

Airbyte handles connector-based extraction from SaaS tools and databases into staging. Its platform repository is licensed under the Elastic License 2.0, which restricts providing the software to third parties as a managed service, so legal and procurement should review the license before any internal or client-facing use. Connector versions, sync logs and credential storage all become controlled items, so confirm how each is versioned and reviewed.

How the control areas differ by pipeline pattern

Scheduled batch loads move period-end balances, subledger extracts and reconciliation files into a ledger or reporting warehouse on a fixed calendar, often through Airflow DAGs or Dagster jobs. The ITGC focus is computer operations: whether jobs run when scheduled, whether failures are detected and rerun, and whether a partial load is blocked from reaching reports. Program change controls matter because a changed transformation in a DAG can alter every downstream figure without any visible error. The main risk is incomplete or inaccurate processing that looks successful.

Schedule integrity: Confirm who can edit schedules, retries and dependencies, and whether those edits are logged.
Failure visibility: Check that failed or skipped tasks alert an owner and that a dependent report cannot publish from an incomplete load.
Row and control totals: Look for record counts or hash totals compared between source and target on each run.
Backfill handling: Document who approves a rerun of a closed period and how the rerun is evidenced.
Metadata database: Confirm the scheduler's metadata store is a supported production database with backup and restore tested.

Before relying on this pattern, confirm with the audit team which batch jobs feed significant accounts and which of them they plan to test as automated or IT-dependent controls.

How change approval works for DAGs and flows

A change to an Airflow DAG or a NiFi flow that feeds financial reporting is a program change, and auditors will treat it that way. SEC Release 33-8810 lists program changes as one of the IT general control areas management might consider, and AS 2110 names unauthorized changes to systems or programs as an IT risk to internal control over financial reporting. The practical consequence is that every DAG file, every flow version and every scheduler configuration that alters how a reporting number is calculated needs a documented request, an approver who is not the author, and a record that the approved version is the one running in production. Keep the code in a version-controlled Git repository, require a pull request with at least one reviewer who did not write the change, and make the merge the only path into the deployment pipeline. Tools and configuration in your deployed Airflow or NiFi environment should be deployable only from that pipeline.

Segregation matters as much as the review itself. Developers should not hold standing rights to deploy to production or to edit DAGs directly in the production scheduler, and deployers should not be able to approve their own changes. Promote each change through separate development, test and production environments, and confirm that test runs and reconciliation outputs are retained for the change ticket. Emergency changes need their own path: a defined on-call role, a recorded justification, and a retrospective review within a window your change policy sets. The main risk to plan for is an unapproved edit made directly on the production server or through the web interface, because it bypasses the review record entirely. Ask your team to confirm that production file permissions and the Airflow or NiFi admin roles block that path, and that a scheduled comparison flags any drift between the repository and what runs. The evidence for each of these points is listed in the evidence table below.

How a SOX-ready pipeline control set is built

The sequence moves from knowing which pipelines touch financial figures to a documented, tested control set that management can evidence for one full control cycle and hand over to the team that runs it. Each step produces an artifact that the next one depends on, so the order matters more than the pace.

Inventory pipelines that touch financial figures
DeployOpen works with finance, accounting and IT owners to list every Airflow DAG, NiFi flow, Dagster job, Airbyte connection and Kafka topic that moves, transforms or summarizes data used in financial statements, reconciliations or key reports. The inventory records the source system, destination, schedule, owner and the report or account each job feeds. The step is done when finance signs off that no in-scope feed is missing, including feeds that run as ad hoc scripts or manual uploads.
Classify pipelines by reliance
Each inventoried pipeline is rated by how much management relies on its output and by whether a downstream control depends on it. A feed that populates a general ledger interface sits in a different tier from a dashboard used only for operational monitoring. The classification decides how much change control, monitoring and evidence each pipeline needs. Management and the external auditor should agree the classification in writing before controls are designed, because the SEC guidance on ITGCs says the relevant areas depend on each company's facts and circumstances.
Set up the production metadata database and scheduler
The production Airflow or Dagster environment is built on a supported metadata database, with SQLite excluded and the version chosen from the currently supported release line. DeployOpen configures the scheduler, executor and secrets handling in the client's own infrastructure and confirms the platform versions against the vendor's lifecycle and prerequisite pages. The step closes when a test run shows that scheduling, logging and state persistence behave as documented and that the environment is separated from development and test.
Define failure, alerting and rerun handling
The team writes the rules for what happens when a batch job fails, runs late or produces partial output: who is alerted, how the job is retried, when a rerun is approved, and how a backfill is kept from double-posting records. Rules are implemented as configuration in the orchestrator, such as retry settings, alert routes and idempotent write patterns, rather than left to operator memory. The step is complete when failure tests have been run on representative jobs and the outcomes are logged and reviewed by the control owner.
Implement change and access controls
DAG and flow code moves through version control, peer review and a promotion path into production, and only named roles can deploy or edit production schedules and connections. Access to orchestrator roles, secrets stores and Kafka ACLs is granted through the organization's identity process and reviewed on a schedule the control owner sets. The step closes when a sample change has been traced from request to approval to deployment, and when the list of users with production write access matches the approved roster.
Add reconciliation for data conversions
Every pipeline that converts, maps or aggregates financial data gets a reconciliation check that compares record counts, control totals and key balances between source and target. Exceptions are routed to a named reviewer and resolved with a documented rationale. Where a mapping table changes, the change follows the same approval path as code. The step is done when reconciliations have run against a known-good period and the team can show how a break would be detected, investigated and cleared.
Collect evidence for one control cycle
Over one full control cycle, the team gathers the artifacts management and auditors will ask for: the inventory and classification, change tickets with approvals, access reviews, job failure logs with resolution notes, reconciliation results and scheduler configuration exports. DeployOpen prepares the evidence and the control descriptions, but the assessment itself belongs to management and the independent auditor. The step is done when the control owner has reviewed a complete set and found no gaps against the agreed control design.
Hand over runbooks and ownership
The final step produces runbooks for routine operation, failure response, rerun approval, upgrades and access reviews, along with a named owner for each pipeline and each control. Upgrade planning for Airflow, Kafka or other components is written into the runbooks so that version support windows are tracked by the operating team. Handover is complete when the internal team has run a supervised failure and rerun drill and the control owners have accepted the documentation.

What auditors ask to see for pipeline controls

What evidence do auditors ask for to show batch job schedules and run history?

Auditors usually ask for the schedule definition, the record of each run, and proof that the schedule was not changed without approval. In Airflow, the scheduler and metadata database produce DAG run and task instance history, and the DAG files in Git show the schedule as approved. Ask your team to export run history for the audit period and to show how the schedule definition matches the version in the repository. Confirm with your platform team how long Airflow metadata is retained, because pruning can remove evidence before the auditor's sample period ends. Also confirm that the metadata database is PostgreSQL or MySQL rather than SQLite, since the production deployment guide warns that SQLite can lead to data loss.

Which alerts and reruns prove that pipeline failures were handled?

The evidence is an alert record for each failure, the rerun or backfill that followed, and a note showing who reviewed the failure and whether downstream data was corrected. Airflow produces task failure states, retry history and callback or notification logs, and your alerting tool (for example, the notification channel you configure) keeps the delivery record. Auditors will want to see that failures in jobs feeding financial reporting reached a named owner. Ask for a sample of failed runs with the ticket that tracked each one. Confirm which alerting channel is in use and whether its messages are retained outside the orchestrator.

How do change tickets link approvals to DAG and flow commits?

Auditors look for a change record that names the requester, the approver, the tested version and the commit that reached production. For Airflow DAGs, the Git repository holds the code history, the pull request holds the review and approval, and the change ticket references the merge commit and the deployment. Ask for a sample of changes and trace each one from ticket to pull request to commit to the deployed version. For NiFi flows, confirm whether flow versions are stored in a registry with change history. Make sure the approver is not the same person who wrote the change, and confirm that direct commits to the production branch are blocked.

What access reviews cover Airflow roles and Kafka ACLs?

Auditors expect periodic reviews that compare who holds each role against the approved list, with evidence of removals. For Airflow, the auth manager defines roles that control who can view, trigger, edit and manage connections, so the review should export role assignments and connection permissions. For Kafka, the StandardAuthorizer stores ACLs in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Ask for the ACL listing and confirm that this setting is false in the production configuration. Also review super user membership, since it bypasses ACL checks.

What reconciliation results prove that data conversion was complete and accurate?

The evidence is a reconciliation report that compares record counts, control totals and key balances between source and target for each conversion or load run. The orchestrator does not produce this on its own unless a reconciliation task is written and its output is stored. Ask the engineering team for the reconciliation task definitions, their outputs for sampled periods, and the procedure for investigating differences. Auditors also look at how exceptions were resolved and approved. Confirm whether reconciliation outputs are kept in an immutable or access-controlled location, and whether a reviewer signs off on them.

What version history does an auditor expect for the orchestrator and Kafka?

Auditors expect a record of which version of each platform ran during the period, when it was upgraded, and who approved the upgrade. For Airflow, the version support table matters: Airflow 2 entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes, so a version out of support is a finding waiting to happen. For Kafka, the supported releases listed on the downloads page are 4.3.1, 4.2.2 and 4.1.2, and rolling upgrades are supported from each release in the past year. Keep the upgrade tickets, the release notes reviewed, and the test results. Confirm the current version in production against these lists before each audit.

Can our own team produce this evidence, or does an implementation partner need to set it up?

Your team can produce most of this evidence once the controls are designed, but setting up the logging, retention, Git workflow and reconciliation tasks takes engineering effort that is easy to underestimate. DeployOpen plans, implements and documents self-hosted open-source platforms in your own infrastructure, and it prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports or attestations. Independent auditors perform the assessment. Ask any partner for a written scope that lists each control, the evidence it produces, the component that produces it, and who keeps it after handover. Confirm the retention period for run history with your auditors before the engagement starts.

What tooling alone does not satisfy

An orchestrator or streaming platform runs jobs, but it does not decide which controls exist, who owns them, or whether they worked during the period. The cards below separate what the software provides from what management, auditors and the operating team must establish and keep proving for financial reporting.

Control design stays with management

Airflow or Kafka can schedule, retry and log work, but they do not decide which pipeline outputs affect a reported balance or which check proves those outputs are complete. Management must map each feed to the financial assertion it supports and document the control that addresses the risk. SEC guidance allows management to limit its evaluation to the IT general controls needed for other controls to operate consistently, so write that scope down. Confirm the scope with your external auditor before the period starts.

Management's annual ICFR evaluation

Section 404 requires management to assess internal control over financial reporting each year, and Rule 13a-15(c) requires that evaluation as of each fiscal year end against a suitable framework. For pipelines, this means an accountable owner must state which scheduled jobs and flows feed reporting and conclude whether their controls work. Auditor attestation depends on filer status, so confirm with your auditor whether it applies to you. The evaluation is a management duty even when a partner builds the platform.

Unsupported orchestrator versions

Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes, so a financial feed running on it carries unpatched risk that an auditor will question. Limited-maintenance versions receive only security and critical bug fixes, which may not cover an issue that affects data accuracy. Record the version of every orchestrator, broker and connector in the period, and show how each upgrade was tested against the controls that depend on it. Ask your partner for a dated upgrade plan.

Operating effectiveness testing

Auditors test whether a control worked across the period by combining inquiry of personnel, observation of operations, inspection of documentation and re-performance of the control. For a failed load, this means someone asks the operator what happened, the auditor watches the alert and rerun, reads the ticket, and recomputes the reconciliation. Build the pipeline so each of these four methods has something to point to, such as a run history, an approval record and a reconciliation output. Ask your auditor which samples and periods they plan to test.

Retained evidence and access records

Tooling keeps logs, but retention, completeness and who can change them are organizational decisions. Decide how long run histories, change approvals, failure tickets and reconciliation outputs are kept, and whether they are stored outside the pipeline itself so an operator cannot edit them. Confirm that the audit trail of the metadata database and the Kafka authorization records can be exported in a readable form. Agree the retention period with legal and your auditor before the first period closes.

Review cadence and named owners

A control that exists on paper but is not reviewed on a schedule will not survive testing. Assign a named owner for each financial pipeline, a second reviewer for changes, and a recurring review of failed runs, manual reruns and access grants. Record who performs each review and when, and what they did when they found an exception. Ownership should survive staff changes, so keep it in a register that is updated with each change, not in a single engineer's head.

Questions buyers ask before hiring help with SOX pipeline controls

Can our own team build and control SOX-relevant pipelines in-house?

Yes, many teams do, provided they can show the controls operate consistently. Airflow, Dagster, Airbyte and Kafka are open source and can be run in your own infrastructure without a license fee, so the question is staffing and evidence rather than software. Check whether you have engineers who can own scheduling, failure handling, change approval for DAGs and flows, and access to production data, and whether those people are independent of the preparers of the financial reports. Management still owns the assessment under Section 404 of the Sarbanes-Oxley Act, so the internal team must be able to produce evidence an external auditor can test, not just working pipelines.

What makes the effort for SOX pipeline controls go up or down?

The main drivers are the number of pipelines feeding financial reporting, how much the reported numbers rely on each output, your current ITGC maturity, and where the platform is hosted. A small set of pipelines that feed a single close process is a different job from dozens of jobs crossing ledgers, subledgers and disclosures. Existing change management, access reviews and job monitoring reduce the work, while a Kafka cluster that carries financial events, or a shared Airflow instance used by many teams, adds scope for access, change and computer operations controls. Ask the partner to break the scope down by pipeline and by control area so you can see which items drive the price.

What do finance and IT need to provide before the work starts?

Finance needs to identify which reports, reconciliations and disclosures depend on each pipeline, and which of those are key controls in your risk assessment. IT needs access to source systems, current job definitions, scheduler and executor configuration, change tickets for recent DAG or flow releases, and a list of who can deploy and who can approve. Both sides should agree who signs off on failed-job remediation and backfills. Expect the engagement to need named owners on each side, because SEC guidance says the IT general controls relevant to ICFR depend on each company's facts and circumstances, which only your team can describe.

How long does a SOX pipeline controls engagement take?

Duration depends on the number of pipelines in scope, the state of your current change and access processes, and whether the platform needs upgrading first. DeployOpen does not publish durations, so the timeline should come from the written scope, which lists deliverables, dependencies and what your team must provide. Ask the partner to show the phases, the points where your finance and audit teams must review evidence, and which activities depend on your change windows. Also ask what happens if an upgrade or a fiscal year end falls inside the schedule.

Does DeployOpen issue audit opinions on our financial controls?

No. DeployOpen plans, implements, hardens and documents the systems and controls, and prepares evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Those come from your independent external auditor under the PCAOB standards, and management's own assessment is your responsibility. A useful engagement gives your auditor clear control descriptions, test populations, configuration exports and change records that they can inspect and re-perform. Confirm in the written scope exactly which evidence is delivered and who in your organization reviews it.

How does the Airflow 2 end of life affect our SOX scope?

Airflow 2 reached end of life on April 22, 2026, after which it receives no fixes, so any production pipeline still on Airflow 2 carries an unpatched platform inside a financial control environment. Moving to Airflow 3 changes the environment, and the upgrading guide describes changes that can affect DAGs, so the migration itself needs change approval, testing and evidence. Airflow 3 requires supported databases and Kubernetes versions, and the production guide recommends PostgreSQL or MySQL rather than SQLite. Ask the partner to inventory every DAG, connection and executor, to plan the migration as a controlled change, and to state how the old and new environments are compared before cutover.

How do Kafka upgrades change the controls we need to evidence?

Kafka upgrades affect scope because they change how the cluster is managed and secured. Kafka 4.0 removed ZooKeeper mode, so ZooKeeper-based clusters must migrate to KRaft before upgrading, and KRaft clusters older than 3.3.x are advised to move to 3.9.x first. Authorization in KRaft is stored in the cluster metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so access control reviews must cover these settings. Clients and brokers have different Java requirements, and the time-based release plan supports rolling upgrades from the last three releases. Ask for the upgrade path, rollback plan and the change records that will be retained.

What should a quote for SOX pipeline controls include?

A quote should follow from the written scope and list each component separately: discovery and control mapping, platform hardening, change process design, access and secret controls, job monitoring and failure handling, data conversion testing, evidence packaging and handover. Ask which items are one-time and which recur, such as access reviews, upgrades, testing or support retainers. Request the assumptions about pipeline count, sources, environments and your team's availability, and ask what changes the price if scope grows. DeployOpen does not publish a price list, so compare quotes on scope and deliverables rather than on totals alone.

How DeployOpen scopes, builds and supports SOX-relevant pipelines

DeployOpen plans, implements, documents and supports self-hosted open-source orchestration such as Airflow, Kafka and NiFi inside the client's own infrastructure. For SOX-relevant pipelines, the work starts with the financial reporting risks each job touches and the change, scheduling and data conversion controls that management needs to evidence.

How we scope

We list every pipeline that feeds financial reporting, with its sources, targets and the reports or balances it supports.
The written scope names the change approval, job failure and data conversion controls that will be configured and documented.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts; ongoing support is available as a retainer.

How we build

We deploy the orchestrator on the production database and executor versions that the vendor currently supports, not on end-of-life releases.
DAG and flow changes move through version control and a documented review path, with access to production limited to named roles.
Reconciliation checks, failure alerts and retry rules are built into the pipelines and tested against defined failure scenarios.

How we hand over

Handover includes runbooks for scheduling, failure handling, backfills and emergency changes, written for the operations team that will run them.
We document control design, owners and evidence locations so internal audit and external auditors can trace each control to its artifacts.
Access reviews, configuration baselines and backup procedures are delivered as part of the handover package, not left to be reconstructed later.

How we support

Upgrades to Airflow, Kafka and related components are planned against each project's published support windows and tested before production.
Security patches and dependency updates are applied through a documented change process that keeps the control evidence current.
Continuous testing and periodic control reviews are available under a retainer, and the client confirms the frequency and scope in the contract.

Open source data pipelines for SOX ITGCs