Why data pipelines for financial services need more than a scheduler
A risk or finance team whose regulatory reports depend on batch jobs that nobody can fully trace has a problem that a scheduler alone will not solve. Self-hosted Apache Airflow or Dagster, paired with Kafka or NiFi for streaming and controlled transfer, can carry these pipelines in the organization's own infrastructure, but lineage, reconciliation, change control and scheduling evidence must be designed in from the first sprint. Those controls are what turn a working pipeline into one a supervisor or external auditor can rely on. General analytics pipelines tolerate a late refresh or a silent schema change. Supervisory reporting does not: it carries fixed submission deadlines, the risk of restating figures already filed, and often data residency rules that decide which regions and tools may touch the records.
The reference most supervisors use for this problem is BCBS 239, the Basel Committee's principles for effective risk data aggregation and risk reporting. In plain terms, it asks a bank to produce accurate, complete and timely risk data that can be traced back to its sources, reconciled to the books and reported under stress or on demand. For a pipeline team, that translates into concrete questions: can every reported figure be traced from the report to the transformation to the source extract, do totals reconcile to the general ledger on each run, and who approved the last change to the DAG or flow that produced the number. Confirm with your own risk and compliance function which principles and internal policies apply to your entity, because the answer varies by jurisdiction and charter.
Start by deciding which pipelines feed reports in scope, then record for each one its owner, its schedule, its deadline, its reconciliation checks and its change approval path. DeployOpen configures the platform, lineage capture, reconciliation jobs and logging to produce that evidence, and documents how it is retained. It does not issue audit reports, attestations or certifications, and it does not determine whether your reporting meets BCBS 239 or any other requirement. Independent auditors and supervisors make that judgment, so the useful test is whether your evidence would answer their questions without a rebuild, and that test should be run before the first regulatory cycle depends on the platform.
Which regulatory duties shape the pipeline build
This table maps each regulatory duty that commonly touches financial data pipelines to what it means for the build, the evidence an assessor will ask for, and where teams usually fall short. Read each row as a design input to settle before the first DAG or connector is deployed, not as a legal opinion on your obligations.
| Duty | What it means for the pipeline build | Where it usually fails |
|---|---|---|
| DORA Article 28(3) register of ICT contracts | Every orchestrator, connector, Kafka cluster and hosting provider used by the pipelines needs an entry in the register of ICT arrangements, so the platform components must be named and mapped to the functions they support. | Open-source components are installed by the data team and never added to the contract or service inventory. |
| DORA Article 28(8) tested exit strategy | If pipelines support a critical or important function, the team needs a documented, tested plan to move workloads, DAG code, metadata and Kafka topics to another platform, including a dry run. | Exit plans exist on paper, but nobody has restored Airflow metadata or replayed a topic elsewhere. |
| DORA Article 30(2) data location and data return | Contracts with service providers must state where data is processed and stored, and how data returns on termination. For self-hosted pipelines, record the hosting regions, backup locations and any support access from abroad. | Managed connector or ELT vendors process data in locations nobody recorded, and return terms do not cover staging copies. |
| SEC Release 33-8810 IT general control areas | For ICFR, pipelines that feed reported numbers fall under program development, program changes, computer operations and access to programs and data. Build change approval for DAGs, flows and connector configs, job scheduling controls and privileged access into the platform from the start. | Pipeline code changes go straight to production, and scheduler or warehouse admin rights are shared across the data team. |
| FTC Safeguards Rule, 16 CFR 314.4(c)(5) | For non-bank institutions under FTC jurisdiction, multi-factor authentication is required for anyone accessing an information system holding customer data, unless the Qualified Individual approves an equivalent control in writing. Apply this to Airflow, Dagster, NiFi, Kafka consoles and warehouse access. | Service accounts and admin UIs are excluded from MFA, and no written exception from the Qualified Individual exists. |
| PCI DSS v4.0.1 cardholder data flows | If card numbers or sensitive authentication data pass through ingestion, staging, logs or backups, those components fall in scope. Minimize or tokenize fields early, restrict where raw payloads land, and confirm which systems the scope boundary covers. | Full card numbers appear in Kafka topics, NiFi provenance records or Airflow task logs that were never scoped. |
| Swift CSCF v2026 payment messaging | Where pipelines ingest or replicate Swift message data, the environment must align with the controls the Swift Customer Security Controls Framework sets for the local Swift infrastructure, including annual independent assessment and attestation. Confirm with the Swift program which controls apply to the replicated data. | Message archives are copied into analytics stores that sit outside the controls assessed for the Swift environment. |
Which open source platforms fit financial reporting pipelines
Risk, regulatory and finance reporting pipelines usually combine a scheduler for batch close cycles, a transport layer for event flows, and a lineage catalog that lets reviewers trace every reported figure back to its source. The platforms below cover those roles, and each one carries licensing, version and operating decisions to settle before go-live.
Apache Airflow for scheduled reporting
Airflow fits batch reporting: daily position files, month-end close jobs and regulatory extracts that must run on a fixed calendar with retries and alerts. Its Apache-2.0 license has no paid edition from the project itself. Production use needs PostgreSQL or MySQL as the metadata database, since SQLite is for testing only. Multiple schedulers can run at once through database row-level locks, which helps resilience. Use KubernetesExecutor where isolating each task matters for audit. Confirm which Airflow 3 patch line your support model covers, because Airflow 2 has reached end of life.
Dagster for asset-based lineage
Dagster models pipelines as software-defined assets, so each reported table or file maps to the code and upstream assets that produce it. Materialization history records when each asset last ran and with what inputs, which supports reconciliation questions from finance and internal audit. Dagster is Apache-2.0 licensed. It suits teams that want lineage and testing built into the orchestrator rather than added later. Check whether your existing DAG estate can move to assets without rewriting controls, and confirm how run history is retained in your own storage.
Apache Kafka for streaming positions
Kafka fits streaming flows such as trade events, payment status changes and intraday risk feeds, where reporting needs data in near real time rather than after a nightly batch. Only KRaft mode is supported in the 4.x line, so ZooKeeper-based clusters must migrate first. Supported releases listed by the project are 4.2.2 and 4.3.1. Secure clients with SSL/TLS and SASL, then restrict topics with ACLs, because resources without ACLs are limited to super users unless you change that default. Plan upgrades from the last three releases.
Apache NiFi for controlled transfer
NiFi suits moving files and records between network zones, such as from a core ledger into a reporting enclave, where each hop must be recorded. Its data provenance feature tracks where each record came from and what happened to it, which supports evidence requests from supervisors. Confirm the NiFi version, the provenance repository sizing and the retention you need before choosing it. Pair it with Airflow or Dagster when scheduling depends on upstream completion, and document who can change flows, since flow edits are a change-control event.
Airbyte for connector ingestion
Airbyte covers connector-based extraction from SaaS tools, core banking exports and databases into a warehouse such as ClickHouse. Its platform repository is licensed under the Elastic License 2.0, which restricts providing the software as a managed service to third parties. Your in-house hosting may be permitted, but confirm the terms with counsel before adopting it for regulated reporting. Also verify connector maintenance, schema change handling and the credentials each connector stores, since these affect both control design and the evidence you can produce.
DataHub or OpenMetadata for lineage and ownership
A metadata catalog records which dataset feeds which report, who owns each dataset and which controls apply to it. DataHub and OpenMetadata are both Apache-2.0 licensed and cover discovery, lineage and governance across the estate. This catalog is the artifact that answers supervisor questions about data origin and transformation. Compare how each tool captures lineage from your orchestrator and warehouse, and confirm that lineage collection will not depend on a SaaS connector. Assign data owners before rollout, because ownership gaps surface quickly in an examination.
Where financial data pipelines can run and what each setting changes
Running Airflow, Kafka and the warehouse in the institution's own data center keeps data residency under direct physical and administrative control. Ingestion, staging and transformation stay on infrastructure the institution already audits, and no third-party hosting provider sits in the data path. The trade-off is that the institution owns capacity planning, hardware lifecycle and high availability design for every component.
- Residency is a settled fact: Source extracts, intermediate files and warehouse replicas sit in named racks or sites, which makes the location statement under DORA Article 30(2) simple to document for any remaining vendor support.
- Metadata database choice: Airflow needs PostgreSQL 14 to 18 or MySQL in production, and SQLite is unsuitable, so the database cluster becomes a core system with its own backup and restore tests.
- Scheduler and worker capacity: Plan for at least 4 GB of RAM recommended by the Airflow prerequisites for each component, and for Kubernetes 1.30 to 1.35 if the executor runs there.
- Separate environments in the same facility: Production, test and development pipelines should use separate clusters, service accounts and network segments, so a test DAG cannot read production source credentials.
- Secrets in an internal vault: Store connection credentials in a secrets backend the institution operates, rotate them on a documented schedule, and keep them out of DAG code and version control.
Confirm with the infrastructure team where each replica, backup and log store physically sits before the register of information is updated.
A private cloud account under the institution's own tenancy, or a dedicated account in a hosting provider's cloud, separates the platform from shared infrastructure while keeping the institution as the account owner. Residency depends on the region chosen and on every managed service the stack uses, such as object storage, managed databases and backups. Contract terms with the hosting provider then become part of the pipeline's compliance file.
- Region pinning: Restrict deployments to approved regions with cloud policy controls, and check that backups and replicas do not cross into other regions.
- DORA Article 30(2) terms: The hosting contract must state where services are provided and data is processed, require advance notice of location changes, and cover data return on insolvency or termination.
- Exit planning: Article 28(8) expects a documented, tested exit strategy for ICT services supporting critical or important functions, so keep the stack portable on Kubernetes and export-ready for Kafka topics and Airflow metadata.
- Account boundaries: Separate production and non-production into different accounts or subscriptions, with independent identity providers and key management.
- Key and secret management: Use keys held in a key service controlled by the institution and avoid default credentials for managed databases.
Ask the hosting provider for the written list of regions, subprocessors and support access paths before the first pipeline runs in the account.
Air-gapped and restricted zones run ingestion and orchestration with no direct internet route, so connector images, Python packages and Kafka and Airflow releases must be brought in through a controlled transfer process. Residency is at its strongest because data cannot leave the enclosure by design. The cost is slower updates, manual security patching and monitoring built without SaaS tools.
- Controlled update path: Mirror container images, Python dependencies and vendor releases into the enclosure, and record each transfer with checksums and approvals so the change trail is auditable.
- Zone-to-zone movement: Move data between enclosures through a controlled gateway, with NiFi or a similar tool logging every transfer and its source.
- Restricted secrets handling: Keep credentials in an offline or enclosure-local secrets store, and prohibit copying them into transfer media without a documented process.
- Production isolation: Restricted production zones should not share build or test infrastructure with lower classifications, so test data does not carry production records.
- Local monitoring: Alerts for failed DAGs, lag on Kafka consumers and database health must reach staff inside the zone without calling external services.
Confirm with the security team which transfers are permitted, who approves them, and how DORA exit and data return obligations apply when the zone's services are supported by an outside partner.
How a financial pipeline rollout moves from one report to the full estate
The sequence starts with one priority report and its data elements, proves lineage and reconciliation on it, and then extends the same controls across the estate. Each phase closes with a named output that risk, finance and compliance owners can review before the next phase begins.
Inventory reports and data elements with owners
The client's risk, finance and regulatory reporting teams list every report in scope, the data elements each one uses, the source systems that supply them, and a named business owner and technical owner for each element. DeployOpen works through this with the teams and records gaps, such as elements with no clear owner or sources that exist only in spreadsheets. The step is done when the inventory is signed off by the report owners and the scope for the first report is agreed in writing.
Design lineage and reconciliation for each report
For the first report, the team maps each data element from source to reported figure, documents every transformation, and defines the reconciliation checks that tie totals back to source balances or to the general ledger. Controls such as completeness counts, threshold checks and exception handling are written down with the owner who will review them. The output is a lineage map and a reconciliation specification that the report owner accepts, which becomes the reference for testing later.
Build the platform on supported versions
The orchestration and streaming components are installed on versions that still receive fixes. Airflow 2 reached end of life on April 22, 2026, so new builds use Airflow 3, currently at 3.3.2, on a PostgreSQL or MySQL metadata database rather than SQLite. Where Kafka is part of the design, the cluster runs in KRaft mode because Kafka 4.x supports only KRaft, and the team confirms the supported release line before installation. The output is a deployed environment with a version record, a tested backup and restore procedure, and a confirmed upgrade path.
Build and test the pipelines for the first report
Engineers implement the pipelines as code, with version control, peer review and automated tests on the transformations and reconciliation checks. Failure alerts, retry rules and backfill procedures are defined so that a missed or partial run is visible and recoverable. The team confirms that the outputs match the reconciliation specification in a test environment using representative data. The step closes when the test evidence and the run logs are filed with the report owner.
Run in parallel with the existing report
For a defined number of reporting cycles agreed in advance with the report owner, the new pipeline runs alongside the current process, and both outputs are compared line by line. Every difference is logged with its cause, whether a data timing issue, a rounding rule, a source correction or a defect, and each one is either fixed or accepted by the owner in writing. The output is a difference log and a parallel-run sign-off that shows the new report is ready to replace the old one.
Switch on change approval and access controls
Before the pipeline becomes the system of record, the team enables change approval for DAGs, flows and transformation code, separates development, test and production environments, and restricts production access to named roles. Credentials move into a secrets store, and privileged actions such as manual reruns and backfills are logged. The output is an access matrix, a change approval record and evidence that the controls operate, which the compliance lead reviews against the organization's IT general control expectations.
Assemble the supervisor or auditor walkthrough pack
The team compiles one pack per report that contains the inventory entry, lineage map, reconciliation specification, test results, parallel-run log, access matrix and change records, with a short guide that explains how to trace one figure from the report back to its source. The pack is walked through with internal audit or compliance before any external review. The output is a reviewed pack that the client can hand to auditors or supervisors, and it sets the template for extending the same steps to the next report in the estate.
How upgrades and DAG changes stay controlled
Version currency is an ICT risk question, not a housekeeping task. Airflow 2 (current patch 2.11.2) entered limited maintenance on October 22, 2025 and reached end of life on April 22, 2026, after which it gets no fixes. Limited-maintenance versions receive only security and critical bug fixes, so a finance pipeline still on Airflow 2 is carrying known risk with a shrinking set of remedies. On the Kafka side, rolling upgrades are supported from each release in the past year to the latest version, and Kafka 4.0 removed ZooKeeper mode entirely. Clusters still on ZooKeeper must migrate to KRaft before upgrading to 4.x, and the upgrade notes recommend moving KRaft clusters older than 3.3.x to 3.9.x first. Brokers and Kafka Connect require Java 17 from 4.0 onward. Confirm the JVM baseline on every broker and Connect host before scheduling any upgrade window, because a Java gap will block the change at the worst moment.
Map each control to the ITGC area an auditor will test. The SEC's guidance on management's ICFR report (Release 33-8810) names program development, program changes, computer operations and access to programs and data as the IT general control areas for SOX Section 404 work. A DAG or flow change should therefore carry a change ticket that links to a reviewed commit, a peer review by someone who did not write it, and proof that it ran in a separate test environment with the same Airflow and Kafka versions as production. Production deployments should go through a restricted promotion path, with access to deploy limited to named roles, so that program change and program access evidence can be pulled from the ticket system and the Git history without reconstruction. Ask your partner to show how a sample change moves from ticket to merge to deployment, and which system holds each record.
Treat upgrades and DAG changes as the same change category. Schema or executor changes in Airflow 3, such as the removal of the hybrid CeleryKubernetesExecutor and LocalKubernetesExecutor as of Airflow 3.0.0, can alter how a reconciliation job runs, so they need the same test evidence as a DAG edit. Record the before and after versions, the test results for a reconciliation run against a known good dataset, and the rollback steps. Reviewing the Apache Airflow upgrading guide for Airflow 3 and the Apache Kafka 4.3 upgrade notes against your own DAGs and topic configurations is the natural first step before a change request is raised.
Which evidence auditors and supervisors ask for and which component produces it
This table maps the evidence that auditors and supervisors typically request for financial reporting pipelines to the platform component that produces it. Read each row as a question the assessor will ask, followed by the artifact the pipeline stack can generate. The main takeaway is that DeployOpen prepares this evidence, while the institution's auditors and supervisors decide whether it is sufficient.
| Evidence area | Component that produces it | What to confirm before the assessment |
|---|---|---|
| Lineage graphs and metadata | DataHub or OpenMetadata for cross-system lineage, ownership and glossary terms; Dagster asset history for materialization records | Whether lineage is captured at column level for each report, and how manual spreadsheet steps are documented outside the catalog |
| Reconciliation reports and exception logs | Orchestrator task results plus control totals (row counts, sums, hash comparisons) written to a reconciliation table each run | Who signs off exceptions, the threshold that triggers investigation, and how breaks are retained for the supervisor's review period |
| Run history and failure handling records | Airflow task logs, DAG run history and alert notifications, retained in the self-hosted metadata database | Retention period for run logs, whether the metadata database is PostgreSQL or MySQL rather than SQLite, and how backfills are recorded |
| Access and ACL records | Kafka StandardAuthorizer ACLs stored in the cluster metadata log; role-based access controls in the orchestrator and catalog | Quarterly access review process, who can edit ACLs, and whether super user access is logged and limited |
| Change approvals for pipelines | Git commit history, pull request approvals and deployment records for DAGs, assets and Kafka configuration | Whether production deployments require a second approver outside the author's team, and how emergency changes are reviewed afterward |
| Vendor contract register inputs for DORA | Inventory of hosting locations, data flows and open-source component versions maintained by DeployOpen for each pipeline | Contract clauses on data location, advance notice of changes, data return and exit planning, which remain the institution's legal responsibility |
What financial institutions ask before building data pipelines on open source
How much does a financial data pipeline build cost and what drives it up?
The build cost follows from the written scope, and no fixed figure applies to every institution. The main cost components are the internal team time (data owners, report owners, risk and finance reviewers), the implementation effort from a partner or in-house engineers, and the infrastructure for production, test and disaster recovery environments. The biggest drivers are the number of source systems, the depth of reconciliation to ledgers and regulatory returns, the number of environments, and whether an on-call rota must cover overnight reporting deadlines. Ask any quote to split one-time build work from recurring costs such as compute, storage, upgrades, patching and support retainers. Ask for the assumptions behind each line so you can see what changes if the source count grows.
How long does a pipeline for regulatory reporting take to deliver?
Duration depends mostly on your approvals and data access, not only on engineering effort. Before build starts, the written scope should name the reports in scope, the sources, the reconciliation rules and the acceptance tests that finance and risk will sign off. Timelines usually stretch when change board approvals, network access requests, data owner sign-offs or security reviews of the platform queue up behind the engineering work. Ask the partner to show which milestones depend on your side and what happens if an approval arrives late. Engagements are typically fixed-scope, so agree the schedule and the dependencies in writing before work starts.
What must a bank or insurer supply before a pipeline project starts?
The institution must supply named data owners for each source, written report specifications, and access to source systems, test data and target environments. It also needs a change board that agrees how DAGs, flows and transformation logic are approved and promoted to production. Other inputs include the existing control inventory, the list of sensitive fields and where they may flow, and a decision on who holds privileged access to the orchestrator and its secrets. Without these, reconciliation rules remain ambiguous and acceptance testing stalls. Confirm these inputs with your own control owners before scoping, because the required approvals differ by jurisdiction and entity type.
Can an internal team run a financial data pipeline without an outside partner?
Some institutions can, provided they have skills across the platform and the operating model. A capable internal team needs experience with the orchestrator, such as Apache Airflow 3.3.2 or Dagster 1.13.26, plus database administration for the metadata store, Kafka operations if streaming is involved, Kubernetes or VM operations, secrets management and data engineering against your reporting logic. Teams also need a run model for on-call, upgrades and backfills. Partners are used to fill gaps in these skills, run fixed-scope implementations, or provide retainers for upgrades and continuous testing. Assess your team against those skills honestly and decide which parts you keep in-house.
Does an open source data pipeline make us compliant with DORA and BCBS 239?
No. Open source software does not certify compliance with DORA or BCBS 239, and neither the platform vendor nor DeployOpen issues an attestation for it. DORA, Regulation (EU) 2022/2554, applies to covered financial entities from 17 January 2025 and requires a register of ICT contractual arrangements, contract terms covering data location and data return, and tested exit plans for services that support critical or important functions. BCBS 239 expects accurate, complete and timely risk data with documented lineage and reconciliation. The institution owns the obligation and its evidence. A pipeline can support that evidence through documented lineage, controlled changes, tested recovery and reconciliation logs, which your internal audit and supervisors will then assess.
Do AI components in our reporting pipeline trigger EU AI Act logging duties?
It depends on whether the AI system is high-risk under the EU AI Act and on how it is used, so confirm the classification with legal and compliance first. If an AI system is high-risk, Article 12(1) requires that it technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Most reporting pipelines use deterministic transformations rather than AI, but a model that classifies transactions or drafts commentary could fall into scope. If so, design log capture, retention and access at the start, and check how the logs interact with your records retention rules.
Which Airflow and Kafka settings should a finance team verify before go-live?
Check the components that a supervisor or auditor will test first. For Airflow, confirm that the metadata database is PostgreSQL or MySQL rather than SQLite, that the version is still supported (Airflow 2 reached end of life on April 22, 2026), and that scheduler redundancy and executor choice match your deadlines. For Kafka, confirm that the supported release is in use, that TLS and SASL are enabled on every listener rather than mixed with unsecured ones, and that ACLs in the KRaft metadata log restrict topics to named principals, since resources without ACLs are restricted to super users by default. Write these checks into the acceptance criteria so the evidence exists before the first regulatory cycle.
What to ask a pipeline partner before signing a fixed-scope agreement
Start with lineage and reconciliation, because those are the controls a supervisor will test first. Ask the partner how each pipeline records field-level lineage from source extract to the final report table, and where that lineage is stored: in the orchestrator metadata, in a catalog such as DataHub or OpenMetadata, or in a separate control table. Ask for the reconciliation design in writing: which control totals are compared at each stage, who owns the tolerance thresholds, and what happens to a run that fails reconciliation. Ask how the partner will demonstrate that a regulatory figure can be traced back to its source records without relying on a person's memory of how it was built.
Next, ask how DAGs and transformations are tested and rolled back. Request the promotion path from development to production, including who approves a change to a DAG file and where that approval is recorded. Ask whether the partner runs data tests against a fixed sample of production data, how a failed release is reverted, and whether a previous version of a transformation can be restored without rerunning unrelated jobs. For backfills of restated data, ask who authorizes a rerun of a closed reporting period, how the old and new outputs are compared, and how the rerun is logged. Illustratively, a restated risk exposure should produce a documented before and after comparison, not an overwritten table.
Then cover platform lifecycle and exit. Ask which Airflow and Kafka versions the partner would deploy today and when each reaches the end of its support window. Airflow 2 reached end of life on April 22, 2026, so ask how any legacy DAGs will be migrated and tested, and how the partner plans upgrades within the supported releases listed for Kafka. For DORA, ask for the contractual data location, the register entries the partner will supply, and a tested exit plan that names the data return format and the steps to move pipelines to another operator. Finish by asking what evidence is handed over at closure: the runbooks, the lineage and reconciliation records, test results, change approvals and access reviews, so your own team and any auditor can verify the pipelines without the partner in the room.
How DeployOpen scopes, builds and supports financial data pipelines
The sequence moves from a written scope and a mapped set of reporting flows to a deployed, documented pipeline estate with an evidence pack that your own auditors and supervisors can examine. Each step closes with a named output so you can check progress before the next one starts.
Agree the written scope and reporting flows
DeployOpen and your team list the in-scope reports, risk and finance data domains, source systems, destinations and the regulatory drivers behind each flow, such as BCBS 239 lineage expectations, DORA register entries or SOX-relevant data conversions. The output is a signed scope document with deliverables, exclusions and acceptance criteria. Quotes follow from this document, so ask for it to be detailed enough that a second partner could price the same work.
Map sources, lineage and critical data elements
Working with data owners, DeployOpen maps each source extract, transformation and report output, including the critical data elements that feed regulatory and financial statements. The mapping records owners, refresh schedules and where sensitive data, including cardholder data where PCI DSS applies, may flow. It is complete when every in-scope report can be traced back to named sources, and gaps are logged as open items with owners.
Design the architecture and control points
DeployOpen produces the reference architecture for the chosen orchestration and ingestion stack, such as Airflow, Dagster, Airbyte, Kafka or NiFi, along with the warehouse, metadata catalog and network zones. The design sets out where credentials, intermediate files and logs are stored, which control points gate each transformation, and how deployments are approved. It is done when your architecture and security teams have signed off the design and the exceptions it records.
Deploy the platform in your infrastructure
DeployOpen installs the self-hosted platform in your cloud accounts, data centers or air-gapped network, applying the hardened configuration from the design. For Airflow, this includes a production-grade metadata database rather than the SQLite default, which is for testing only. For Kafka, it includes authentication, TLS and authorization configured before any production data arrives. The output is a running environment with access records and a configuration baseline stored under version control.
Configure lineage, reconciliation and data quality
DeployOpen builds the pipelines, lineage capture into the catalog, reconciliation checks between source and reported totals, and data quality tests that run on every load. Reconciliation breaks are routed to named owners and cannot be silently overwritten. The step is complete when test runs show lineage and reconciliation results for each in-scope report and your finance and risk owners have reviewed sample outputs.
Test changes and failure handling
DeployOpen tests scheduling, failure and retry behavior, backfills and change approval for each pipeline, using the same promotion path your change process will use in production. The tests record who approved each DAG or flow change and whether failed jobs alert the right owner before a reporting deadline. The output is a test record that maps to your change and job-scheduling control expectations.
Hand over runbooks and the evidence pack
DeployOpen delivers runbooks for operations, incident response, backfills, restores and exit or transition planning, plus an evidence pack covering configurations, access reviews, test results and lineage records. DeployOpen prepares this evidence for your compliance and audit teams but does not issue audit reports, attestations or certifications, which independent auditors and certification bodies provide. It is complete when your operations team has run the core runbooks in a supervised exercise.
Support through optional retainers
After handover, you can choose a retainer covering platform upgrades, patching, monitoring and continuous testing of the pipeline estate. Because Airflow and Kafka publish support windows, the retainer scope should state which versions are covered and how upgrades are planned before a line goes end of life. Ask the partner to confirm in writing what the retainer includes, how security findings are reported, and how upgrade testing is evidenced.
