Why a data catalog for financial services starts with risk and regulatory data
A risk or finance lead at a bank, insurer or fintech is often asked to show how a regulatory report figure was produced, and the honest answer today usually depends on a few people who remember the logic. A self-hosted catalog such as OpenMetadata or DataHub, scoped to critical data elements, lineage and ownership, is the practical way to answer that question repeatably. The catalog records which source systems feed each element, which pipelines and transformations move it, who owns each step, and how it appears in the reports. The starting point is a short, agreed list of the data elements that matter most to the report, not an attempt to catalog the whole estate at once.
A catalog for risk and regulatory data differs from a generic catalog project because supervisors judge the evidence, not the tool. They look for a traceable path from a reported figure back to its sources, named owners who can explain definitions, and records of the quality checks that were run and what happened when they failed. The metadata itself is sensitive. Lineage graphs and sample values can expose cardholder data, customer data or confidential risk models, so the catalog and its search index should run in the organization's own infrastructure, with access limited to people who need it. Confirm with your supervisor and internal audit which evidence they expect before designing the catalog, since the requirements behind the BCBS 239 principles on data aggregation and reporting shape what must be traceable.
The platform choice matters less than the scoping work around it. Before any connector is configured, decide which critical data elements are in scope, which report lines depend on them, who is accountable for each definition, and how far lineage must go, from table to column to pipeline. Those decisions drive connector coverage, lineage capture and the glossary effort far more than the product does. The short answer on platforms: OpenMetadata and DataHub are both Apache-2.0 licensed and can run self-hosted, so the choice comes down to connector coverage for your warehouses, pipelines and reporting tools, the lineage features you can verify in a proof of concept, and the operating skills your team already has.
Which regulatory requirements shape the catalog build
This table maps the regulatory expectations most often raised in financial services reviews to concrete catalog decisions: scope, ownership fields, lineage depth and hosting location. Read each row as a question to settle with your supervisor, counsel and assessors before the build starts, since the catalog design follows from the answers. BCBS 239 is not covered by the facts we could verify here, so confirm its text and application with your supervisor and counsel.
| Requirement | What it means for the catalog | Evidence an assessor will look for | Where it usually fails |
|---|---|---|---|
| BCBS 239-style lineage and data quality (confirm text with supervisor and counsel) | Trace risk and regulatory report figures from the report back to source systems, with table and column lineage for critical data elements | Lineage graphs for named reports, critical data element definitions, owners, and data quality test results | Lineage stops at the warehouse, and manual spreadsheets between sources and reports are not catalogued |
| DORA register of information (Article 28(3)) | Record the ICT services and providers that support data pipelines and the catalog itself, linked to owners and critical functions | A register that maps each ICT arrangement to the systems, data sets and business functions it supports | The catalog holds data assets but not the vendors and services that process them |
| DORA contract terms (Article 30(2)) | Store the processing location and data return terms for each ICT service that handles regulated data | Contract metadata showing service locations, data processing regions and exit or data return clauses | Hosting location is recorded as a single value and not updated when providers or regions change |
| DORA exit strategies (Article 28(8)) | Flag data sets and pipelines that support critical or important functions so exit planning can find dependencies | Dependency lineage from critical functions to the data stores and tools that would need migrating | Dependencies are documented only in architecture diagrams that go stale |
| PCI DSS scope for cardholder data | Tag systems and columns that store, process or transmit cardholder data or sensitive authentication data, and mark which are in the cardholder data environment | Classification tags on in-scope tables, lineage showing flows into and out of the cardholder data environment, and named owners | Scope is inferred from column names only, so cardholder data in free text or logs is missed |
| SOX ITGC areas (SEC Release 33-8810) | Assign owners and change history to data and reports that feed internal control over financial reporting, covering program changes, computer operations and access | Ownership fields, change records for pipelines and definitions, and access request history tied to in-scope data | Owners are assigned at department level, so no one is accountable for a specific report or control |
| FTC Safeguards Rule (16 CFR 314), where applicable | Classify customer information for non-bank institutions under FTC jurisdiction and record who may access it and how access is approved | Classification labels on customer data, access policy references, and records of the Qualified Individual's approvals | Access policies live in the identity system and are not linked to the data assets they govern |
| Hosting location and sovereignty | Deploy the catalog, its database and search index in infrastructure the institution controls, and record where metadata about regulated data is stored | Architecture records showing deployment regions, network boundaries and data flows for catalog metadata | The catalog is hosted in a region or SaaS tenant that was never assessed under outsourcing rules |
Which financial data and systems the catalog must cover first
Start the catalog where supervisors and auditors will look first: the report figures and the critical data elements behind them, not the whole data estate. A typical financial institution holds this material across a general ledger, risk engines that calculate capital and liquidity measures, regulatory reporting marts, payments and card data, core banking extracts, and spreadsheets that sit between these systems and the final return. The spreadsheets are the part most often missing from inventories, and they are frequently where a figure changes hands without any system log. Confirm in each client environment which of these sources feed each regulatory return, which are owned by which team, and which are maintained by hand.
Choose the first scope with three filters. The first is every report submitted to a supervisor, traced from the submitted figure back to its inputs. The second is the set of critical data elements those reports depend on, with agreed business definitions and named owners. The third is the cardholder data environment, where the catalog should show which systems store, process or transmit cardholder data, since that boundary also determines the systems in scope for PCI DSS v4.0.1. Record the decision and the owner of each filter in writing before connectors are configured, because it sets the connector list and the lineage depth.
Column-level lineage matters because a report total is only defensible if each contributing column can be followed through transformations to a source system. Table-level lines show that a mart feeds a report, but they cannot show which field produced a capital figure or where a currency conversion or manual adjustment was applied. OpenMetadata supports column lineage and OpenLineage event integration, but whether your pipelines emit the events needed for that depth, and whether spreadsheets can be represented at all, must be tested on your own stack. Ask the implementation partner to run one report end to end as a pilot and document every hop that cannot yet be traced.
Which open-source catalog platforms fit a regulated financial estate
Four platforms and one adjacent category cover most catalog decisions in a regulated financial estate. The common question is whether the catalog must hold lineage, ownership and quality evidence for risk reporting, or whether it only needs to publish data. Each card names the fit condition and what to confirm before choosing.
OpenMetadata for governed risk data
OpenMetadata is Apache-2.0 licensed and maintained by Collate, which also offers commercial enterprise capabilities and premium support. It lists more than 130 connectors and supports table, column, dashboard, pipeline and metric lineage with OpenLineage events, plus glossaries, classifications, data quality tests and domains. A good fit is a team that needs column-level lineage from source systems to regulatory reports. Confirm that your core risk warehouse and report tools have supported connectors, and that the lineage you need can be captured from your pipelines.
DataHub for broad metadata graphs
DataHub is an Apache-2.0 metadata platform, currently at v1.7.0.1. It suits organizations that want a metadata graph across many domains, with ownership, tags and lineage fed by a wide ingestion ecosystem. Compare it with OpenMetadata on the features your stewards will use daily, such as glossary workflows, column lineage depth and data quality integration. Confirm the operating model for its search and graph stores, the upgrade path for your version, and who in your team will own the ingestion recipes.
Pipeline tools as lineage sources
Apache Airflow and Dagster orchestrate the jobs that move and transform risk data, and their run metadata is a primary source of lineage evidence. They are not catalogs, so they do not provide ownership, glossary or search for business users. Map which DAGs or assets produce each critical report figure, and confirm that your catalog can ingest their lineage events. Airflow is Apache-2.0 at 3.3.2, and Dagster is Apache-2.0 at 1.13.26.
Ownership and stewardship model
Any catalog is only as reliable as its owner assignments. Before selecting a platform, decide who is accountable for each critical data element, who approves glossary definitions, and who reviews quality failures for risk and regulatory datasets. Those roles must be entered in the catalog and linked to reporting controls. Ask any implementation partner how ownership is bootstrapped from existing data dictionaries and how stewards are trained and measured.
CKAN for published portals only
CKAN is a platform for publishing open datasets through a portal. It is a poor match for internal lineage, column-level ownership or report traceability, which are the core needs of a BCBS 239 program. Choose it only if the goal is a public or member-facing data publication site. Keep the internal catalog separate, and confirm how published datasets are kept consistent with the governed sources behind them.
Licensing and hosting checks
Confirm the license and support model before deployment, even for Apache-2.0 projects. Commercial enterprise features and premium support may come from the maintainer, and some extensions can have separate terms. Record the chosen version, the database and search services it depends on, and the hosting location in the register of ICT services and the contract terms that govern the platform. Validate that the deployment stays inside your own cloud account or data centers.
How hosting, outsourcing and supplier rules change the deployment
If the catalog will hold risk or regulatory metadata for an EU financial entity, the deployment decision is also a third-party decision. Under DORA Article 30(2), every ICT services contract must state where the services are provided and where data is processed, require advance notice of changes, and cover return of data on insolvency or termination of the provider. That applies whether the catalog runs in the institution's own cloud account or under a partner's operations. Before any build starts, confirm with the third-party risk team which legal entity holds the contract, which regions the metadata, lineage graph and sampled column values may touch, and how the search index and backups are handled. Lineage and profiling output can contain fragments of customer or counterparty data, so it belongs inside the same boundary as the source metadata.
The catalog also has to be recorded in the register of information under Article 28(3), which covers all contractual arrangements for ICT services. Article 28(8) adds a further test if the catalog supports a critical or important function, such as the evidence trail behind a BCBS 239 report. The institution needs a documented, tested exit strategy and transition plan, which in practice means being able to export entities, lineage, glossary terms and owner assignments in a usable form and to stand up a replacement. Using an Apache-2.0 platform helps with the exit story because no license has to be bought back, but the export path is still the institution's responsibility to prove. Ask the implementer to demonstrate a full export and a restore into a clean environment.
The technical footprint should be planned against the published prerequisites. OpenMetadata's bare metal documentation lists Java 21, PostgreSQL 15 or later (or MySQL 8.0.42 or later), and Elasticsearch 9.x or OpenSearch 3.x, with a minimum server of 2 vCPUs, 4 GB RAM and 20 GB storage, and at least 2 vCPUs, 2 GB RAM and 30 GB storage each for the database and search services. Those are floors, not production sizing. For production, the Kubernetes route is documented for AWS EKS, Azure AKS, Google GKE and on-premises clusters, which fits an institution that already runs a sanctioned cluster. Confirm which of these platforms your cloud and resilience standards approve, who holds the database and search administrator roles, and who is the named support contact for OpenMetadata releases. Check the OpenMetadata deployment documentation against the approved list before the architecture is fixed.
How a catalog rollout for a bank or insurer runs in phases
The sequence moves from scoping the reports and critical data elements in scope, through a platform build inside the institution's network, to evidence that internal audit and supervisors can test. Each phase closes with a named output so progress is visible to the data governance committee.
Scope reports and critical data elements
The team inventories the regulatory and risk reports in scope, identifies the figures each report publishes, and agrees which data elements are critical enough to carry definitions, owners and controls. Risk, finance, compliance and the data office sign off the list. Done means every in-scope report has a named list of critical data elements and a documented reporting chain that starts at the published figure and ends at the source system. Ask the partner to show how this inventory will be kept current when reports change.
Decide the platform and build the environment
The institution confirms whether OpenMetadata or DataHub fits its skills, hosting standards and upgrade appetite, then the partner builds the environment inside the bank's or insurer's own cloud accounts, data centers or private network. The build covers the metadata database, the search service, SSO, network controls, backups and secrets handling, following the prerequisites in the vendor's deployment documentation. Done means a hardened non-production environment passes a security review and restore test that the institution's own teams witnessed.
Connect sources and capture lineage for the first reporting chain
Engineers configure connectors to the source systems, warehouses, and pipeline orchestrators in the first reporting chain, then enable lineage capture at table and column level so each report figure can be traced back through transformations. Where pipelines emit OpenLineage events, those are wired in; where they do not, lineage is documented manually and flagged as lower confidence. Done means a reviewer can pick a sample figure from the report and follow an unbroken path to its source, with any gaps listed in writing.
Bootstrap the glossary and name data owners
Business and data stewards draft glossary terms for each critical data element, link them to the physical assets they describe, and record a named accountable owner and steward for every element. The partner runs working sessions to resolve conflicting definitions between finance and risk, and records each decision with its approver. Done means every critical data element has an approved definition, an owner who has accepted the role in writing, and a link to the assets that implement it.
Set data quality tests on critical data elements
Quality tests are written for each critical data element, covering completeness, validity, reconciliation to source totals and freshness, then scheduled through the catalog or the existing pipeline orchestrator. Thresholds are agreed with the data owner, and failures route to a named responder with a documented escalation path. Done means each test has a recorded threshold, an owner, a run history and a tested failure notification, and that the team can explain how a failed test affects the report.
Package evidence for internal audit and supervisors
The team assembles the evidence that internal audit and supervisory reviewers will ask for: the scope inventory, lineage exports for each reporting chain, ownership and approval records, test definitions and run logs, access review records for the catalog itself, and change records. Each item is mapped to the control or supervisory expectation it supports. Done means internal audit has walked through the package, has confirmed which items it can rely on, and has recorded any gaps for remediation. DeployOpen prepares this evidence, but it is the institution and its auditors who assess it.
Plan upgrades before the platform goes live
Upgrade planning starts before go-live because OpenMetadata 2.0 introduced breaking changes, including moving embedding credentials into a new configuration block, a new Databricks authentication format, rebuilt custom ingestion images for Python 3.12, and a Great Expectations migration to 1.3 that must be completed before upgrading. The partner documents each required migration step, tests it in a non-production copy, and agrees a change process with the institution's change advisory board. Done means a written upgrade runbook exists and has been rehearsed against the current release before the first production upgrade is scheduled.
What supervisors and auditors ask for and which catalog component produces it
This table maps the evidence that supervisors and auditors typically request for risk and regulatory data to the catalog component that produces it. Read each row as a question an examiner may ask, the artifact the catalog can export, and the gap to close before the request arrives. DeployOpen prepares this evidence, but auditors and supervisors decide whether it is sufficient.
| Evidence requested | Producing catalog component | What the reader should confirm |
|---|---|---|
| Lineage from a regulatory report figure back to its source systems | Lineage graph (table and column level) with OpenLineage events from pipelines | Which pipelines emit column-level lineage and which steps are still manual extracts or spreadsheets |
| Owner and steward for each critical data element | Glossary terms linked to owners, stewards and domains | That every CDE has a named accountable owner and a documented backup steward |
| Change history of CDE definitions and business rules | Glossary and asset version history, with audit logs of edits | Whether the retention period for catalog audit logs matches the records retention the institution applies |
| Data quality test results for CDEs feeding reports | Data quality tests, profiling and freshness checks stored against the asset | Which tests are control-grade, who reviews failures and how exceptions are approved and closed |
| Evidence of who could view or change catalog metadata | Role-based access control and SSO integration with audit logs | How access reviews are scheduled and whether the identity provider groups map to catalog roles |
| Classification of confidential and personal data in report sources | Classifications and tags applied to columns, with sample-based review | How tags were validated by people, not only by automated rules, and how false positives were handled |
| Register of information entries for ICT services supporting reporting | Custom metadata or domain records linking data assets to providers and systems (confirm with the catalog version in use) | Whether the register is maintained in the catalog or exported from a separate contract management system |
| Exports for examination requests on short notice | API and export of lineage, glossary and test results in machine-readable form | That export formats, scope and approval steps have been rehearsed before an examination begins |
What do financial institutions ask about catalog cost, effort and control?
What drives the cost of an open-source data catalog?
The software license is not the main cost. OpenMetadata and DataHub are both released under Apache-2.0, so the spend comes from four components: the infrastructure that hosts the catalog and its database and search services, connector and lineage setup, metadata curation and stewardship time, and ongoing operations. OpenMetadata's documented minimums for a server, database and search service are modest, but production sizing depends on how many sources you connect, how deep your lineage goes and how often profiling runs. Infrastructure is a recurring cost. Setup and curation are largely one-time at the start, then recur as the estate changes. Ask any partner to split its quote into these four lines so you can see who is paid for what.
How long does the first BCBS 239 lineage chain take to build?
No reliable duration can be stated before the scope is known, so treat any single figure with caution. The time depends on how many report figures you need to trace, how many source systems sit between each figure and its origin, whether those systems have connectors in your catalog, and whether pipelines expose lineage through OpenLineage events or must be documented by hand. A first chain for one regulatory report usually takes longer than later chains, because the team also agrees on critical data element definitions and owners. Ask the partner to name the report, the chain depth and the connectors in the first written scope, and to state which steps depend on your staff's availability.
What must the institution's own team provide for the catalog project?
The catalog cannot be populated without named people who know the data. The institution must provide access to source systems and their service accounts, a decision on who owns each critical data element, and business definitions reviewed by data stewards from risk and finance. It must also provide the network path, identity integration for single sign-on, and the hosting environment, whether that is its own cloud account, a data center or an air-gapped network. Ask the partner for a written list of roles, the number of hours each role is expected to give, and the approvals needed before lineage is treated as evidence for a supervisor.
Can an in-house team run an open-source data catalog without a partner?
Yes, a capable internal team can run OpenMetadata or DataHub, provided it has engineers who can operate Kubernetes or bare metal, manage the database and search services, and handle upgrades. OpenMetadata's 2.0 release notes list breaking changes, including embedding credential moves, a new Databricks auth format, Python 3.12 rebuilds of custom ingestion images and a Great Expectations migration before upgrading, so upgrade work is real and recurring. The harder part is usually stewardship: keeping owners, definitions and classifications current. Decide early which of these you will staff internally, which you will buy as a fixed-scope project, and which you will retain as ongoing support.
How do PCI DSS and DORA change the scope of a catalog project?
They add evidence and contract requirements rather than new software. PCI DSS applies where cardholder data or the cardholder data environment is in scope, so the catalog must show where that data is stored and flow it through lineage, and access to the catalog itself must be controlled. DORA, which has applied since 17 January 2025, requires a register of information on ICT contractual arrangements under Article 28(3), and Article 30(2) requires contracts to state where services are provided and data processed. A catalog hosted by a vendor therefore changes the register and contract review. Confirm with your compliance lead which requirements apply to your entity before scoping.
How should a bank compare build, partner and SaaS catalog options?
Compare the three options on the same written scope, because each one moves different costs. Building in-house puts the license at zero but places setup, upgrades, stewardship and on-call work on your staff. A partner-delivered fixed-scope engagement front-loads implementation and can be followed by a support retainer. A managed SaaS catalog, such as DataHub Cloud, shifts hosting and upgrades to the vendor but changes where metadata is stored and what your DORA register and exit plans must record. Ask each option for lineage depth, connector count, data residency, exit and data return terms, and the annual cost of stewardship time you will carry.
What evidence will supervisors and auditors ask the catalog to produce?
Supervisors and auditors usually ask to see which source feeds each reported figure, who owns each critical data element, how data quality checks are run and what happened when they failed. The catalog can hold lineage graphs, ownership records, glossary entries and test results, and OpenMetadata supports data quality tests, profiling and freshness checks. Preparing these records is not the same as certifying them. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any report or attestation. Ask your auditor which evidence formats they accept before the catalog is designed.
How are catalog changes controlled under IT general control expectations?
Treat the catalog as a system in scope for change control. SEC Release 33-8810 names program development, program changes, computer operations and access to programs and data as example IT general control areas for SOX Section 404 assessments, and a catalog that feeds finance reporting falls under similar expectations. Document who can change connectors, lineage definitions, ownership records and classifications, keep those changes in version control or an audit log, and separate production from test instances. Ask the partner how it records each change and who approves upgrades, since OpenMetadata releases arrive every one to two weeks in the 2.0 maintenance series.
What to ask an implementation partner before you sign
Start with how the partner maps critical data elements to reports. Ask them to show, for one regulatory report that you choose, the chain from the published figure back to its critical data elements, the owner of each element and the source tables or files behind it. A credible partner will walk through a worked example from your own environment, not a demo instance. Ask how they capture lineage from regulatory marts, and how they handle the parts of the chain that run through spreadsheets and manual adjustments, because those are usually where BCBS 239 traceability breaks. Ask which lineage events they emit from your pipelines (for example, through OpenLineage) and what they will do about steps that produce no machine-readable lineage at all.
Next, ask how they keep cardholder data and customer data out of the catalog's sampled content. Profiling and sampling can copy field values into the catalog, so the partner should state which sampling settings they will use, which columns they will exclude by pattern or tag before any profiling runs, and who approves that exclusion list. Ask where the catalog is hosted, which accounts, regions and networks it runs in, who holds administrative access to the database and search services, and whether the partner's own support staff ever see production metadata. Confirm these answers against the PCI DSS scope your organization has already defined, since the catalog may fall inside that scope if it stores or processes cardholder data.
Then cover change and exit. Ask how they will handle upgrades of OpenMetadata or DataHub, including how they test a new release against your connectors and custom ingestion, how they manage breaking changes listed in the release notes, and who owns rollback. For DORA, ask for the contract terms that cover where services are provided and data processed, advance notice of changes, data return on termination or insolvency, and a documented, tested exit plan. Finally, ask what evidence they hand over to internal audit: configuration baselines, access reviews, ingestion and lineage run records, change logs for the catalog, and the test results behind each control claim. Ask them to name the evidence they will not produce, because an auditor will ask for it anyway.
