Why an open source data catalog starts with knowing who owns your data
Most regulated organizations cannot answer two basic questions about their data: who owns a given dataset, and where personal data actually flows. Data sits in warehouses, lakes, operational databases and SaaS tools, often with no shared inventory, so the answers live in the heads of a few engineers. DataHub and OpenMetadata are the two main open source options for closing that gap, and both are licensed under Apache-2.0 and can be self-hosted inside your own cloud account, data center or air-gapped network. The choice between them is less about features on a slide and more about your operating model. DataHub is a strong fit where a platform team will run a sizable ingestion estate and wants to build event-driven metadata pipelines around it. OpenMetadata tends to suit teams that want a more opinionated out-of-the-box experience for ownership, glossary, data quality and governance workflows. Confirm both against your own connector list and your team's skills before committing to either. Treat the catalog as a governance tool, not a search box. A search index tells analysts that a table exists. A governed catalog records the owner accountable for each asset, the business glossary terms it maps to, its classification (for example, whether it holds personal data), and the lineage that shows how that data was derived and where it was copied. Those records are what a privacy team needs to respond to access requests, to scope a data protection impact assessment, or to show an auditor how a data subject's information moves across systems. Before deployment, decide who may assign ownership, who approves classification changes, and how stale ownership is escalated when a person leaves the organization. Those decisions, not the search interface, determine whether the catalog produces usable evidence.
Which requirements to agree before choosing a catalog
A catalog only works when the organization agrees on what it covers, who owns each record, and how sensitive data is labeled before anyone installs DataHub or OpenMetadata. The cards below list the decisions that shape scope, effort and evidence, and name who should sign off on each one.
In-scope sources and connector coverage
List the databases, warehouses, BI tools, streaming systems and file stores the catalog must describe in the first phase, then check each one against the connectors shipped with DataHub or OpenMetadata. Gaps usually mean custom ingestion code that someone must write, test and maintain. Ask the data platform lead to confirm the list, and ask the implementation partner to document which connectors are native, community-built or custom. Source owners should approve their own systems being scanned.
Metadata ownership model
Decide whether ownership sits with the data producing team, a central data office, or a mix in which domains own their assets and the central team owns standards. Every table, dashboard and pipeline in scope needs a named owner before the catalog goes live, or stale records will accumulate. The CDO or data governance lead should sign off, with input from each domain head.
Classification scheme for personal and sensitive data
Define the labels the catalog will apply, such as personal data, special category data, confidential financial records and public data, and write a short rule for each. Labels must be applied consistently across systems or they will not support access reviews or data protection evidence. The privacy officer and security lead should approve the scheme, and legal should confirm it matches the categories used in data processing records.
Lineage depth
Decide how far lineage must reach: table-level only, column-level for regulated reports, or end-to-end from source system to dashboard. Deeper lineage requires more parser coverage and more pipeline metadata, which raises both setup and upkeep effort. Ask the head of analytics engineering to name the reports that need column-level trails, and confirm with the implementation partner which orchestrators and transformation tools can emit lineage automatically.
Identity and SSO integration
Choose how users sign in, which groups map to catalog roles such as viewer, editor and admin, and how group membership is synchronized from the directory. Without this, access reviews happen in two places and leavers keep access. The identity lead should approve the role model and confirm which protocols the chosen catalog supports in the version being deployed.
Hosting location and data residency
Decide where the catalog and its metadata store run, which regions and accounts may hold metadata about personal data, and whether air-gapped access is needed. Metadata itself can reveal sensitive information, such as schema names that expose customer segments. The CISO and the data protection officer should approve the location, and the architect should document where backups and search indexes are kept.
Curation responsibilities
Decide who writes descriptions, approves glossary terms, resolves duplicate assets and retires obsolete ones, and how much of that work automation can handle. Curation is the recurring cost that most often decides whether the catalog stays accurate after launch. The data governance lead should assign curators per domain and agree a review cadence with the domain heads.
DataHub or OpenMetadata: which platform fits which team
This table compares DataHub and OpenMetadata, both Apache-2.0 metadata platforms, on the factors that usually decide a shortlist. Read it as a list of questions to test in a proof of concept, not as proof that either product covers every source or workflow. The main takeaway is that licensing is not the deciding factor here, so connector fit, operating model and governance workflows should drive the choice.
| Factor | DataHub | OpenMetadata |
|---|---|---|
| Licensing | Apache-2.0 (DataHub v1.7.0.1, September 3, 2026) | Apache-2.0 (OpenMetadata 2.0.5, October 8, 2026) |
| Core architecture | Separate services for metadata storage, search and event streaming, which the platform team must size and run | Central server with its own storage and search, typically deployed as a single application stack, to confirm against the current deployment guide |
| Connector breadth | Confirm coverage for each source in your estate, including legacy warehouses and on-premises databases | Confirm coverage for each source in your estate, including BI tools and message brokers you rely on |
| Ingestion model | Ingestion runs as scheduled recipes, usually orchestrated outside the catalog, to confirm for your sources | Ingestion workflows are configured inside the platform, to confirm for your sources and schedules |
| Lineage depth | Test column-level lineage on your most complex pipelines, including transformations in dbt and Spark | Test column-level lineage on your most complex pipelines, including transformations in dbt and Spark |
| Ownership and governance workflows | Test tagging, glossary terms, domains and approval flows against your data owner model | Test glossary, classification, tagging, domains and approval flows against your data owner model |
| Access control and SSO | Confirm SSO, role model and policy controls against your identity provider | Confirm SSO, role model and policy controls against your identity provider |
| Operating effort | Expect a multi-service platform team skill set and more components to patch and monitor | Expect fewer moving parts to run, but confirm upgrade and backup procedures for your topology |
| Questions for a proof of concept | Which connectors fail on your versions, how long lineage takes to refresh, and who maintains custom ingestion | Which connectors fail on your versions, how workflow failures surface, and how classification scales across domains |
Reference architecture components and how they connect
A metadata catalog is a set of connected services rather than a single application. The components below show how DataHub or OpenMetadata ingests metadata, stores it, serves search and lineage, and gets its credentials and backups handled, along with what the platform team runs after handover.
Ingestion framework and scheduler
Connectors pull schemas, owners, usage and lineage from source systems on a schedule. Each connector runs with its own service account and read-only scope, and each run produces a success or failure record. Confirm which scheduler each platform bundles or expects (for example, an external Airflow deployment) and who owns connector upgrades when source vendors change their APIs.
Metadata store and search index
The metadata store holds entities, ownership, tags and lineage edges, while a search index serves discovery queries. Two datastores mean two backup and capacity plans, and a reindex can be required after upgrades. Ask the vendor which stores each release requires, which versions are supported, and how long a full reindex takes on your metadata volume.
Change capture and event path
Schema changes and pipeline runs should reach the catalog without waiting for the next full crawl. An event stream or message bus carries those changes from sources and orchestrators into the metadata service. Check which sources emit change events natively, which need polling, and how the team detects a stalled event consumer before stale ownership or lineage reaches users.
Web and API layer
The web interface and API are the access points for stewards, engineers and automated tooling. Place them behind your reverse proxy or ingress with TLS, and restrict write APIs to named service identities. Confirm how the API authenticates tokens, how access tokens are issued and revoked, and which endpoints are required for your integrations.
SSO and role mapping
Users sign in through your identity provider, and group claims map to catalog roles such as owner, steward and reader. Mapping decisions determine who can edit ownership and glossary terms, so they belong in the governance design. Confirm which OIDC or SAML options the chosen release supports and how deprovisioning removes access to the catalog and its stored tokens.
Connector credentials and secrets
Every connector needs a credential to read a source, and those credentials are the most sensitive configuration in the catalog. Store them in a dedicated secrets manager, reference them by name in ingestion recipes, and rotate them on a set schedule. Ask whether the platform can read secrets from an external vault or Kubernetes secrets, and who approves a new source credential.
Backup and recovery
Backups must cover the metadata store, the search index strategy, and the configuration that defines connectors and policies. Restoring the database without the matching index or config produces an inconsistent catalog. Test a restore into a separate environment on a fixed cadence, and confirm the recovery point and time targets with the business owner before go-live.
Build in-house, work with a partner or buy a SaaS catalog
This table compares running DataHub or OpenMetadata yourself, delivering it with an implementation partner, and buying a commercial SaaS catalog. Read each row as a question about who holds the data, the skills and the upkeep. The main takeaway is that open source removes license cost but moves connector maintenance, upgrades and metadata freshness onto someone, and the choice is about who that someone is.
| Factor | In-house operation | Partner-delivered open source | Commercial SaaS catalog |
|---|---|---|---|
| Data location | Metadata stays in your accounts, data centers or air-gapped network under your own controls | Deployed in your infrastructure with the partner working inside it; confirm the partner's access model and where any support tooling runs | Metadata is processed and stored by the vendor; confirm region, tenancy and export options in the contract |
| Staffing and skills | Needs platform engineers for Kubernetes or hosting, search and graph storage, and catalog administrators for domains | Partner supplies deployment and handover skills; your team still owns stewards, ownership rules and daily use | Vendor runs the platform; your team needs catalog administrators and stewards but not platform operators |
| Connector maintenance | Your engineers patch and extend ingestion recipes when source systems change | Partner builds and tests connectors in scope; confirm who fixes connectors after handover | Vendor maintains listed connectors; confirm coverage for your niche or legacy sources |
| Upgrades and patching | Your team tracks releases such as DataHub v1.7.0.1 or OpenMetadata 2.0.5, tests them and schedules downtime | Upgrades are planned in the retainer or statement of work; confirm test environments and rollback steps | Vendor applies upgrades on its schedule; confirm notice periods and what changes need your testing |
| One-time setup cost | Internal time for architecture, identity integration, first connectors and policy design | Fixed-scope implementation fee agreed in writing, plus your internal time for owners and review | Onboarding and professional services, often with connector build work quoted separately |
| Recurring cost | Infrastructure, on-call coverage, skilled staff time and tooling you keep current | Infrastructure, optional support retainer and your own steward time | Subscription fees, often scaled by users, sources or metadata volume; confirm how each is measured |
| Metadata freshness | Ingestion schedules and owner follow-up are your responsibility | Partner can set schedules and alerts; ownership of stale entries still sits with your domains | Vendor provides scheduling; ownership and certification workflows remain your process |
| Audit evidence | You produce access logs, lineage exports and change records from your own systems | Partner can prepare runbooks and evidence collection; independent auditors still assess the controls | Vendor may provide attestation reports; confirm scope and whether they cover your configuration |
How a domain-by-domain catalog rollout runs
The sequence moves from a governance charter and a running platform to one domain proven end to end, then to wider connector coverage, and finally to a handover your own team can operate. Each phase closes with a named output so progress is visible before the next phase starts.
Agree the scope and governance charter
Your data owners, the platform team and DeployOpen agree in writing which domains, metadata types and source systems are in scope, and who decides ownership, glossary terms and classification labels. DeployOpen drafts the charter from your data governance policy and your existing data inventory, and you confirm the named roles. The step is done when the charter is signed off and each in-scope domain has a named steward and a named technical owner.
Install the platform and connect identity
DeployOpen installs DataHub or OpenMetadata inside your cloud account, data center or air-gapped network, along with its search index, database and message components as the chosen platform requires. Authentication is wired to your identity provider through SSO, and role mapping is set so that stewards, editors and read-only users receive the correct permissions. The output is a running, backed-up test environment where a sample user can sign in and see only what their role allows.
Pilot one domain with its highest-value sources
The pilot connects the sources that the pilot domain's teams use most, typically a warehouse, a transformation layer and one BI tool, and ingests their schemas, descriptions and usage statistics. Stewards review what appears, correct source mismatches and confirm that search results and ownership fields are useful to the people who will use them. The step closes when the pilot domain's stewards accept the catalog as their working reference and the list of connector issues is either fixed or logged with an owner.
Expand connectors domain by domain
Each further domain follows the same pattern: confirm the connector supports the source version in use, set ingestion schedules, test the run and review the first results with that domain's stewards. Connector licensing and maintenance differ by source, so DeployOpen records any source that needs a custom or vendor-specific connector and confirms the platform's support status for it. The output for each domain is a documented connector list, a scheduled ingestion job and a steward sign-off.
Curate ownership and the business glossary
Stewards assign owners to datasets, define glossary terms with their business meaning, and link terms to the columns and reports they describe. DeployOpen sets up review routines and reports on datasets that still lack an owner or description, so gaps are visible rather than hidden. The step is complete when every high-value dataset in scope has an owner, and the glossary has an approval path that your governance team has tested.
Validate lineage against known data flows
Lineage collected from the orchestrator, transformation tool and BI layer is compared with the data flows your engineers and data owners describe for the pilot domain. Gaps such as lineage that stops at a custom script or a manual file upload are documented, and where feasible they are closed by adding metadata or extending the ingestion. The output is a lineage validation record that states which flows are complete, which are partial and who accepts the residual gaps.
Hand over with runbooks and training
DeployOpen delivers runbooks for installation, upgrades, backup and restore, connector onboarding, ingestion troubleshooting and access reviews, along with a short training session for the platform and stewardship teams. Your team runs at least one ingestion failure and one access change from the runbooks to confirm they work without DeployOpen present. Ongoing support, upgrades and testing are offered as a separate retainer, so you can confirm the scope of what remains after handover before the engagement closes.
What it takes to keep metadata current after launch
Stale metadata is the main way a data catalog fails. A catalog that launches with good coverage but is not refreshed quietly loses trust: owners leave, tables are renamed, columns change type, and lineage stops matching the pipelines that actually run. Plan the ongoing work before go-live, because it is the part most often left unfunded. Combine scheduled ingestion for broad metadata sweeps (for example, nightly or weekly runs of DataHub or OpenMetadata ingestion recipes against each source) with event-driven updates from orchestrators and warehouses where your platform supports them. Confirm in your own environment which sources can push change events and which can only be polled, and measure how long a schema change takes to appear in the catalog.
Ownership is a control, not a field. Set attestation cycles per domain, for example a quarterly review in which each data owner confirms the owner, description and classification of their assets, with unanswered items escalated to the domain lead. Decide how schema changes and deprecated tables are handled: whether a removed column is flagged, whether a deprecated table keeps its lineage for a defined period, and who is notified. Accountability should sit with named domain owners for their assets, the platform team for connector health and upgrades, and the security or governance function for classification rules. Connector upgrades need their own test cycle in a non-production copy, because a new connector release can change how metadata is mapped.
Measure what your environment actually does rather than trusting a rate. Track the share of production assets with a current owner, the age of the last successful ingestion per source, the number of failed ingestion runs, the count of assets whose schema drifted without an owner response, and the time from attestation request to completion for each domain. Set these baselines during the pilot and ask any implementation partner to show how they are collected from the platform's own run history. The effort to maintain the catalog depends on how many sources you connect, how often they change and how many domains share one catalog, so request an estimate of recurring operator hours for your source list in the written scope rather than relying on a general figure.
Systems the catalog must connect to and what each involves
A catalog is only as current as its connections. The systems below determine whether metadata arrives automatically or depends on people updating it by hand, and each one carries a different integration type and set of connector questions. Confirm coverage for your exact versions before you commit to a scope.
Data warehouses and lakes
Warehouses and lakes are the primary source of table, column and usage metadata, so they set the baseline for discovery and lineage. Connectors typically read schemas and query history, and some platforms also capture profiling statistics, which adds load on the source. Confirm with the vendor which warehouse and lake table formats, catalogs and authentication methods the connector supports, and whether it reads query logs or only information schema objects.
Orchestrators such as Airflow and Dagster
Orchestrators show which jobs produce which datasets, which is the most direct source of pipeline lineage. Airflow is Apache-2.0 licensed and Dagster is Apache-2.0 licensed, but their lineage emitters differ in how they publish run and dataset events. Confirm the lineage integration method, the versions it supports and whether task-level or only dataset-level lineage is captured.
BI tools such as Superset and Metabase
Dashboards and charts are where business users consume data, so cataloging them shows which reports depend on which tables. Coverage for dashboards, charts and their owners varies widely between connectors, and Metabase's enterprise directory uses a separate commercial license. Confirm which BI objects are ingested, whether field-level lineage reaches charts, and how permissions on dashboards are represented.
Transformation and pipeline tools
Transformation tools and ingestion platforms define how raw data becomes governed data, so their lineage is needed to answer where a sensitive field came from. Ingestion tools such as Airbyte carry their own license terms, and transformation projects often need manifest or log parsing to produce column-level lineage. Confirm which transformation framework versions the connector parses, and whether column-level lineage is produced or must be built.
Identity providers for SSO and group sync
The catalog needs single sign-on and group membership so that access to metadata and ownership assignments follow the same rules as the source systems. Group sync determines whether a data owner in the identity provider maps to a catalog owner without manual entry. Confirm the supported protocols, whether group membership syncs on a schedule or on login, and how removed users lose access to documentation and sensitive tags.
Ticketing and approval workflows
Access requests, ownership changes and classification decisions usually happen in a ticketing or approval system, and the catalog must record the outcome. Integration is typically webhook, API or event based, and the catalog should store the approval reference next to the tag or owner it changed. Confirm whether the connector can create or update tickets, or only read them, and how approval history is retained.
Data protection registers
Records of processing activities and data inventories are often maintained in a separate register or GRC tool, and they drift from the catalog unless linked. The catalog can hold the dataset identifiers and classifications that the register cites, which supports data protection evidence. Confirm the export or API format the register accepts, who owns the mapping between register entries and datasets, and how changes in either system are reconciled.
Using lineage to answer AI training data questions
Lineage and ownership in a catalog let a team answer the first questions an auditor or regulator asks about an AI system: which datasets fed it, who approved them, where they came from and what transformations touched them on the way. DataHub and OpenMetadata both model lineage between datasets, pipelines and downstream assets, so the reader should confirm that the lineage for training and evaluation datasets is captured at the orchestration layer (for example, Airflow or Dagster jobs that produce the curated tables) and not only described in documents. Each dataset used for training should carry an owner, a data steward, a classification and a retention tag, so that a question about a specific dataset returns a named person and a dated record rather than a search through email.
The EU AI Act sets record-keeping duties that are easy to confuse with catalog work. Article 12(1) requires high-risk AI systems to technically allow for the automatic recording of events (logs) over the lifetime of the system, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Those logs record what the system did in operation. A catalog records what data existed, where it moved and who was accountable for it. The catalog is a complement to those logs, not a substitute, and it cannot satisfy the event-logging duty on its own. Teams should confirm applicability with counsel and check the AI Act Article 12 text alongside the Article 19 text.
Practically, the catalog should link each training or fine-tuning dataset version to the model version it produced, and that link should be kept current when datasets are refreshed. Ask the implementation partner to define the identifiers that join catalog entries to model registry records and to application logs, and to test a sample question end to end: which model used which dataset version, who approved it, and where the event logs for its operation are stored. Lineage gaps usually appear where data is copied outside managed pipelines, such as notebooks or manual exports, so the plan should name how those paths are either routed through pipelines or recorded when they cannot be.
Data catalog questions buyers ask before hiring help
How much effort does setup and ongoing metadata curation take?
Setup is a one-time phase of platform installation, connector configuration, and identity integration, while curation recurs as sources change, owners move teams, and new tables appear. Effort rises with the number of source systems, how much of the metadata is ingested automatically versus entered by hand, and how strictly ownership and glossary terms must be maintained. Ask any partner to split the quote into initial deployment, connector work per source type, and a recurring operating model that names who reviews ownership gaps each month. Your own data stewards will still need time, because no catalog can infer business definitions or accountability on its own.
How long does the first data domain take to go live in a catalog?
The first domain usually goes live after the platform is running, its identity provider is connected, and at least one source connector has been validated against real metadata. Duration depends on how many systems the domain spans, how clean its existing documentation is, and how quickly the owners can confirm who is accountable for each dataset. Agree on a written scope with a defined set of sources and a named owner group before work starts, so the timeline reflects a bounded domain rather than the whole estate. Ask for the exit criteria in writing, such as which lineage paths and ownership fields must be populated before sign-off.
What does a catalog project need from our data and security teams?
Data teams provide source system access, connector service accounts, and the domain owners who will validate descriptions and lineage. Security teams need to approve how the catalog authenticates users, which network paths the connectors use, and where metadata and credentials are stored. Plan for the catalog to read schema and usage metadata, so decide early whether sample data or query logs are in scope, since that changes the review. Ask the implementation partner to list every credential and network path required per connector before any deployment begins.
Can an internal team run a data catalog on its own?
Yes, an internal team can run a catalog alone if it has platform engineering skills for Kubernetes or the vendor's deployment method, someone who owns connectors and upgrades, and data stewards who maintain ownership and glossary content. The common gap is not installation but sustained operations: upgrades, backup and restore testing, connector failures, and access reviews. Compare the cost of hiring or training those roles against a fixed-scope implementation followed by a retainer for upgrades and support. Confirm whether DeployOpen's ongoing support fits your model before choosing that path.
How do I choose between DataHub and OpenMetadata?
Both are metadata and catalog platforms licensed under Apache-2.0, so the choice rests on architecture, connector coverage for your sources, and the operating skills your team already has. DataHub's current release is v1.7.0.1 (September 3, 2026) and OpenMetadata's is 2.0.5 (October 8, 2026), according to their GitHub repositories. Run a proof of concept against your three most important sources, measure how each handles lineage and ownership, and check which connectors you need are maintained for your versions. Ask the partner to document the selection criteria and the evidence behind the decision.
What are the license implications of Apache-2.0 for a catalog?
Apache-2.0 permits commercial internal use, modification, and redistribution of DataHub and OpenMetadata, subject to retaining notices and the license terms. It does not by itself require you to publish changes you make for internal use. Your obligations change if you ship a modified version to third parties or bundle it into a product, so have legal review that use case. Check any separately licensed connectors, plugins, or enterprise add-ons, because a catalog's full dependency tree may include components under other terms.
How does a data catalog support data protection audits?
A catalog supports audits by showing where personal data is held, who owns each dataset, how data flows between systems through lineage, and which classifications and access owners are recorded. It produces evidence an assessor can sample, but it does not itself make you compliant or issue any attestation. Independent auditors and certification bodies make those determinations. Ask the partner which evidence exports are configured, how classification tags are reviewed, and how stale ownership is flagged, so assessors can see that the records are maintained.
How DeployOpen scopes, builds, hands over and supports a data catalog
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source catalogs such as DataHub or OpenMetadata inside your own cloud accounts, data centers or air-gapped networks. Engagements start with a written, fixed scope so that connectors, domains, ownership model and evidence outputs are agreed before work begins.
How we scope
- Inventory the source systems, connectors and domains that the catalog must cover in the first phase
- Agree in writing who owns each domain, how ownership is recorded and who approves metadata changes
- Define the evidence the catalog must produce, such as lineage records and ownership reports for data protection reviews
- Confirm which platform fits your environment after reviewing licensing and deployment requirements for DataHub or OpenMetadata
How we build
- Deploy the catalog in your infrastructure with identity integration, network controls and backup configuration agreed upfront
- Configure ingestion recipes for the agreed connectors and schedule metadata refresh jobs
- Test discovery, lineage accuracy, access controls and ownership workflows against the written acceptance criteria
- Document configuration, runbooks and the metadata stewardship process so your team can operate the catalog
How we hand over
- Deliver architecture records, configuration documentation, operating procedures and test results as named deliverables
- Walk your platform, data governance and security teams through administration, upgrades and troubleshooting
- Provide a rollout plan by domain so each data owner knows what is onboarded, when and what they must maintain
- Record open items and the metadata gaps that remain, so the handover is explicit about what is not yet covered
How we support
- Offer ongoing support, platform upgrades and periodic review of connector coverage as a retainer
- Keep the catalog patched and backed up, with upgrade testing before changes reach production
- Prepare control and evidence summaries for compliance assessments, while auditors and certification bodies issue the actual conclusions
- Confirm with DeployOpen the retainer scope, response expectations and service terms before signing
