Data catalog requirements checklist: how to use it before choosing OpenMetadata, DataHub or CKAN
A governance, risk or data team that must choose and run a data catalog inside its own infrastructure needs requirements it can test, not a feature wish list. This checklist turns business goals into testable requirements and separates internal governance, where the catalog serves stewards, analysts and auditors, from a public data portal, where the catalog publishes datasets to outside users. Those two goals lead to different platforms, so the first decision is scope. Write down which teams will search the catalog, which systems must be cataloged in the first phase, and whether any dataset will be published outside the organization. Once scope is fixed, work through each requirement in turn: sources and connectors, lineage depth at table, column and pipeline level, ownership and glossary, classification of personal data, data quality, search, single sign-on and access policies. For each one, write a pass or fail test that a proof of concept can run against your own sources, such as a named warehouse, a dbt project or a BI dashboard. Ask the vendor or partner to confirm connector support for your exact versions rather than relying on a published count. OpenMetadata and DataHub are both released under Apache-2.0, so license terms alone will not separate them. CKAN is the more natural fit where publication-focused goals dominate, such as an open data portal with dataset catalogs. Set a rule for the proof of concept before it starts: any requirement that fails on a representative source is either removed from scope, moved to a later phase, or handled with custom development that is priced in the written scope. Record each result in a shared matrix so that procurement, security and the data owners review the same evidence. This is also the point to confirm how SSO is configured on the self-hosted build, how access policies map to your identity provider groups, and how upgrades will be tested, because these decisions carry forward into operations.
Which functional requirements must a data catalog meet?
This table lists the functional requirements a data catalog should meet, why each matters, and how to verify it in a proof of concept against your own sources. Vendor-stated counts and features are starting points, not guarantees, so treat each one as a claim to test. The main takeaway is that a catalog is judged on how well it covers your estate and keeps metadata current, not on its connector total.
| Requirement | Why it matters | How to verify in a proof of concept |
|---|---|---|
| Sources and connectors | Coverage decides whether the catalog reflects the real estate. OpenMetadata lists more than 130 connectors and DataHub lists more than 150 integrations, but your specific warehouses, legacy systems and BI tools are what count. | Map every priority source to a named connector, version and owner. Confirm the connector runs against your own versions, credentials and network paths, and note the effort for each gap. |
| Custom connector path | Internal systems and older databases are often missing from vendor lists. A documented SDK or ingestion framework keeps those sources in scope without manual spreadsheets. | Build one custom connector for a source without native support. Record the time, skills and tests needed, and confirm how it is versioned and upgraded. |
| Table and column lineage | Table lineage answers where data came from. Column lineage is needed to trace report figures and derived fields, which regulators and model owners often ask about. | Pick one report figure and one derived column. Trace both back to source systems in the catalog and check whether any hop is missing or inferred. |
| Dashboard and pipeline lineage | Lineage that stops at the warehouse leaves BI dashboards and orchestration jobs unexplained. Impact analysis for changes depends on these links. | Change a source column in a test environment and check which dashboards, pipelines and owners are flagged. Compare the result with a manual impact review. |
| Lineage capture method | Lineage from parsed SQL, query logs or OpenLineage events differs in completeness. OpenMetadata documents OpenLineage event integration, and DataHub documents its own ingestion paths. | Ask which mechanism produces each lineage type for each source. Run a job that uses dynamic SQL and confirm the lineage it produces. |
| Ownership and glossary bootstrapping | Without owners, findings and access requests have no destination. Glossary terms that are not linked to datasets rarely get used. | Import owners from your identity or HR source and seed a small glossary of critical terms. Measure how many priority datasets have an owner and linked terms at the end of the trial. |
| Personal data classification | Records of processing, access controls and retention rules depend on knowing where personal data sits. Automated tags need tuning and human review. | Run auto-classification on a sample database with known personal data. Record true and false positives, confidence thresholds and who approves each tag. |
| Data quality and freshness | A catalog that shows stale or failing data without warning erodes trust. Quality tests and freshness checks must link to owners who act on them. | Add a failing test and a stale table in the trial. Confirm the alert reaches the owner, appears on the asset page and can be reported on. |
| Search and discovery | Analysts judge the catalog by whether they can find a dataset in one or two searches. Ranking, synonyms and filters by owner, domain or classification matter. | Give five analysts a set of business questions and time how long they take to find the right table. Review the failed searches with the catalog team. |
| SSO and access policies | Catalog metadata often reveals sensitive structure. Single sign-on, role mapping and restrictions on who can edit or view are part of the security baseline. | Test OpenID Connect login with your identity provider, confirm group-to-role mapping and attempt an unauthorized edit. Confirm how the self-deployed frontend is configured. |
| Internal governance or public portal | An internal catalog for stewardship and lineage has different needs from a public portal that publishes datasets. Choosing the wrong tool forces a later migration. | Write down whether consumers are internal staff or external users. If external publication is needed, test the publication workflow in a separate tool, such as CKAN, before committing. |
Security requirements that a catalog must satisfy from day one
A catalog holds a map of every system, owner and dataset, and often connects to credentials for the sources it scans. Access control, audit trails and hardening therefore need to be in place before the first connector runs, not added after the catalog becomes popular. The cards below cover the requirements to set and the checks that show they work.
SSO through OpenID Connect
Staff should sign in to the catalog with the corporate identity provider, using OpenID Connect, so that joiners, leavers and group changes flow from one place. On self-deployed DataHub Core, the operator configures the frontend server for OIDC, while DataHub Cloud enables OIDC SSO through its interface. Verify that a disabled account loses access, that group claims map to the intended roles, and that local administrator accounts are restricted and documented. Confirm the configuration steps with your DataHub or OpenMetadata partner before go-live.
Role-based access to metadata and samples
Separate who can read metadata, who can edit ownership and glossary terms, and who can view sample rows, since sample data can expose personal or regulated values even when schema metadata does not. Define roles per domain and test them with named users from each group. Check that sample profiling and preview features respect the same restrictions as the table itself, and that restricted datasets do not leak through search results, lineage graphs or descriptions.
Audit logging of catalog changes
Record who changed an owner, classification, glossary term, lineage edge or access setting, and when. These records support accountability for personal data records and show auditors how ownership was assigned. Confirm which events the chosen platform logs by default, where the logs are stored, how long they are retained, and whether they can be forwarded to the central security monitoring system. Test by making a change in each category and tracing it in the log.
Secrets for connector credentials
Every connector to a warehouse, BI tool or pipeline needs credentials, and those credentials are often broader than the catalog needs. Use a dedicated read-only service account per source, store secrets in a managed secrets store rather than in ingestion recipes or configuration files, and rotate them on a schedule. Verify that secrets do not appear in logs, the user interface or exported metadata, and that a revoked account stops ingestion within the expected window.
Encryption in transit and at rest
Metadata includes sensitive descriptions, lineage that reveals system architecture and, for some setups, sample values. Require TLS between browsers, the catalog service, the database, the search cluster and the message broker where DataHub uses Kafka. Encrypt database and search storage and backups using keys held in the organization's key management system. Test by inspecting network paths and confirming that disks and backup snapshots are encrypted.
Hardening the search and database tiers
The database (MySQL or PostgreSQL) and the search service (Elasticsearch or OpenSearch) hold the full catalog index, so they need the same controls as production databases. Restrict network access to the catalog services only, disable anonymous access, apply security features the search product requires for authentication, patch on a documented cadence, and tie upgrades to a tested restore. Use the OpenMetadata security documentation as the checklist and confirm version-specific settings with your partner.
Internal governance or public data portal: which goal changes the checklist
The first decision that changes the checklist is whether the catalog serves internal governance or a published data portal, because the two need different requirements even when the same tool is considered for both. An internal catalog has to reach across the estate: connectors to warehouses, BI tools and pipelines, table, column and pipeline lineage, named owners, glossary terms, personal data classifications and access policies tied to SSO. Lineage and classification are the heart of that work, and they are what the sibling pages on financial services, healthcare and GDPR records depend on. Start by listing which systems hold regulated or personal data, then confirm each one has a supported connector in the candidate platform before anyone commits to a scope.
A published portal asks different questions. Its center is dataset publication metadata, such as descriptions, licensing terms, update frequency, distribution formats and contact points, and a public search experience that does not expose internal lineage, restricted columns or access requests. Where datasets are shared with other public bodies, data sharing agreements define who may use what and under which conditions, and those agreements are not captured by lineage graphs. For metadata that must be interoperable across agencies, teams usually look at DCAT-AP style publication models, which is why CKAN appears alongside OpenMetadata in the content group. Confirm with each candidate whether its publication features cover the metadata schema your public bodies or regulators expect, rather than assuming a governance catalog can publish directly.
Many organizations need both, and the practical step is to split the requirements into two lists before the vendor conversation: one for internal discovery and governance, one for publication and sharing. Assign each item to an owner, mark which items are required on day one and which can wait for a later phase, and note which items would force a second tool. For depth on public publication, read the public sector data catalog page on internal catalogs versus portals, and for the platform options, check each vendor's own documentation before relying on this summary.
Integration requirements: sources, pipelines, BI tools and identity
A catalog is only as current as the connections feeding it. The cards below cover the sources and pipelines it must read from, how lineage events arrive, the BI and transformation tools that sit downstream, and how users and automation authenticate against it.
Warehouse and database connectors
List every warehouse, operational database and file store that holds in-scope data, then confirm each has a maintained connector in the catalog. OpenMetadata lists more than 130 connectors, and DataHub's README lists more than 150 integrations including Snowflake, BigQuery and Databricks. Confirm the connector version supports your exact engine version and the auth method your security team requires, such as key pair or service principal. Test: run a metadata-only ingestion against one schema per source and compare table and column counts with the source system.
Orchestrator and pipeline lineage capture
Lineage is only as deep as the events the catalog receives. Decide whether lineage comes from pipeline emitters, such as OpenLineage events sent by Airflow or Dagster jobs, or from parsing SQL in transformation code. Confirm which events your orchestrator version emits and which fields, including job names and run IDs, are populated. Test: trigger one pipeline run that reads from a source table and writes to a target, then check that the table-level edge and the job node appear without manual edits.
BI and dashboard connections
Dashboards and reports are where business users meet data, so a catalog that stops at tables leaves the question of which report uses which column unanswered. Confirm the catalog can ingest dashboards and charts from the BI tools you run, and that column-level lineage reaches the dashboard layer where the tool exposes it. Ask the vendor which BI platforms are supported natively and which require custom ingestion. Test: pick one executive report and trace a figure back to its source column in the catalog.
dbt and transformation metadata
dbt models carry descriptions, tests and column documentation that should flow into the catalog rather than being maintained twice. Confirm the connector reads the dbt manifest and run results, maps model lineage to warehouse tables, and keeps owner and test metadata current after each run. Ask how the catalog handles dbt projects that span several repositories. Test: change a model description in dbt, rerun the project, and confirm the change appears in the catalog without manual editing.
API, SDK and automation access
Ownership assignment, glossary updates and tagging at scale are easier to automate than to click through. OpenMetadata provides REST APIs, a Python SDK, TypeScript packages and Java libraries, and DataHub offers its own API surface. Confirm which operations need service accounts, how tokens are rotated and which rate limits apply to bulk updates. Test: script the assignment of owners to fifty tables from a CSV, then confirm the changes are audited with the service account name.
Identity provider and SSO integration
Users should sign in with the corporate identity provider and lose access when they leave. Confirm the catalog supports OpenID Connect with your provider and how group claims map to catalog roles. DataHub supports OIDC with providers such as Google, Okta and Azure AD, but on self-deployed DataHub Core the operator configures the frontend server. Test: sign in with a user from each group, confirm role mapping, then disable the account and confirm access ends at the next token refresh.
Operational requirements: deployment, sizing, upgrades and backup
This table compares the operational requirements for a self-hosted OpenMetadata or DataHub catalog, from deployment model and sizing to upgrades, backup and ownership. Read each row as a question to answer before a platform is chosen, with the verification column showing how the answer should be proven. The main takeaway is that a catalog is only as recoverable as its metadata store and search index, so restore and upgrade rehearsals matter as much as the initial install.
| Requirement | What to confirm or specify | How to verify it |
|---|---|---|
| Deployment model | Kubernetes for production (OpenMetadata documents EKS, AKS, GKE and on-premises clusters), bare metal without Kubernetes, or Docker for trials only | Written architecture approved by the platform team, with the chosen model matched to existing cluster and hosting standards |
| Runtime prerequisites | OpenMetadata bare metal needs Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later, and Elasticsearch 9.x or OpenSearch 3.x | Clean install on a non-production host that confirms every prerequisite version before go-live |
| Minimum sizing | OpenMetadata server at 2 vCPUs, 4 GB RAM and 20 GB storage, with database and search each at 2 vCPUs, 2 GB RAM and 30 GB storage as minimums | Load test with the real number of connectors and profiling jobs, since minimums do not cover large estates |
| Search and graph dependencies | DataHub needs Kafka, a MySQL, PostgreSQL or MariaDB database, Elasticsearch for search and a graph index on Neo4j or Elasticsearch | Dependency inventory with versions, owners and support status for each component |
| Upgrade handling | OpenMetadata 2.0 breaking changes include moved embedding credentials, a new Databricks auth format, Python 3.12 ingestion images and a Great Expectations 1.3 migration | Upgrade rehearsal on a non-production copy with a documented rollback path before any production change |
| DataHub schema changes | The Helm system-update job applies schema changes and reindexes search when needed, so job success must be checked after each upgrade | Post-upgrade check that the job completed, search returns expected results and lineage still renders |
| Backup and restore | Metadata database and search indices are separate stores, each needing a backup schedule, retention period and documented restore order | Scheduled restore test into an isolated environment, timed and signed off by the data owner |
| Monitoring | Ingestion run status, failed connector jobs, search cluster health, database capacity and API error rates | Alerts routed to the platform team, with a monthly review of failed ingestion runs |
| Named platform owner | One accountable owner for upgrades, connectors, access and backup, plus a deputy for continuity | Owner named in the RACI and the runbook, confirmed at each quarterly governance review |
What compliance evidence a data catalog can produce, and what it cannot
A data catalog can produce much of the raw evidence that regulators and auditors ask for: a maintained inventory of data assets, named owners, classification tags, and lineage showing where a figure or dataset came from. For GDPR, that evidence feeds the records of processing described in Article 30, where the catalog can hold the purpose, categories of personal data, recipients and transfers for each dataset, provided owners keep those fields current. For DORA, the inventory and dependency mapping can support the identification of information assets and their supporting dependencies. For ISO 27001:2022, the catalog can hold the inventory of information assets, classification levels and labels, and the owner of each dataset. In every case, the catalog is a system of record for the metadata, not proof that the controls work.
What the catalog cannot do is decide whether your processing is lawful, whether your classification scheme is adequate, or whether your controls operate as described. Those judgments belong to your data protection officer, risk and compliance functions, and the independent auditor or certification body you engage. DeployOpen plans, implements, configures and documents the catalog, prepares the controls and evidence that it can generate, and supports assessments. DeployOpen does not issue audit reports, attestations or certifications. Ask your auditor early which artifacts they will accept, such as exports, timestamps, change history or access logs from the catalog, and confirm with them how each regulatory requirement maps to a catalog field or process.
Plan the mapping before the catalog is populated. A useful exercise is to take each clause or control your auditor will test, write down the catalog object that should evidence it (for example, a dataset entity, an owner field, a classification tag or a lineage edge), and assign a steward who keeps it current. Confirm in your own environment whether the chosen platform can export metadata history and audit trails in a form your evidence process accepts, since capabilities differ between the open-source releases and any commercial add-ons. Gaps found at this stage are cheaper to close than gaps found during an assessment.
How to verify the checklist in a proof of concept
This sequence turns the requirements checklist into evidence by testing the highest-risk requirements against real sources, real groups and a rehearsed upgrade and restore before any production commitment. Each step ends with a scored result the buyer can defend.
Fix scope and the proof of concept goal
The sponsor, the data owner and the platform lead agree in writing which business goal the catalog must serve first, such as internal governance or a published data portal, and which requirements are must-have for that goal. They also name the sources, teams and sample datasets in scope and the exit criteria. The step closes when the team signs a one-page scope note that lists each checklist requirement with its owner and its pass, gap or unknown status to be filled in later.
Load the highest-risk sources first
The platform engineer connects the sources that carry the most regulatory, personal data or financial risk, such as the core warehouse, one operational database and the main BI tool, using the connectors that the candidate platform lists for them. The engineer records the connector versions, credentials handling, ingestion schedule and any failed runs. The step is done when each source shows ingested assets, owners can be assigned and the ingestion logs are kept as evidence of coverage and error handling.
Test lineage on one critical report
A data steward chooses one report that leadership or a supervisor relies on and traces each reported figure back through pipelines, transformations and tables to its source. The steward checks whether lineage appears at table, column and pipeline level, and notes every hop that is missing, manual or stale. The step is complete when the trace is compared against the documented data flow, and the gaps are logged with the pipeline or connector responsible for each one.
Test classification on a known personal data sample
The privacy lead prepares a sample dataset in which the personal data fields are already known, including names, contact details, identifiers and free-text fields that may hide personal data. The team runs the platform's classification and compares its tags against the answer key, counting missed fields and false positives at the confidence settings in use. The step closes with a confusion table per field type and a written decision on who reviews tags before they drive records of processing or access rules.
Test SSO and role access with real groups
The identity team connects the catalog to the organization's identity provider through OpenID Connect or the platform's supported method and maps real directory groups to catalog roles. Testers sign in with accounts from each group and try to view, edit and search assets they should and should not reach, including restricted classifications. The step is done when every access test has a recorded expected and actual result, and the team confirms that removing a user from a group removes access.
Rehearse an upgrade and a restore
The platform operator performs a test upgrade from the current release to the next supported release in a non-production copy, following the vendor's breaking-change notes, and records any schema migrations, reindexing and manual steps. The operator then restores the metadata database and search index from backup into a separate environment and checks that lineage, owners and glossary terms return. The step closes with timed runbook notes, the exact backup and restore commands used and a list of anything that required manual intervention.
Score each requirement as pass, gap or unknown
The steering group reviews the scope note and the evidence from each earlier step and assigns every checklist requirement one status: pass when the test met the agreed criterion, gap when it failed or needed unplanned custom work, and unknown when the evidence is insufficient. Each gap and unknown gets an owner, a proposed fix and a question for the vendor or partner. The step is complete when the signed scorecard, the evidence folder and the open-question list are filed together for the procurement decision.
Questions buyers ask before they choose a data catalog platform
How much does a self-hosted data catalog project cost?
The software license is not the main cost, because OpenMetadata and DataHub Core are both released under Apache-2.0. The spend comes from four components: the infrastructure for the metadata database and search service, the setup of connectors and lineage capture, the time stewards spend curating ownership, glossary terms and classifications, and ongoing operations. Infrastructure is one-time for the initial build and recurring after that. The main cost drivers are the number of source systems, how deep lineage must go (table, column, pipeline), and how much metadata must be curated by hand. Ask any implementation partner for a quote that breaks out each component, names who pays for each one (your cloud account, your staff, the partner), and states which items recur after go-live. DeployOpen does not publish prices, so the quote follows from a written scope.
How long does a data catalog rollout take?
Duration depends on scope, not on the product alone. The biggest factors are the number of connectors to configure, whether lineage must reach column level, how many domains need ownership and glossary work, and how quickly access approvals and identity integration can be completed. OpenMetadata documents production deployments on Kubernetes and on bare metal, and DataHub documents a Helm-based Kubernetes path with separate dependency charts, so the platform choice affects the build steps. Ask for a timeline in the written scope with phases for connector setup, lineage validation, stewardship and SSO. No fixed duration is stated here because no researched figure applies to your environment.
What does the internal team need to provide?
A catalog needs named people on both the technical and business sides. Expect to provide a platform owner who approves changes, administrators for the cloud or data center where the catalog runs, owners for each source system who can confirm connector credentials and scope, and data stewards who maintain definitions, classifications and ownership after go-live. Legal or privacy staff should review any personal data tags that feed records of processing. Without these roles, connectors often run but the metadata stays incomplete. Ask the partner to list every role, the hours each one is expected to commit, and the decisions each person must sign off.
Can our own engineers run a data catalog without a partner?
Yes, if your team has the skills to operate Java-based services, a relational database, a search cluster and the ingestion framework. OpenMetadata's bare metal guide lists Java 21, PostgreSQL 15 or later or MySQL 8.0.42 or later, and Elasticsearch 9.x or OpenSearch 3.x. DataHub relies on Kafka, GMS, consumers and a frontend on Kubernetes. Self-hosting also means you own upgrades, backups, monitoring and OIDC configuration on DataHub Core, where the frontend server must be configured by the operator. Many teams use a partner for the first deployment and hardening, then keep operations in-house or move to a retainer. Decide this by counting the skills you can staff, not by license cost.
OpenMetadata, DataHub or a managed catalog: which fits best?
Both OpenMetadata and DataHub are Apache-2.0 metadata platforms that can be self-hosted, so the choice depends on your operating model and features. OpenMetadata lists more than 130 connectors and documents table, column, dashboard, pipeline and metric lineage, plus data quality, profiling, glossaries and classifications. DataHub lists more than 150 integrations and column-level lineage, glossary and data quality assertions, and its company sells DataHub Cloud as a managed, SLA-backed service. A managed option reduces infrastructure work but adds subscription cost that recurs. Compare the connector list against your actual systems, test lineage on one critical pipeline, and ask for a written comparison of total recurring cost.
How are OpenMetadata and DataHub upgrades handled?
Upgrades are planned changes with breaking-change review, not routine patching. OpenMetadata 2.0 introduced breaking changes, including moving embedding credentials into an llmConfiguration block, a new Databricks auth format, rebuilding custom ingestion images for Python 3.12, and migrating Great Expectations to 1.3 before upgrading. OpenMetadata maintenance releases followed 2.0.0 about every one to two weeks. The DataHub Helm chart runs a system-update job on upgrades that applies schema changes and reindexes search when needed. Ask the partner for a written upgrade runbook covering test environments, backups before each change, rollback steps, and who reviews release notes.
Does the catalog support SSO and access control for internal users?
Yes, but the configuration depends on the platform and deployment. DataHub supports OpenID Connect providers such as Google, Okta and Azure AD, and on self-deployed DataHub Core the operator configures the frontend server. OpenMetadata publishes security documentation that your team should review against its identity requirements. Confirm which roles map to which groups, how service accounts for ingestion are scoped, and how access to sensitive metadata is limited. Ask the partner to show the configuration in your test environment before production.
How DeployOpen scopes, delivers and supports a data catalog selection
DeployOpen is a Chainscore Labs company that plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For data catalog requirements, we turn the checklist into a written scope covering sources, lineage depth, ownership, classification, search, SSO and access policy before any platform is installed.
How we scope
- Inventory the source systems, warehouses, BI tools and pipelines the catalog must cover, with their owners.
- Record the lineage depth required at table, column and pipeline level, and which reports or datasets it must trace.
- Agree in writing whether the goal is internal governance, a public data portal, or both, before selecting a platform.
How we build
- Deploy the chosen catalog, such as OpenMetadata or DataHub, in the client's own cloud account, data center or air-gapped network.
- Configure connectors, SSO through the identity provider, and role-based access policies agreed in the scope.
- Test ingestion, lineage capture, classification rules and search results against the requirements checklist before handover.
How we hand over
- Deliver architecture, configuration, runbooks and an ownership map as documented deliverables.
- Walk stewards and platform staff through connector onboarding, glossary maintenance and classification review.
- Provide a written record of which requirements were met, deferred or confirmed with the vendor.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the fixed-scope engagement ends.
- Plan catalog upgrades in advance, including breaking changes and dependency migrations listed in release notes.
- Review new connectors, lineage sources and access policies as the data estate changes over time.
