Which catalog fits a Dataplex renewal or a multi-cloud estate
If your renewal is driven by Google Cloud data, a managed catalog that discovers BigQuery and other Google Cloud resources with little setup is the lower-effort fit, and Dataplex (Knowledge Catalog) is built for that. DataHub fits when your metadata sits across several clouds or on-premises databases, when your data control rules require the catalog to run inside your own network, and when your team can operate Kubernetes, Kafka, a relational database and a search service. The decision is less about feature checklists than about where your sources live and who will run the platform after go-live.
Start with a source inventory. Count the systems you need cataloged, grouped by cloud and by on-premises location, and note which ones only have connectors in one ecosystem. If most of the estate is in Google Cloud, Dataplex keeps ingestion simple, while a multi-cloud estate will need custom entries or connectors for everything outside Google Cloud. Then document your data control requirements: whether metadata itself, including schema names, descriptions and sample profiles, may leave your environment, and whether a vendor-operated control plane is acceptable for your regulators and internal policy. A self-hosted DataHub keeps that control with you, but only if you also own its upgrades and backups.
Finally, assess operating capacity honestly. DataHub's Kubernetes guide lists GMS, the MAE and MCE consumers and the frontend, plus Kafka, a MySQL, PostgreSQL or MariaDB database, and a search and graph index, so someone on your team must own that stack. Ask whether you already run these components, who will apply Helm chart upgrades, and whether a managed or supported option such as DataHub Cloud, which the project describes as SLA-backed, is part of your plan. If the source mix is mostly Google Cloud and the operating capacity is thin, Dataplex may remain the right answer for the renewal.
Which option fits each scenario
This table matches common estate shapes to the option that fits better and the reason for it. Read each row against your own source mix, data residency rules and operations staffing, then confirm the details that the table flags.
| Scenario | Better fit | Reason |
|---|---|---|
| Mostly BigQuery and Google Cloud estate | Dataplex (Knowledge Catalog) at first | Dataplex catalogs Google Cloud resources with less setup. DataHub's Snowflake, BigQuery, Looker and dbt integrations are useful only if you want one catalog across tools, and it adds Kafka, search and database components to run. |
| AWS, Azure or on-premises sources alongside Google Cloud | DataHub | DataHub's 150+ integrations and column-level lineage cover non-Google sources in one catalog. Confirm connector coverage for your specific databases and on-premises systems in your own environment before deciding. |
| Data that must stay in the organization's own environment | DataHub, self-hosted | DataHub Core runs in your cloud account or data center under an Apache-2.0 license, so metadata stays under your control. Confirm where Dataplex stores and processes your metadata against your residency policy. |
| Team with no Kubernetes or search operations skills | Dataplex, or DataHub with a managed or partner-run deployment | DataHub on Kubernetes needs GMS, MAE and MCE consumers, Frontend, Kafka, a relational database, Elasticsearch or OpenSearch and a graph index. Without that skill set, budget for staffing or outside operations support. |
| Team that wants a managed vendor SLA | Dataplex, or DataHub Cloud | DataHub Core is self-operated and carries no vendor SLA. DataHub Cloud offers a 99.5% uptime SLA and is the managed route, while Dataplex is a Google Cloud managed service; confirm the terms in each contract. |
| Team needing curated governance with access request workflows | DataHub Cloud or a custom build on DataHub Core | Access request workflows and compliance forms are listed as DataHub Cloud features, not open-source features. Confirm which capabilities your team needs before choosing the edition. |
Where the metadata lives and who controls it
Dataplex keeps catalog metadata inside Google Cloud as a managed service, while self-hosted DataHub places the metadata service, search and queues wherever you choose to run them. The cards below cover where data sits, who operates it, how users sign in, and what evidence you will need to show an assessor.
Managed hosting inside Google Cloud
Dataplex runs as a managed service, so Google operates the catalog infrastructure and you do not install or patch it. Metadata for Google Cloud resources stays in the Google Cloud environment you already use, which simplifies the picture if all your data is there. Confirm with Google which regions and resource locations hold catalog metadata for your organization, and whether any metadata leaves those regions through features you enable.
Self-hosted DataHub in your environment
With DataHub Core you choose the placement: your Google Cloud project, another cloud account, a data center or an air-gapped network. The stack is a metadata service, Kafka, a relational database, Elasticsearch or OpenSearch, and a graph index, so every component's location becomes your decision. Document which network zone each one runs in and who holds the credentials for its storage.
Operator-configured OIDC sign-in
On DataHub Core, the operator configures OpenID Connect single sign-on on the frontend server, so the identity provider and its settings are part of your implementation scope. Google, Okta and Azure AD are named as OIDC providers in the DataHub guide. DataHub Cloud handles this through its interface instead. Plan who maintains client secrets, redirect URLs and group mappings after go-live.
Audit and compliance evidence
DeployOpen prepares systems, controls and evidence for compliance assessments, including access settings, backup records and change history for the catalog. Independent auditors decide whether those controls meet a framework, and DeployOpen does not issue audit reports or certifications. Ask your assessor which evidence they expect for a metadata platform, and whether they want logs kept outside the catalog itself.
Metadata export and residency questions
Before you commit, confirm the scope and format of any Google-side metadata export, because that export is the main route for moving curated fields such as descriptions, glossary links and ownership out of Dataplex. Confirm also whether catalog entries that describe BigQuery policy tags or sensitive fields are copied into the export. Record the answers in your data residency register so the next review does not start from zero.
Upgrades and version pinning
Self-hosted DataHub upgrades are your responsibility, and the Helm chart runs a system-update job that applies schema changes and reindexes search when needed. Plan a test environment for each release and record the version you run in production. Current DataHub release numbers should be checked on the project's releases page at the time of your evaluation, since this list changes frequently.
What DataHub needs to run: Kafka, search and graph services
The metadata service (GMS) is the center of DataHub, and everything else exists to feed it or read from it. Metadata changes move through Kafka, which DataHub uses to stream changes between its components. The Kubernetes guide describes four main components: GMS, the MAE Consumer, the MCE Consumer and the Frontend. The MCE Consumer writes incoming metadata change events, the MAE Consumer reacts to them to update search and graph indices, and the Frontend serves the web interface. Storage is MySQL, PostgreSQL or MariaDB, search runs on Elasticsearch or OpenSearch, and a graph index runs on Neo4j or Elasticsearch. Ingestion works in two directions: push ingestion, where your pipelines send metadata to DataHub, and pull ingestion, where DataHub's connectors read from sources such as BigQuery on a schedule. Your team must decide which of these your Google Cloud estate will use.
Deployment is handled by Helm charts from the acryldata/datahub-helm repository, with a separate prerequisites chart for the dependencies (Kafka, the relational database, search and the graph index). Keep the two charts at compatible versions and test upgrades in a non-production cluster first. The Helm chart runs a system-update job on upgrades that applies schema changes and reindexes search when needed, so plan a maintenance window and confirm the expected duration with your own staging run. Resource sizing is the first cost to estimate. The Docker quickstart requires Docker with 8 GB of RAM and Python 3.10 or later, which is only a local evaluation footprint. The Kubernetes guide says more than 7 GB of RAM is needed to run DataHub and its dependencies on Minikube. Production sizing for Kafka, search and the graph index depends on your metadata volume and must be tested, not copied from these figures.
These components are the operating surface you must own or staff. Someone has to patch and monitor Kafka brokers, manage search index health and capacity, back up the relational database, and track version compatibility across GMS, the consumers, the Frontend and the prerequisites. Dataplex needs none of this because Google runs the catalog. With DataHub, the cost is paid in engineering hours, cloud compute and storage, and any managed services you choose for Kafka, search or the database. Ask any implementation partner to quote these components separately, state which ones they will install, harden and support after go-live, and what the handover covers. Confirm with the DataHub project which Kafka and search versions the current release supports before you choose managed services.
What drives the cost of running DataHub against Dataplex
How much does Google Cloud Dataplex cost compared with running DataHub?
The honest answer depends on your usage, so the first step is to ask Google for current pricing for the Dataplex Universal Catalog (Knowledge Catalog) features and billing dimensions that apply to your organization. Google bills these as consumption on Google Cloud, so the cost moves with the resources you catalog, the metadata you process and any related services you use. Do not compare a Google invoice line with a DataHub infrastructure estimate directly. Instead, model both sides across the same twelve months: your Google Cloud metadata usage today, the growth you expect, and the infrastructure, staff and migration costs listed in the other answers on this page. Ask your Google account team for the pricing terms in writing and keep the figures with your scope.
Does the DataHub open-source edition have a licence fee?
DataHub Core is licensed under the Apache License 2.0, so there is no licence fee to run the open-source edition. The license cost you do not pay is the one you should still verify in your own procurement process, because the company behind DataHub also sells DataHub Cloud, a separate commercial product. Confirm with your legal team that the Apache-2.0 terms fit how you plan to modify, bundle and distribute the software. Any subscription you choose later, such as DataHub Cloud, is a different line item and should be quoted separately.
What infrastructure does a self-hosted DataHub need to run?
Infrastructure is the main recurring cost of self-hosting DataHub. The DataHub Kubernetes guide describes four main components, GMS, the MAE Consumer, the MCE Consumer and the Frontend, which depend on Kafka, a MySQL, PostgreSQL or MariaDB database, Elasticsearch for search and a graph index on Neo4j or Elasticsearch. Costs rise with the number of sources, the volume of metadata, the frequency of ingestion and the redundancy you require for production. Your cloud provider or data center bills the compute, storage, network and managed services you choose. For a sizing estimate, ask your platform team to model the clusters and managed services in your own environment, and treat the minimum local test setup described in the documentation as a development baseline, not a production sizing.
How much staff time does running DataHub take each year?
Staff time is recurring and usually the largest cost after infrastructure, and it grows with the number of connectors and environments you run. The main activities are installation, upgrades, SSO configuration, monitoring the Kafka and search components, and maintaining ingestion recipes as source systems change. The DataHub Helm chart runs a system-update job on upgrades to apply schema changes and reindex search indices when needed, which reduces manual work but still needs a maintenance window and rollback planning. Configuring OIDC single sign-on on self-deployed DataHub Core is an operator task, so budget for it. Ask any implementation partner to itemize these tasks, the roles involved and the number of hours they expect for each, and to state who owns each task after handover.
What does a one-time migration and integration project include?
A migration from Dataplex to DataHub is a one-time cost. It covers scoping which entries, glossaries and curated fields move, setting up BigQuery and Looker ingestion in DataHub, exporting metadata from Dataplex, mapping entry groups and glossary terms, and validating the result with the business owners. Costs rise with the number of sources outside Google Cloud, the amount of curated metadata that must be remapped, the number of access and policy rules to rebuild and the approval steps your governance program requires. Ask for a written scope with deliverables, assumptions, acceptance criteria and a list of what the client team must provide, such as access to source systems and named data owners. Most DeployOpen engagements are fixed-scope and agreed in writing before work starts.
Should I pay for DataHub Cloud instead of self-hosting?
DataHub Cloud is an optional subscription, and it replaces some of the infrastructure and operations costs above with a vendor-run service. According to DataHub's comparison documentation, it adds features not in open source, including data quality monitoring, AI anomaly detection, compliance forms, access request workflows, a 99.5% uptime SLA and AWS PrivateLink. Choose it only if those features or the SLA justify the subscription for your organization, and compare the quote with the infrastructure, staff and support costs of your self-hosted plan. Also check where your data would be processed, because a managed service changes the data residency picture. Ask the vendor for the subscription terms, the SLA definitions and the exact features included at your deployment size.
What should I ask for in a quote for self-hosted DataHub?
Request a quote that separates recurring and one-time costs and states its assumptions. Ask for the following:
- A list of the Google Cloud and other source systems in scope, with the expected metadata volume and ingestion frequency.
- The infrastructure design for Kubernetes, Kafka, the database and search, with the assumed sizes, redundancy and cloud pricing source.
- The staff effort for installation, upgrades, OIDC SSO setup and connector maintenance, with the roles and the handover plan.
- The migration deliverables, acceptance criteria and client responsibilities.
- Whether support, upgrades or continuous testing are included, or offered as a separate retainer.
- Any optional DataHub Cloud subscription, quoted separately.
DeployOpen does not publish a price list, and a quote follows from the written scope.
Can my team run DataHub without an implementation partner?
Yes, many teams run DataHub themselves if they have Kubernetes, data engineering and identity skills in house. The open-source edition is available under Apache-2.0, and the Docker quickstart is useful for evaluation, but production deployment on Kubernetes needs planning for Kafka, the database, search, backups and upgrades. The skills gap usually appears in SSO configuration, connector maintenance and security hardening. Assess your team against those tasks before deciding. If you want outside help, DeployOpen plans, implements, integrates, hardens, backs up, upgrades, documents and supports self-hosted open-source platforms inside your own infrastructure, under a written, fixed scope, with ongoing support available as a retainer.
How a move from Dataplex to self-hosted DataHub is planned
The sequence moves curated Dataplex metadata, lineage and access into a self-hosted DataHub instance that your team operates, and proves the result against the old catalog before users switch over.
Inventory Dataplex entries and their origins
Export the current entry inventory and split it into entries that Dataplex created automatically for Google Cloud resources and entries that someone created by hand or through custom entries. Record the owners, glossary links, descriptions and tags on each group, because the automatic entries will be rebuilt by connectors while the manual ones must be carried over explicitly. The step is done when every entry group has a named owner and a decision on how it will be recreated in DataHub. Confirm the export scope options in the Knowledge Catalog documentation before you start, since the export is the only bulk source of curated fields you will use later.
Decide where DataHub runs relative to Google Cloud
Choose whether the DataHub cluster sits in the same Google Cloud project or region as your BigQuery datasets, in another cloud, or in your data center. This choice drives network paths for ingestion, egress costs, latency for users and who holds the cluster credentials. Your cloud platform team, security team and data owners should sign off on the placement in writing. The step is complete when the placement, network routes and the service accounts that the ingestion jobs will use are documented and approved.
Deploy DataHub on Kubernetes with Helm
Install the prerequisites chart for Kafka, the relational database, Elasticsearch or OpenSearch and the graph index, then install the DataHub chart from the acryldata/datahub-helm repository. The core services are the GMS, the MAE Consumer, the MCE Consumer and the Frontend, and the Kubernetes guide sets out their dependencies. Size the cluster for production load, which is more than the 7 GB of RAM the guide cites for a Minikube test. Your platform team operates this layer, and the step is done when all pods are healthy, the Frontend loads and a test ingestion run writes a visible dataset. Plan a staging environment first so that chart upgrades, including the system-update job, can be rehearsed.
Configure OIDC single sign-on on the Frontend
On self-deployed DataHub Core, your operators configure the Frontend to trust your OpenID Connect provider, such as Google, Okta or Azure AD, following the OIDC SSO guide. Map the identity provider groups to DataHub roles before opening access, and confirm that a user who leaves the directory loses access. Security engineering should test the login flow, session lifetime and group mapping in staging. The step is complete when a test user from each role can sign in through the provider and no local accounts are left enabled without a documented reason.
Ingest BigQuery, Looker and other sources
Set up DataHub ingestion recipes for BigQuery, Looker and the other sources in your estate, using the connectors listed for DataHub. Schedule the runs, store source credentials in your secrets manager and confirm which service accounts can read which projects. Compare the resulting dataset counts, schemas and lineage for a sample of tables against what Dataplex shows, and record the differences. The step is done when each source has a scheduled recipe, its credentials are stored outside the recipe files, and the sample comparison has been reviewed by the data owners. The sibling migration page on BigQuery and Looker ingestion covers recipe details.
Load curated fields from the metadata export
Use the JSONL metadata export from Knowledge Catalog as the source for descriptions, owners, tags and glossary links that connectors cannot infer. Transform each record into the DataHub metadata change format, and map Dataplex glossary terms and entry groups to DataHub glossary terms, domains and ownership. Load the curated set into a staging instance, then compare a sample of entries field by field. The step is complete when every curated field type has a mapping, unmapped fields are listed with an owner, and the data stewards have signed off the sample. The sibling export and migration pages describe the file format and transformation in detail.
Run parallel validation against Dataplex
Keep Dataplex and DataHub running side by side for a defined review period agreed with the data governance lead, and have stewards search for their most-used assets in both catalogs. Check that search results, lineage paths, glossary links and access restrictions match for the critical data sets, and log each gap with an owner and a fix. Confirm that DataHub's data quality assertions and any compliance records are reproduced or documented as retired. The step is done when the agreed review period ends, the open gap log is empty or formally accepted, and the governance lead records approval to proceed.
Cut over users and retire Dataplex usage
Publish the DataHub URL as the catalog of record, update onboarding material and links from BI tools and internal portals, and tell users which catalog to use for new work. Turn off scheduled Dataplex export jobs and automatic ingestion where no longer needed, and keep a final export archived under your retention policy. Remove Dataplex access grants that are no longer required, and confirm which Google Cloud audit and policy tag settings must remain. The step is complete when usage of Dataplex has stopped for day-to-day work, the final export is stored, and the cutover record is signed by the data owner and security team.
Questions buyers ask before choosing DataHub over Dataplex
What does it cost to run DataHub instead of Dataplex?
DataHub Core has no license fee, so the cost is the infrastructure and the people who run it. The main drivers are the Kubernetes cluster that hosts GMS, the consumers and the frontend, the Kafka, relational database and search services it depends on, storage and backup for those services, and the engineering time for upgrades, monitoring and ingestion pipelines. Costs rise with the number of sources, metadata volume, search load and how many environments you keep (development, test, production). Dataplex is billed through Google Cloud, so compare its usage charges with your own infrastructure and staffing. When you ask for a quote, request a line-item breakdown of one-time setup, recurring infrastructure, recurring support, and what each assumes about source count and environments.
How long does a DataHub deployment take?
Duration depends on scope: how many sources you connect, whether you need single sign-on, how many environments you build, and how much glossary and ownership content must be loaded before the catalog is useful. A proof of concept on a single machine is much smaller than a production cluster with high availability, backups and integrated identity. DeployOpen agrees timelines in a written, fixed scope before work starts, so ask for the milestones, the assumptions behind them and what would move them. Your own readiness, such as cluster access and identity provider configuration, is often the largest variable.
Can our team run DataHub without outside help?
Yes, if your team already operates Kubernetes and can support Kafka, a relational database and Elasticsearch or OpenSearch in production. The DataHub Kubernetes guide describes Helm charts from the acryldata/datahub-helm repository plus a separate prerequisites chart, and the upgrade process runs a system-update job that applies schema changes and reindexes search when needed. Teams without these skills usually need help with capacity planning, upgrade testing and incident response. Assess the gap honestly: list who will own each service, who is on call, and who has tested an upgrade in a non-production environment.
What do we need to provide for a DataHub engagement?
The work depends on access and decisions from your side. You will typically need a list of data sources with owners and connection methods, cloud and cluster access in your own accounts, identity provider details for OpenID Connect single sign-on, and network rules for connectors. For the business glossary and ownership model, you need named people who can approve term definitions and assign data owners, since the tool cannot decide those for you. Agree on who signs off on classifications and on which environment is the system of record before ingestion begins.
Do we need DataHub Cloud for an SLA or AWS PrivateLink?
Only if you require a vendor-backed uptime commitment or private connectivity through AWS PrivateLink. DataHub's comparison documentation lists a 99.5% uptime SLA and AWS PrivateLink as DataHub Cloud features that are not in open source. DataHub Cloud also adds data quality monitoring, AI anomaly detection, compliance forms and access request workflows. If your organization needs these, factor the subscription into the cost comparison. If you can meet availability and network needs with your own cluster design and operating processes, DataHub Core under Apache-2.0 avoids that recurring fee. Confirm the current feature list directly with the vendor before deciding.
Can we export our Dataplex metadata to move it to DataHub?
Yes. Knowledge Catalog (formerly Dataplex Universal Catalog) supports metadata export jobs that write JSONL files to Cloud Storage, which you can query with BigQuery and transform for another catalog. Choose the scope carefully: organization, project or entry group. Export gives you the metadata, but you still need to map entry groups, aspects and glossary terms to DataHub concepts and re-ingest technical metadata from sources such as BigQuery and Looker. Curated descriptions, owners and policy tags need explicit mapping, and BigQuery policy tags stay tied to Google Cloud IAM unless you redesign access. Ask for a mapping review as part of the migration scope.
What risks should we check in DataHub before committing?
Check the operational dependencies first: Kafka, search and the metadata database each need capacity, backups and monitoring, and a failure in one affects catalog availability. Verify the upgrade path in a staging environment, because the system-update job reindexes search when required. Confirm single sign-on setup on self-deployed DataHub Core, since the operator configures the frontend server. Finally, review whether the open-source feature set covers your governance workflows, particularly access requests and compliance forms, which are listed among DataHub Cloud features.
What to confirm before committing to either option
Before you commit to DataHub, inventory every metadata source you need to catalog and check connector coverage for each one. DataHub's README lists more than 150 integrations, including Snowflake, BigQuery, Databricks, dbt, Looker and Tableau, but that list does not prove your specific source versions, custom databases or on-premises systems are supported. Ask the DataHub project and your own platform team to confirm each source against the current ingestion documentation, and note which sources would need custom ingestion work. Estimate expected metadata volume too: the number of tables, columns, dashboards and lineage edges drives the sizing of Kafka, the search index and the graph store, so ask for those counts from every source owner before sizing anything.
Confirm where the catalog may run. Residency rules, customer-managed key requirements and network boundaries determine whether DataHub runs in the same Google Cloud region as your BigQuery datasets, in another cloud or on-premises. Identify the identity provider that will sign users in, because self-deployed DataHub Core requires the operator to configure OpenID Connect for the frontend server, and the OIDC SSO guide describes the settings that involve. Check also whether your team has the Kubernetes and Kafka capacity to run GMS, the MAE and MCE consumers, the frontend, the database, Elasticsearch or OpenSearch and the graph index. The DataHub Kubernetes guide lists these as prerequisites, and you should confirm the resource requests for your expected volume with a test deployment rather than relying on the 7 GB minimum for a Minikube trial.
Pin the release you plan to deploy. The most recent release in the facts we reviewed is DataHub v1.7.0.1, released on September 3, 2026, but DataHub publishes releases frequently, and v1.8.0 release candidates were already in circulation in October 2026. Check the DataHub GitHub releases page on the day you plan to deploy, read the upgrade notes, and test the Helm chart's system-update job against a copy of your data before going live. Finally, review Google's current terms for Knowledge Catalog, since Dataplex has been renamed and Data Catalog has been discontinued, and confirm what export and API access those terms and your contract allow for moving metadata out of Google Cloud.
How DeployOpen works on this
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a Dataplex to DataHub move, we start from your written scope so the deployment, source connections and handover are agreed before any work begins.
How we scope
- Inventory which Google Cloud projects, BigQuery datasets and non-Google sources must appear in the catalog.
- Agree in writing where DataHub runs relative to Google Cloud, including network paths to each source.
- Define the curated Dataplex fields to carry over, such as glossary terms, owners and descriptions.
How we build
- Deploy DataHub on Kubernetes with its Helm charts, with Kafka, the database, search and graph dependencies configured.
- Configure ingestion for BigQuery and other in-scope sources, and set up OIDC single sign-on on the frontend.
- Test lineage, glossary mapping and search results against a sample of assets agreed with your data owners.
How we hand over
- Document the deployment, ingestion recipes, SSO settings and upgrade procedure in a form your platform team can run.
- Walk through the system-update job used during upgrades and the backup approach for the metadata stores.
- Record decisions on what stays in Dataplex or Knowledge Catalog during the transition and what is retired.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, version upgrades and continuous testing are available as a retainer after handover.
- Ask for a quote based on the written scope, which covers infrastructure, integration and handover effort.

