How to migrate from Google Cloud Dataplex to DataHub without losing curated metadata
Moving from Dataplex (Knowledge Catalog) to self-hosted DataHub is mostly two jobs: rebuilding ingestion so that BigQuery, Looker and your other sources feed DataHub again, and carrying the curated fields that people spent time maintaining, such as descriptions, owners, glossary links and classifications, across to the new catalog. The usual trigger is a renewal, a lock-in concern or a residency requirement that makes a self-hosted catalog the preferred end state. Ingestion is the easier half, because DataHub's Apache-2.0 licensed open-source edition, DataHub Core, already ships connectors for BigQuery and Looker. The harder half is deciding which curated metadata is authoritative and proving it arrived intact. Platform engineering usually owns the deployment and ingestion, data teams own the descriptions and ownership, and governance teams own glossary terms and classifications, so agree on those owners before any data moves.
Start by inventorying what Dataplex holds for you today. Export the entries with the Knowledge Catalog metadata export job described in the sibling guide on exporting, then sort fields into three groups: fields DataHub ingests automatically from the source (schemas, table names, basic lineage), fields that must be re-applied from your export (descriptions, owners, glossary links, custom aspects), and fields with no direct equivalent that need a modeling decision. Confirm with your own environment which BigQuery policy tags and access settings must stay in Google Cloud and which move into DataHub's governance features. Do not assume a one-to-one mapping for aspect types, because Dataplex aspects and DataHub aspects are not the same objects.
The main risk is silent loss: a catalog that looks complete but lost owners or glossary links during transformation. Plan a parallel run where Dataplex and DataHub are compared on a sample of high-use tables, and sign off on field-level checks before you retire the Dataplex estate. Decide early where DataHub runs relative to Google Cloud. Running it in the same region and project boundary as BigQuery keeps ingestion traffic and residency simple, while running it elsewhere requires you to confirm network paths and credentials for each source. Ask your implementation partner for a written scope that names these checks, the owners for each field group and the acceptance criteria, because that scope, not the tool, determines whether the curated metadata survives.
Which Dataplex capabilities map to DataHub and which need rework
This table maps the main Dataplex (Knowledge Catalog) capabilities to their closest DataHub equivalents and marks each as clean, needs rework or gap. Read the status column first, then the notes on what has to be rebuilt. The main takeaway is that BigQuery and Looker metadata and glossary content move well, while Google-specific aspects, policy tags and access enforcement need explicit planning and confirmation in your own environment.
| Dataplex capability | DataHub equivalent | Status and what to plan |
|---|---|---|
| Automatic cataloging of BigQuery and Google Cloud resources | BigQuery ingestion source with 150+ integrations overall | Clean for BigQuery. Confirm project scope and recurring ingestion schedule in your environment. |
| Entry groups and entry scope | Domains, containers and platform instances | Needs rework. Rebuild the grouping model by hand from the export scope you chose. |
| Aspect types and custom aspects | Custom properties and structured metadata | Needs rework. Each aspect schema must be mapped to a property definition before load. |
| Business glossary and term descriptions | Business glossary with terms and term groups | Clean for terms and descriptions. Map term relationships and ownership during the load. |
| Ownership and stewardship | Owners and ownership types on datasets | Needs rework. Map steward roles to DataHub owner types and confirm who approves changes. |
| BigQuery policy tags | Tags and glossary terms used for classification, not enforcement | Gap for enforcement. Keep policy tags in BigQuery and confirm how access is enforced in your environment. |
| Lineage | Lineage with column-level lineage from supported sources | Clean for supported sources. Validate lineage for each connector before retiring Dataplex views. |
| Data quality results | Data quality assertions and monitoring | Needs rework. Rebuild assertions in DataHub. Historical results from Dataplex are not carried over automatically. |
| Metadata export as curated-field source | Ingestion from transformed JSONL export files | Needs rework. Transform the export into DataHub metadata before loading curated fields. |
What data and integrations move, and what you rebuild
Some of what Dataplex holds can be exported and reloaded, while connectors, identity and lineage must be rebuilt in DataHub. The cards below separate the curated metadata you carry over from the integrations you configure again and verify.
Curated metadata from the JSONL export
The Dataplex metadata export writes JSONL files to Cloud Storage and is the documented path for pulling your curated fields out of Knowledge Catalog. Descriptions, owners, glossary links and tags are the fields you need to map into DataHub's ownership, glossary and tag models. Check that each exported entry resolves to the same dataset URN you will create in DataHub, because mismatched names will split one asset into two records. Also confirm whether exported aspects include the tags and glossary links you rely on, since the export is a transformation step, not a direct import.
BigQuery ingestion as DataHub recipes
DataHub's README lists BigQuery among its supported integrations, so the BigQuery catalog is rebuilt by running an ingestion recipe rather than copying Dataplex's automatic entries. Define the recipe with the projects, datasets and service account scope you need, and store the credentials in your own secrets system. Run the ingestion once in a non-production DataHub, compare entry counts against Dataplex for the same projects, and resolve differences before you retire the Dataplex view that business users depend on.
Looker and dbt ingestion as DataHub recipes
The DataHub README also names Looker and dbt among its integrations, and both feed lineage and descriptions that teams often assume come from Dataplex. Rebuild each as its own ingestion source so that dashboards and transformations appear with their upstream tables. Confirm with your Looker and dbt owners which projects, models and explores are in scope, and check that the lineage shown in DataHub matches the lineage your analysts see today.
Identity and SSO rebuilt on DataHub Core
On self-deployed DataHub Core, the operator configures OpenID Connect single sign-on on the frontend server, with providers such as Google, Okta or Azure AD named in the OIDC guide. Dataplex users authenticate through Google Cloud IAM, so their access does not carry over automatically. Plan group-to-role mapping before cutover, decide whether DataHub's own roles or your identity provider's groups control edit rights, and test that removed users lose access. The frontend configuration is your responsibility on Core, not a feature you switch on.
Lineage and access integrations to re-verify
Column-level lineage is listed among DataHub's features, but lineage built from Dataplex or BigQuery audit data will not be recreated by copying metadata. Re-run lineage extraction and compare key pipelines against the lineage you rely on for impact analysis. Also review access controls: BigQuery policy tags and IAM bindings stay in Google Cloud, so any access decision that DataHub makes must be checked against the source system rather than assumed. Record each integration in a verification log for your auditors.
Migration sequence from inventory to decommission
This sequence moves Dataplex metadata into self-hosted DataHub in controlled stages, so each stage has a named owner and a verifiable output before the next one starts. Dataplex stays authoritative until the DataHub copy has been checked against it.
Inventory entry groups, glossaries, aspect types and policy tags
The data platform team lists every Dataplex entry group, glossary term set, aspect type and BigQuery policy tag in use, with the owner of each. Include which entries carry manually written descriptions, since those are the fields that cannot be regenerated from source systems. Confirm the current scope of your metadata export job with the team that runs it. Done when you hold a signed inventory that maps each Dataplex object to a planned DataHub home, or to an explicit decision to drop it.
Stand up a pilot DataHub instance
The platform team deploys DataHub Core in a non-production environment, using the Docker quickstart for evaluation or the Helm charts on Kubernetes for a shared environment. Check the README and Kubernetes guide for the current prerequisites, including the Kafka, database, search and graph index components. Done when the frontend loads, OIDC login works through your identity provider, and the operator has confirmed the system-update job completes on a test upgrade.
Pilot ingestion on one BigQuery dataset and one Looker instance
Engineers configure the BigQuery and Looker ingestion sources for one dataset and one Looker instance, then compare the results with what Dataplex shows for the same assets. Check schema, descriptions, ownership, tags and lineage side by side, and record every field that did not carry across. Done when the pilot diff lists each gap, and each gap has an owner and a decision on whether to fix it in DataHub or accept it.
Load curated fields from the JSONL export
Using the Dataplex metadata export JSONL files in Cloud Storage as the source, the team transforms curated descriptions, glossary terms and entry group ownership into DataHub's model and ingests them into the pilot. Stewards then review a sample of those assets in DataHub. Done when the stewards sign off that glossary terms link to the correct assets and that no curated description was lost in transformation.
Run DataHub in parallel with Dataplex as the authority
Ingestion runs on a schedule in both catalogs, and Dataplex remains the system where users edit metadata and where governance decisions are recorded. A recurring reconciliation report compares counts and curated fields between the two. Done when the reconciliation report shows no unexplained differences across several consecutive review cycles that your governance board has agreed on.
Cut over ingestion and glossary ownership
The governance board approves the switch, after which stewards edit glossary terms and descriptions in DataHub and the ingestion schedule for those sources moves to DataHub. Dataplex is set to read-only for the migrated scope, and the access model is confirmed, including how BigQuery policy tags remain enforced in BigQuery itself. Done when a steward can complete a change request in DataHub and the audit trail of the change is retained.
Decommission Dataplex export jobs and retain records
The team stops the scheduled Dataplex metadata export jobs and removes their service account permissions, then archives the final JSONL export and the migration inventory under your retention policy. Confirm with your compliance lead which records must be kept and for how long. Done when no export job is running, the archive is verified readable, and the decommission record is approved by the owner of the data catalog.
Where DataHub should run relative to Google Cloud
Running DataHub on Google Kubernetes Engine (GKE) inside the same Google Cloud organization keeps the catalog close to BigQuery and Looker. Ingestion jobs call Google APIs over Google's network, and the metadata export from Dataplex stays in Cloud Storage in the same project family, which simplifies the export and load steps. Your team still runs the DataHub Helm charts, plus the prerequisites chart for Kafka, the relational database, and Elasticsearch or OpenSearch, so the platform work does not disappear.
- Latency: Ingestion from BigQuery and Looker runs within Google's network, so the main delay is the size of the crawl rather than the network path.
- Service account keys: Prefer Workload Identity Federation on GKE over exported JSON keys, and grant the ingestion service account only the read roles the connectors need.
- Operations: Your platform team owns upgrades, including the system-update job that runs on Helm upgrades, backups of the database, and Kafka and search capacity.
- Blast radius: A misconfigured project-level IAM binding can expose the catalog to the same teams that administer the data, so separate the DataHub cluster into its own project.
- Cost driver: GKE node pools, persistent disks and egress to any non-Google destination are the recurring items to model.
Confirm with the security team which projects may hold catalog metadata, whether Looker and BigQuery metadata are classified, and whether the cluster falls under the organization's existing logging and key management policies.
Hosting DataHub in another cloud account or a client data center, and reaching Google Cloud over private connectivity, separates the catalog from the data platform's administrators. This suits organizations that already run shared platform infrastructure elsewhere. Ingestion crosses a network boundary, so the connectors need stable, private routes to the BigQuery and Looker APIs, and the export files must be copied or read across that boundary.
- Latency: Each ingestion run and search query depends on the path length and bandwidth of the private link, so test a full BigQuery crawl before committing to a design.
- Service account keys: Keys are often the only option for cross-cloud access, so store them in the destination's secrets manager, set a rotation schedule, and avoid placing them in Helm values files.
- Operations: Kafka, the database and search each need separate backup, patching and monitoring, and their failure modes can be harder to trace across two networks.
- Data egress: Metadata leaving Google Cloud can incur transfer charges, which depend on your provider's pricing and the volume of catalog data moved.
- Network controls: Allow only the specific ingestion endpoints and document each firewall rule for review.
Confirm with the security team which key management system protects the credentials, how traffic is encrypted across the link, and whether the cross-boundary flow appears in the data flow diagrams for your assessment.
On-premises Kubernetes suits estates where residency rules or contractual terms prevent metadata from being hosted in a public cloud. DataHub runs on your own clusters with the same Helm charts, and you control every layer, from storage to network access. Latency to BigQuery and Looker is the highest of the three options because every API call leaves your network, so ingestion schedules and page sizes deserve attention. Access to Google APIs still requires credentials, and those credentials remain a central control point.
- Latency: Plan ingestion windows around the slower path, and avoid running frequent full crawls of large BigQuery projects.
- Service account keys: Long-lived keys are harder to rotate in a data center, so confirm whether your policy allows short-lived credentials or requires a hardware-backed store.
- Operations: Your team must provision Kafka, the database, search and the graph index, plus storage, load balancing and certificates, because no managed service covers any of them.
- Hardware sizing: The DataHub Kubernetes guide notes more than 7 GB of RAM for a Minikube test, so size production nodes from a vendor-neutral capacity test rather than that figure alone.
- Air-gapped options: If the cluster cannot reach Google APIs at all, metadata must arrive by file transfer, which changes the ingestion design.
Confirm with the security team and residency owner exactly which metadata fields count as regulated, whether the cluster needs a documented data flow to Google Cloud, and who approves the keys used for ingestion.
What drives migration cost and duration
This table lists the factors that move the cost and effort of moving Dataplex metadata to self-hosted DataHub, with the reason each one matters and how to keep it down. Use it to read a quote: each driver should map to a line item, and one-time work should be separated from recurring operation.
| Driver | Why it changes cost or effort | How to keep it down |
|---|---|---|
| Entry groups and sources in scope | Each entry group, BigQuery project and Looker instance adds ingestion recipes, credentials, and validation work before anything reaches DataHub. | Confirm which entry groups hold live assets and retire stale ones during inventory. |
| Depth of curated metadata to transfer | Descriptions, tags, owners and aspects that need mapping from the JSONL export take transform and review time, not just a load. | Limit the transfer to fields your users actually search or govern, and agree the field list in writing. |
| Glossary size and ownership model | Mapping glossary terms and entry groups into DataHub's business glossary requires decisions about hierarchy and who approves changes. | Name a single owner per glossary domain before the transform starts. |
| Custom aspect types | Each custom aspect needs a mapping to a DataHub aspect or custom property, and each mapping needs testing. | Inventory aspect types early and retire unused ones instead of migrating them. |
| Looker and BigQuery project count | Lineage and usage ingestion must be configured and tested per project, which multiplies pilot and cutover checks. | Pilot one project end to end, then repeat the same pattern for the rest. |
| SSO and access model | Configuring OIDC for self-deployed DataHub Core is an operator task, and group-to-role mapping must match existing Google Cloud IAM intent. | Document the target groups and roles before configuring the frontend. |
| Platform team Kubernetes experience | Running GMS, consumers, Frontend, Kafka, a relational database and search on Kubernetes requires operating skill that is paid for either in staff time or in a partner's time. | Assess in-house skills honestly and confirm who handles on-call and incidents. |
| One-time versus recurring costs | One-time: inventory, transform, pilot and cutover. Recurring: hosting, upgrades, the Helm system-update job after each upgrade, and support. | Ask the quote to show each one-time item and each recurring item as separate lines. |
What usually goes wrong and how to prevent it
Most failed Dataplex-to-DataHub moves do not fail on ingestion. They fail on the metadata that people curated by hand, the access model nobody mapped, and the cutover that nobody owned. Each card below names one risk, the step that prevents it, and the check to run before anyone switches users over.
Curated descriptions and owners lost in transform
Descriptions, owners and glossary links that analysts entered in Dataplex live in the JSONL metadata export, not in the BigQuery connector output. Your transform script must map each aspect field to a DataHub aspect deliberately, or those edits disappear silently. Before cutover, sample a set of tables with known owners and descriptions and confirm each field appears on the same entity in DataHub, with no empty owner fields left behind.
Assumed policy tag and access carryover
BigQuery policy tags enforce column access in Google Cloud, but DataHub does not enforce those tags. Teams often assume that classifications copied into DataHub will restrict who sees data, which exposes sensitive columns in the catalog. Decide which tags DataHub will show as metadata only, and keep enforcement in BigQuery. Check that every restricted column still has its tag in the source and that DataHub's access settings match what your governance owner approved.
Lineage gaps outside BigQuery
DataHub's 150+ integrations cover many sources, but lineage only appears where a connector or emitter reports it. Dataplex lineage captured for non-BigQuery systems, such as custom pipelines or on-premises databases, will not arrive automatically. List every upstream and downstream system your analysts rely on, then confirm each one has a DataHub source and that column-level lineage shows for your most-used tables before cutover.
Undersized Kafka and search for the estate
DataHub needs Kafka for streaming metadata changes, a relational database, and Elasticsearch or OpenSearch for search, with a graph index on Neo4j or Elasticsearch. The Kubernetes guide notes that more than 7 GB of RAM is needed just to run DataHub and its dependencies on Minikube, so production sizing depends on entity count and ingestion frequency. Load-test the ingestion of your full BigQuery scope in a staging cluster and confirm search latency and consumer lag stay acceptable before go-live.
Parallel runs that drift
Running Dataplex and DataHub side by side is useful, but without an owner the two catalogs diverge as people edit one and forget the other. Name a single owner for the cutover, freeze edits in Dataplex on a fixed date, and schedule a final JSONL export after the freeze. Before switching, compare owner, description and glossary counts between the export and DataHub, and record who signed off on any differences.
How the migration changes by estate size and starting point
When most governed data sits in BigQuery and only a handful of Looker dashboards are in scope, the migration is mainly a metadata transfer. DataHub's BigQuery source is the main ingestion path, and the JSONL metadata export from Dataplex covers the curated descriptions, tags and entry details that the source does not reproduce on its own. The number of moving parts stays small, so the work concentrates on verifying that what was curated in Dataplex is visible and correct in DataHub.
- Ingestion scope: Run the BigQuery source against the same projects Dataplex covers today and compare entity counts before you retire anything.
- Looker scope: Connect Looker only if dashboards are part of the governed estate. A few dashboards can be reviewed by hand against their lineage.
- Curated fields: Map descriptions and ownership from the export into DataHub. This step is small in volume but easy to skip.
- Operations: A small estate may justify a single-node evaluation first, but confirm the sizing for your own environment before you commit.
Before you start, decide whether BigQuery policy tags must stay the enforcement point, since DataHub does not replace them.
A multi-project estate with a heavily curated glossary changes the emphasis. Ingestion is repeated per project and per source, so the effort lies in deciding how the Dataplex entry groups and glossary terms map onto DataHub's business glossary, its domains and its ownership model. Glossary terms that have been edited by several teams usually carry the most inconsistency, and those conflicts need a named owner before any import runs.
- Term inventory: Export the glossary and list duplicate or overlapping terms across entry groups, then pick one canonical term for each.
- Entry group mapping: Decide whether each Dataplex entry group becomes a DataHub domain, a tag or a container. Record the decision in writing.
- Ownership: Assign a steward for each glossary area so imported terms are reviewed rather than accepted as exported.
- Ingestion repetition: Plan one recipe per project or environment so that scheduled runs stay separate and easy to debug.
Confirm with your data governance lead who approves glossary changes, because that approval, not the tooling, usually sets the pace of this migration.
A team that already runs Kubernetes and Kafka can deploy DataHub with the Helm charts from the acryldata/datahub-helm repository and the separate prerequisites chart. The DataHub Kubernetes guide lists the core services: GMS, the MAE Consumer, the MCE Consumer and the Frontend, which depend on Kafka, a MySQL, PostgreSQL or MariaDB database, Elasticsearch for search and a graph index on Neo4j or Elasticsearch. Much of the platform work is reuse, so the migration shifts toward configuration, backup and upgrade routines.
- Reuse: Use existing Kafka clusters only if topic and retention settings can be dedicated to DataHub.
- Storage: Choose the relational database and search backend your operators already support.
- Upgrades: Plan for the system-update job the Helm chart runs on upgrades to apply schema changes and reindex search.
- SSO: Configure OIDC on the frontend, since self-deployed DataHub Core requires the operator to set it up.
Confirm which graph index option your platform team will support over time before you choose a deployment pattern.
A team without Kubernetes or Kafka operations faces a larger first decision. Standing up the DataHub services, the streaming layer and the search and graph backends is the main cost, and it recurs as versions change. The Docker quickstart is useful for evaluation, since the README states it needs Docker with 8 GB of RAM and Python 3.10 or later, but it is not a production layout. The choice is therefore between building that platform skill, or engaging an implementation partner to plan, deploy and operate it inside your own infrastructure.
- Skills gap: Identify who will patch Kafka, the database and Elasticsearch or OpenSearch, and how often.
- Production layout: Treat the quickstart as a test environment and plan a separate, supported deployment.
- Alternative managed option: DataHub Cloud is a managed product from the company behind DataHub, with features and terms that differ from open source, so compare them against your data residency rules.
- Hosting: Confirm which cloud or data center will host the components and what it costs your team to run them.
Ask your platform team to estimate the operating hours for this stack before the migration calendar is set, since no figure applies to your environment by default.
Where DataHub runs relative to Google Cloud changes both latency and the data path. Running it inside the same Google Cloud project as BigQuery keeps ingestion traffic local, but it ties the catalog's availability to that project's controls. Running it in your own data center or another cloud means the BigQuery source reads metadata across the network, so credentials, egress and private connectivity need explicit design. The choice should follow your data residency and network policy, not the convenience of the first deployment.
- Same project: Simplest access to BigQuery, with the service account scoped to metadata read permissions only.
- Separate network: Requires private connectivity and a documented credential rotation process.
- Hybrid estates: If Looker or other sources sit elsewhere, place DataHub where most ingestion runs.
- Auditability: Keep the export and ingestion logs in the same region as the governed metadata where policy requires it.
Confirm your residency requirements with your compliance lead and the exact network path with your cloud team before choosing a location.
Questions buyers ask before migrating from Dataplex to DataHub
What does a move from Dataplex to self-hosted DataHub cost?
The cost is a set of components, not a single price: the infrastructure DataHub runs on, the people who build and operate it, and the time spent mapping and validating your metadata. Infrastructure covers the Kubernetes cluster, the database, Kafka, the search service and the graph index that DataHub's Kubernetes guide lists, plus backups and monitoring. Labor covers platform engineering to deploy and secure the stack, ingestion work for each source, and the glossary and ownership cleanup that comes from your Dataplex export. Dataplex billing is one line you stop paying, but it does not offset the work above. Ask any partner for a quote that separates one-time setup from recurring hosting, support and upgrade effort, and that states which sources are in scope. DataHub Cloud is a separate option with its own commercial terms, which you should compare on the same basis.
How long does a Dataplex to DataHub migration take?
The timeline depends on how many sources you ingest, how much of your Dataplex metadata is curated, and how quickly your platform team can stand up the dependencies. A small BigQuery estate with a clean glossary moves faster than one with many entry groups, custom aspect types and years of inconsistent descriptions. The main steps are sizing and deploying DataHub, ingesting BigQuery and Looker, loading curated fields from the JSONL metadata export, reviewing glossary term mappings, and running parallel checks before cutover. Each step has its own dependencies, so the schedule should follow the order of work in a written plan. Ask for a timeline with named milestones and an explicit list of assumptions, such as source count and data owner availability.
Can our own team migrate to DataHub without outside help?
A capable platform team can do it without outside help if it has Kubernetes operations experience, database administration for MySQL or PostgreSQL, and some streaming and search operations knowledge for Kafka and Elasticsearch or OpenSearch. Data engineers need to write or run ingestion recipes and understand the JSONL export structure. Someone must own the OIDC SSO configuration, since on self-deployed DataHub Core the frontend server is configured by the operator. Teams without those skills in-house often bring in a partner on a fixed-scope engagement for deployment, hardening and knowledge transfer. Be honest about the skill gaps before committing, and ask any partner to document the runbooks so your team can operate the platform afterward.
What do Google Cloud admins and data owners need to provide?
Google Cloud admins need to provide read access for the DataHub ingestion service account to BigQuery and Looker, and they need to run or approve the Dataplex metadata export jobs and the Cloud Storage bucket permissions that hold the JSONL output. Their input also determines whether DataHub runs inside Google Cloud, in the same VPC as your data, or in another network that can reach these sources. Data owners need to confirm which descriptions, owners, glossary terms and classifications are authoritative, because those decisions drive the mapping work. Assign a named owner for each domain before ingestion starts so questions do not stall the project. Agree on the access model in writing before any data moves.
Do Dataplex policy tags and access controls carry over to DataHub?
Not automatically. BigQuery policy tags enforce column-level access in BigQuery itself, and DataHub does not replace that enforcement, so keep those controls in BigQuery as the source of truth. DataHub can ingest and display classification and ownership metadata, and its governance features help teams find and request data, but authorization rules have to be designed and tested in DataHub separately. Map each Dataplex role to a DataHub role or policy on paper first, then verify with test users that restricted data stays restricted. Confirm the exact behavior of each feature with the current DataHub documentation for your version before relying on it for compliance evidence.
How do we upgrade DataHub versions safely?
Upgrade in a staging environment first, using the same Helm chart values and dependency versions as production, and take backups of the database and search indices before you start. The DataHub Helm chart runs a system-update job on upgrades to apply schema changes and reindex search indices when needed, so plan for that job to take time and watch its logs. Read the release notes for each version between your current and target release, because some upgrades carry breaking changes. Keep a rollback path that includes restoring the database backup, and test SSO, ingestion and search after each upgrade. Pin your chart and image versions in version control so upgrades are deliberate, and decide whether your team or a retainer partner owns this routine.
Should DataHub run inside Google Cloud or outside it?
Run DataHub close to the metadata sources and to the people who use it, and keep it inside the network boundary your policies require. Running in the same Google Cloud project or VPC reduces network paths to BigQuery and Looker and simplifies private access, but it ties the catalog to that cloud's operations. Running outside Google Cloud, such as in your own data center, gives you more control over hosting and fits estates that span several clouds, but it requires secure connectivity for ingestion. Either choice needs the same controls for backups, identity and logging. Document the decision with your security team and revisit it if your source estate changes.
How DeployOpen scopes, runs and supports a Dataplex to DataHub migration
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted DataHub inside your own infrastructure, including the Google Cloud, Kubernetes and data services it depends on. Engagements start from a written scope that sets out which Dataplex metadata moves, where DataHub runs and what the team takes over.
How we scope
- Inventory the Dataplex entry groups, sources, glossary terms and curated fields that must carry over into DataHub.
- Decide with you where DataHub should run relative to Google Cloud, including network paths to BigQuery and Looker.
- Agree in writing which BigQuery and Looker ingestion sources are in scope, and which metadata is retired instead of moved.
How we build
- Deploy DataHub on Kubernetes with the Helm charts, plus Kafka, the MySQL or PostgreSQL database and search services it needs.
- Configure BigQuery and Looker ingestion, and map JSONL metadata export fields to DataHub glossary terms and ownership.
- Set up OIDC single sign-on and test lineage, glossary and search against the scope before any user is onboarded.
How we hand over
- Document the architecture, ingestion recipes, field mappings and the decisions taken during scoping.
- Walk your platform and data governance teams through running ingestion, glossary changes and access control.
- Provide a runbook for upgrades, including the system-update job that DataHub's Helm chart runs to apply schema changes.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer after the handover is complete.
- Ask any quote to name the Kubernetes, database, Kafka and search components in scope, and who operates each one.
