Export metadata from Google Cloud Dataplex

How to export metadata from Google Cloud Dataplex before your contract ends

The documented export path is a Knowledge Catalog metadata export job, which writes your catalog entries as JSONL files to Cloud Storage. You choose the scope of the job at the organization, project or entry group level, and the output covers the entry metadata inside that scope. It does not carry over everything your team configured in the console, so treat the export as the start of your inventory, not the end of it. If your renewal date, a pricing change, a data residency rule or a wish to control the catalog is forcing the move, pick the scope that matches the estate you need to keep, then confirm with Google which entry metadata and configuration fields the current export includes before you rely on it.

The export is a copy of the catalog, not a migration. Output is split into files of up to 1,000,000 entries, so a large estate produces several files that your loading scripts must read as one set. Query the files with BigQuery to count entries by project, entry group and type, and compare those counts with the catalog before you delete anything. Expect to rebuild the parts the export does not transport: IAM access policies, the lineage sources you connected, and the curated business context such as glossary terms, owners and descriptions that people edited in the interface. Your target catalog, whether OpenMetadata or DataHub, needs these rebuilt as its own configuration.

Before you run the job, decide where the Cloud Storage bucket sits and who can read it, because the export contains metadata about your data estate and may fall under the same residency rule as the data itself. Set bucket access controls, retention and location deliberately, and document them as part of your exit record. Keep the export files until the target catalog has been loaded, checked against your inventory counts and signed off by the data owners who depend on it. Ask Google, in writing, how long the catalog and the export remain available during your notice period and whether any export feature changes before your end date.

What Knowledge Catalog metadata export jobs can export

This table lists what a Knowledge Catalog metadata export job produces, how the organization, project or entry group scope changes which entries land in the files, and what to confirm before planning a migration. Read the scope rows first, because they set the file set you will later transform. The main takeaway is that the export gives you a copy of catalog metadata, so mapping that copy into OpenMetadata or DataHub carries most of the migration effort.

Export element	What it covers	What to confirm
Entries and their metadata	Catalog entries and their metadata, exported at one of three scopes	Which metadata fields each entry type includes in the JSONL output
Scope: organization	Entries from every project in the organization	Export identity has read access across all projects in scope
Scope: project	Entries registered in one Google Cloud project	Whether you need one job per project for a multi-project estate
Scope: entry group	Entries that belong to one entry group	Whether custom entries and curated fields sit in the group you export
Export method	A metadata export job started from Knowledge Catalog	Current console and API steps, and the IAM roles the job requires
Output format	JSONL files written to a Cloud Storage bucket, one record per line	Field names and schema version in the current documentation
File size limit	Each output file holds up to 1,000,000 entries	This limit in Google's current export documentation, and how files are split
Query with BigQuery	Load the JSONL output to filter, count and join entries before migration	Current load or external table options for JSONL, and any access controls on the bucket
Transform for OpenMetadata or DataHub	Map descriptions, owners, glossary terms and custom properties to the target model	Separate transform for each target, since OpenMetadata and DataHub ingest metadata differently

What the metadata export does not carry and how to rebuild it

A metadata export captures the catalog entries you see in Knowledge Catalog, but several things that teams assume will come along sit in other Google Cloud services or in the console. The cards below cover each item, the workaround for it, and what to confirm in Google's documentation before you plan the migration.

Access control and IAM bindings

IAM policies on projects, datasets and entry groups decide who can read or change catalog content, and those bindings are not metadata you can carry into OpenMetadata or DataHub. In the target, access comes from the target's own roles, groups and SSO mapping, so you must decide which Google groups map to which catalog roles. Confirm with Google's documentation whether your export job includes any policy data, and list every principal with a binding before you cut over.

BigQuery policy tags and access rules

Policy tags attached to BigQuery columns enforce column-level security in BigQuery itself, and they do not stop being enforced when the catalog changes. Treat the taxonomy as a separate artifact: export the taxonomy and tag assignments through the Data Catalog and BigQuery interfaces, then map tag names to OpenMetadata classifications or DataHub tags as descriptive labels only. Confirm current coverage in Google's documentation before assuming the export covers taxonomy or assignments.

Lineage captured by Google Cloud sources

Lineage that Google Cloud recorded for BigQuery jobs, pipelines and other services is held in Google's lineage system and is not guaranteed to appear in the metadata export. Plan to re-create lineage from the source systems: use the OpenMetadata connectors or DataHub ingestion for BigQuery, and add OpenLineage events for pipelines that do not emit lineage natively. Verify which lineage edges your export contains before deciding how much to rebuild.

Glossary terms and curated business context

Business glossary entries, owners, descriptions and stewardship notes often live in the Knowledge Catalog console and were edited by people rather than pipelines. Check whether your export job returns these fields for the scope you choose, and compare the output against a sample of entries in the console. Where a field is missing, rebuild it from a stewardship spreadsheet or a short review with data owners, then load it through the target's glossary and ownership APIs.

Aspect types and custom metadata definitions

Aspect types define the structure of custom metadata attached to entries, and they are the schema behind any custom fields you use. An export carries entry values, but the definitions that give those values meaning need to be re-created as custom properties in OpenMetadata or as custom aspects or structured properties in DataHub. Inventory every aspect type in use, document its fields and allowed values, and confirm with Google's documentation which definitions the export file includes.

Scans and data quality results

Data quality scans and their results are run by Google Cloud and stored with the scan configuration, so historical pass and fail records and rule definitions are not automatically portable. Export the scan rule definitions through the documented Google Cloud interfaces, then re-create the rules as tests in OpenMetadata or assertions in DataHub. Keep the historical results in a separate archive if auditors need them, and confirm current export coverage in Google's documentation.

Running a complete metadata export in seven steps

This sequence produces a verified, dated copy of your Dataplex metadata in Cloud Storage, checked against the catalog and ready to transform for OpenMetadata or DataHub. Each step ends with a check you can record as evidence.

Inventory entry groups and projects in scope
Decide whether the export runs at organization, project or entry group scope, then list every project and entry group that falls inside that scope. Record the entry count for each one from the catalog so you have a baseline. Assign an owner to the list, since the export can only be checked against an inventory that someone has approved. You are done when the inventory has a named owner, a scope decision and a per-project entry baseline saved with the change record.
Confirm IAM permissions and the export bucket
Confirm which IAM roles the export job and the person running it need, using the current Google Cloud documentation for the Knowledge Catalog export feature, and assign only those roles. Create or select a Cloud Storage bucket in a location that meets your data residency rules, and set retention and versioning according to your records policy, which your compliance team should confirm. You are done when a test run of the permissions check passes and the bucket location, retention and access settings are written into the change record.
Run the metadata export at the chosen scope
Start the metadata export job at the scope you chose in the first step, pointing its output to the bucket you prepared. Watch the job until it reports completion, and note the job identifier, start time and any errors or warnings. Keep the job configuration with the record so the export can be reproduced later. You are done when the job shows a successful status and its output location is recorded.
Reconcile JSONL file count and entry totals
Count the JSONL files the job wrote and count the entries across them. Compare the total against the baseline from the catalog for the same scope. Each file holds up to 1,000,000 entries, so the number of files should match the total entries divided by that limit, rounded up. Investigate any gap before continuing, and document the reason for any difference that is expected. You are done when the file count and entry total match the catalog baseline or each difference has a written explanation.
Spot-check entry types and required fields in BigQuery
Load or external-table the JSONL output in BigQuery and query it to see which entry types are present. For each type, check that the fields your target platform needs are populated, such as display names, descriptions, owners and glossary links. Compare null counts against what you expect from the catalog, and sample a few entries by hand to confirm the values match the catalog UI. You are done when the query results for each entry type are saved and every gap has an owner.
Store a dated, verified copy for audit
Copy the verified export to a separate, dated location in Cloud Storage that your team controls, following the retention and access rules you confirmed in the second step. Record the job identifier, scope, timestamp, file count, entry total and checksums for each file in an export manifest. Restrict write access to the copy so the record cannot change without a trace. You are done when the manifest is stored alongside the files and an independent reviewer has confirmed the figures match.
Transform the files and repeat the export before cutover
Convert the JSONL into the input your target platform expects, using the REST APIs or Python SDK for OpenMetadata, or the ingestion tooling for DataHub, and load a test environment first. Verify the loaded counts and a sample of entries against the export manifest. Shortly before cutover, repeat steps three to six so the final copy reflects the current catalog, and reconcile the new manifest with the previous one. You are done when the test load matches the manifest and the final export is verified and dated.

Converting the JSONL for OpenMetadata or DataHub

OpenMetadata fits teams that want the export's curated fields, such as glossary terms and descriptions, written through the platform's own interfaces. The project documents REST APIs, a Python SDK, TypeScript packages and Java libraries, so a team can script the conversion from the JSONL and call the API to set terms and descriptions on matching assets. Technical metadata such as schemas, tables and lineage normally comes from the more than 130 connectors, which leaves the custom work focused on the curated fields.

Field mapping: Decide which Dataplex fields become OpenMetadata glossary terms, descriptions, tags and owners, and which describe Google Cloud resources that have no equivalent and should be dropped.
Asset matching: Match each exported entry to the fully qualified name that the connector generates, so updates land on the correct table or column. Confirm the naming with a test ingestion first.
Load method: Use the Python SDK for batch jobs run by a platform team, the REST APIs for integrations built in other languages, and the Java libraries for JVM services.
Upgrade checks: Review the OpenMetadata 2.0 release notes before upgrading. The breaking changes include moving embedding credentials into an llmConfiguration block, a new Databricks auth format, rebuilding custom ingestion images for Python 3.12 and migrating Great Expectations to 1.3.

Run the mapping against a small set of entries in a non-production instance, compare the results in the interface, and only then run the full load.

What to plan before the Dataplex contract ends

The export is the easy part to start and the easiest to leave late. These decisions set the order of work, the cost of the parallel period and who signs off that the new catalog matches the old one. Settle them before the final export runs.

Retention of export copies

The JSONL files land in Cloud Storage, and they become your only copy of curated descriptions, glossary links and ownership once the catalog is gone. Decide how long they must be kept, which bucket and project hold them, and who can read them. Apply object versioning or a retention policy and confirm that the archive stays in the jurisdiction your data rules require. Check whether your records policy treats catalog metadata as a business record.

Timing of the final export

Schedule the last export run close to the cutover date so curated edits made during the parallel period are not lost. Run an earlier full export as a dry run to size the output, then take a final snapshot after editing freezes. Record which export ran last and keep its job identifier with the archive. Confirm the scope you export (organization, project or entry group) matches everything the target catalog must hold.

Ownership of transform and verification

Name one owner for the transform from JSONL to OpenMetadata or DataHub and a separate reviewer for the results. The owner writes the mapping for glossary terms, entry groups and descriptions; the reviewer samples entries and compares owners, tags and lineage against the source. Agree in writing who approves the cutover. Keep the mapping script in version control so changes after the first run are traceable.

Hosting choice for the target platform

Choose the hosting model before procurement. OpenMetadata documents production deployment on Kubernetes (Helm), on bare metal and Docker Compose for evaluation only. Its bare metal minimum is 2 vCPUs and 4 GB RAM for the server, with the database and search services each needing at least 2 vCPUs, 2 GB RAM and 30 GB storage. DataHub's Kubernetes guide needs Kafka, a relational database and search, and notes more than 7 GB of RAM on Minikube. Size the production cluster with your vendor or DeployOpen, and do not treat these minimums as production capacity.

Parallel run and reconciliation

Run the old and new catalogs side by side for a fixed period and compare them on a checklist: entry counts per source, owners, glossary terms, descriptions and lineage for a sample of critical tables. Track which differences are expected, such as metadata that the target now generates automatically, and which are defects. Users should search and report issues in the new catalog during this window. Define the exit criteria before the run starts, and set the period length with your team, since no duration is fixed here.

Decommissioning Google Cloud resources

Once the cutover is signed off, remove the Google Cloud resources that only existed for the catalog: Dataplex configuration, entry groups, custom aspect types, scheduled export jobs and any IAM bindings granted to service accounts. Keep the archived exports and policy tags that BigQuery still enforces. Confirm billing, cancellation terms and any notice period directly with Google Cloud, and do not assume one. Remove the resources only after the archive has been verified as readable.

Questions teams ask before exporting Dataplex metadata

How much effort does the export itself take compared with the transform and verification?

For most teams, the export job is the smallest part of the work. Setting up a metadata export job in Knowledge Catalog and writing its JSONL output to Cloud Storage is a configuration task that an engineer with the right Google Cloud access can run. The larger effort sits in three places: transforming the JSONL records into the structure OpenMetadata or DataHub expects, loading curated fields such as descriptions, glossary links and ownership, and reconciling counts and identifiers between source and target. Verification often takes longer than the export because each mismatch requires a decision about which system is the source of truth. Ask any implementation partner to split its scope into export, transform, load and verification so you can see where the effort goes.

What makes the migration effort go up or down?

Four factors drive the effort most:

The number of projects and entry groups in scope, since each one usually needs its own export scope and reconciliation.
How much curated context lives in Dataplex, such as descriptions, glossary links, owners and classifications. Teams with little curation have less to load and verify.
The number of custom aspect types that must be mapped to custom properties or classifications in the target catalog.
The number of target sources the new catalog will ingest. Connectors for BigQuery and other systems add configuration and testing work, even when the metadata is already exported.

Reducing scope to the entry groups that people actually search, and retiring unused aspect types before the export, usually shortens the work.

What does our side need to provide before the work starts?

Your team needs to supply six things before work starts:

IAM permissions to create and run metadata export jobs at the organization, project or entry group scope you choose. Confirm the exact role names with Google Cloud documentation and your security team.
A Cloud Storage bucket that receives the JSONL output, with access controls and retention set to your policy.
BigQuery access for querying the export and checking counts.
A named owner who approves mapping decisions and signs off the reconciliation.
A target environment, such as your own cloud account or data center, where OpenMetadata or DataHub will run. The OpenMetadata deployment documentation lists the supported hosting patterns.

Missing any of these typically delays the first export run.

Can our team run the export in-house?

Yes, your own engineers can run the export job itself. The common gap is the steps after the export: mapping aspect types, carrying curated fields into the target catalog, and reconciling entries against what the new catalog ingests. Before deciding, check whether someone on your team has loaded metadata into a data catalog before and can own the mapping decisions for the length of the project. If that skill is missing, a fixed-scope engagement can cover the transform and verification while your team keeps ownership of the export. DeployOpen plans, implements, integrates and documents self-hosted platforms inside client infrastructure, and it prepares evidence for compliance assessments. Scope, deliverables and timelines are agreed in writing before work starts.

How does the 1,000,000-entry file limit affect processing?

Each export output is split into JSONL files of up to 1,000,000 entries, so a large estate produces several files that your transform must read as one dataset. Pipelines should loop over every file, track entry counts per file and avoid assuming that one file maps to one project or entry group. For example, an illustrative export of 2,500,000 entries would produce three files. Before loading anything into the target, count the records in each file with BigQuery and compare the total with the number of entries Knowledge Catalog reports for the same scope. Keep the file names and counts in the reconciliation record so a later rerun can be checked against the same baseline.

How do we confirm the exported metadata is complete?

Completeness checks should run at three levels. First, compare entry counts by project and entry group between the export, BigQuery and the catalog itself. Second, sample entries across each type and confirm that identifiers, descriptions, glossary links and owners survived the transform. Third, check lineage and classification fields against a sample of assets that business owners know well. Record each result in a reconciliation report, with the count, the sample method and the owner who accepted the differences. Ask your implementation partner which checks it will run, how many differences it will accept before escalation, and who signs the final report. This record supports internal review, but it is not an audit report.

What should we ask for in a written scope?

A written scope should state:

Which export scope applies, at organization, project or entry group level, and the list of projects included.
Which target runs the catalog, OpenMetadata or DataHub, with the version pinned and the hosting location named.
Which fields are transformed, how aspect types map to custom properties or classifications, and what is left out.
Which verification method applies, the acceptance criteria and who signs them off.
Who handles IAM, the Cloud Storage bucket and BigQuery access, and what your team does.
What documentation and handover you receive, and what post-cutover support is included.
How change requests are priced and approved.

Ask for a fixed-scope quote built from this scope. DeployOpen does not publish a price list, so the quote follows from the written scope.

How DeployOpen scopes and delivers a Dataplex metadata exit

DeployOpen plans and implements self-hosted open-source catalogs inside the client's own infrastructure. For a Dataplex metadata exit, we start by agreeing in writing which export scope, output location and target catalog the work covers, then build and hand over the pipeline that moves the metadata into your environment.

How we scope

Confirm whether the export runs at organization, project or entry group scope, based on the catalog areas you need to keep.
Record where the JSONL output will sit in Cloud Storage and who in your organization controls access to it.
Agree in writing whether the target is OpenMetadata or DataHub, and which Dataplex fields are in scope for transfer.

How we build

Set up the metadata export job and a reproducible transformation that maps exported entries into the target catalog's model.
Query the JSONL output with BigQuery to check record counts, field coverage and handling of files split at 1,000,000 entries.
Test the transformed output in a non-production instance of the target catalog before any production load.

How we hand over

Document the export configuration, transformation logic and mapping decisions so your team can rerun the exit without us.
Provide a reconciliation report that compares entry counts and curated fields between Dataplex and the new catalog.
Walk your engineers through the runbook for scheduled exports and the steps to retire the Dataplex copy when you decide.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing of the target catalog are available as a retainer.
Ask for a written quote based on your scope, since DeployOpen does not publish a price list.

How to export your metadata from Google Cloud Dataplex