Automated PII discovery and classification with an open source catalog

Why automated PII discovery is a different problem in a regulated estate

A regulated organization must know where personal data sits before it can answer GDPR records of processing, data subject access requests or breach scoping. OpenMetadata auto-classification can find and tag candidate PII in a self-hosted catalog, but its tags are a starting point that needs tuning and human review. The tags describe where personal data probably sits, not where it is confirmed to sit, and an auditor or data protection lead will ask how each tag was decided and by whom. For most estates, OpenMetadata is the platform to pilot for classification, with DataHub as the alternative to evaluate if its lineage, search and governance model fits the organization better.

Tagging is harder than it looks because the catalog only sees what the connectors reach. Shadow copies of customer extracts in analyst file shares or notebooks are often outside any connector scope. Legacy warehouses may use cryptic column names such as cust_ref2 or addr_l3, so name-based rules miss them. Free-text columns such as comments, case notes and free-form address fields can hold personal data that no column name reveals. Each of these gaps means the catalog's view of personal data is partial, and the reader should record which sources are in scope before trusting any count of tagged columns.

Before enabling auto-classification on production sources, decide three things. First, which classifiers run: name-based rules alone are cheaper to run, while sample-based entity recognition reads row values and needs its confidence threshold reviewed (OpenMetadata defaults to 80). Second, who approves a tag and who can overturn it, since a wrong tag flows into access controls and retention decisions. Third, how the PII.Sensitive tags map to your confidentiality labels, because tag mappings can apply a second tag automatically. Confirm with the OpenMetadata release you plan to run, because sample data storage handling changes between releases and the deprecation timeline should be checked in the current documentation.

Which GDPR obligations drive the discovery build

This table maps five GDPR obligations to what the discovery output must show and what each one means for the build. Read it as a list of evidence the catalog has to produce, not as a claim that any single tool satisfies the regulation. The GDPR requires no audit or certificate, and Article 42 certification is voluntary and does not reduce the controller's or processor's responsibility.

Obligation	What the discovery output must show	What it means for the build
GDPR Article 30 records of processing	Inventory of personal data columns by system, with owners, purposes and storage locations	Scan the databases and warehouses in scope, tag personal data columns, and assign a named owner to each dataset before the inventory is treated as complete
Article 32 security of processing	Which personal data sits in which systems, and which of those systems carry the highest confidentiality labels	Map PII tags to confidentiality labels so access controls and encryption reviews can be prioritized, and confirm the catalog's own access and storage controls
Articles 33 and 34 breach notification scoping	Lineage and tags that show which systems and tables would contain personal data if an incident occurred	Keep lineage current for pipelines feeding personal data tables so scoping after an incident does not depend on tribal knowledge
Chapter V transfers to third countries	Flags showing where personal data is stored, replicated or consumed outside the EU or EEA	Record storage location and downstream consumers as catalog attributes, and confirm with legal which transfer mechanisms apply to each flagged system
Article 5(2) accountability	Dated, reviewable evidence of classification runs, reviewer decisions and remediation of false positives	Keep classification results, review sign-offs and tuning changes in a retained, versioned form so the organization can show its process to a supervisory authority

Column name rules versus entity recognition on sample data

The column name scanner reads only metadata the catalog has already ingested, such as table names, column names and data types. It matches those names against regex rules, so a column called email_address or date_of_birth is tagged PII.Sensitive without the catalog reading a single row. Because no values leave the source, this method is fast, inexpensive to run on every metadata sync and easy to explain to an auditor.

Strength: results are deterministic. The same name produces the same tag on every run, which keeps steward review queues stable.
Blind spot: opaque names. A column called col_17 or attr_b carries no signal, so personal data inside it stays untagged.
False positives: names that sound sensitive but hold codes, such as a phone_model field in a device inventory or a dob_index key in an analytics table. Stewards reject these most often.
Naming drift: legacy abbreviations, local-language names and vendor-specific prefixes fall outside default patterns. The team that knows the schema must extend the rules.
Repeatability: because the method is cheap, it is the natural first pass across the whole estate before any value-level scanning starts.

Before you trust a clean scan, confirm which rules ship with your OpenMetadata version and test them against your own naming conventions.

When sample data ingestion is enabled, OpenMetadata's NLP-based entity recognition engine reads row values and assesses whether a column holds entities such as email addresses, person names or phone numbers. This catches personal data in columns with neutral names and in free text fields, where a name-based rule has nothing to match. The output depends heavily on the confidence threshold, which defaults to 80.

Coverage: finds personal data in columns called notes, comments or field_3, and in text fields that mix names with other content.
Confidence threshold: raising the threshold above the default of 80 reduces false positives but can miss true matches, and lowering it does the reverse. Test candidate values on a sample of tables from each source before setting a production value.
False positives: product codes that resemble phone numbers, place names that match person names and reference numbers formatted like dates. Expect these in numeric and code-heavy columns.
Operational footprint: reading rows means the catalog connects to production data, so the database account, network path and sample retention need approval from the data owner and security.
Tuning record: keep the threshold, the sample scope and each reviewer decision per source so the setting can be defended later.

Ask the implementation partner to show the false positive rate observed on your own tables during the pilot, and who signed off the threshold.

Which catalog platforms fit PII discovery and why

Two catalog platforms cover most PII discovery work in open source, and each brings a different classification model. Supporting pipeline and gateway components matter only when sample data has to pass through them. The right choice depends mainly on whether the estate is warehouses and databases or a wider mix of tools, and on how much classification depth the team needs out of the box.

OpenMetadata for classification-led discovery

OpenMetadata is Apache-2.0 licensed and runs auto-classification as an Auto Classification Agent on a database service or as an external YAML workflow. Its column name rules and, when sample data ingestion is enabled, entity recognition over row values tag columns as PII.Sensitive. It fits best when the estate is mainly warehouses and databases, because the classification workflow is the core feature rather than an add-on. Confirm which connectors your sources need, since the project lists more than 130.

Tag mappings between PII and confidentiality

OpenMetadata tag mappings link related tags so that applying one applies the other, for example Personal Data.Personal to Data Classification.Confidential. This turns a PII finding into a confidentiality label without a manual second step. Check the mappings with your data governance lead before enabling them, because a mapping that is too broad will label more columns than the policy intends.

DataHub for lineage and wide integration

DataHub is Apache-2.0 licensed and its README lists more than 150 integrations, column-level lineage and data quality assertions. It is a strong choice when PII findings must travel with lineage across BI tools, pipelines and warehouses. Its classification depth is the point to verify: confirm with the vendor and in a pilot whether its sampling and tagging rules reach the column-level detection you need, and how those rules are configured on self-deployed DataHub Core.

DataHub Cloud is a separate commercial offer

The company behind DataHub also sells DataHub Cloud, which its README describes as managed and SLA-backed with no infrastructure to run. Self-hosting the open-source project keeps metadata and sample values inside your own environment, but the operations work moves to your team. Ask any vendor which features are in the Apache-2.0 release and which are in the paid offer before you compare costs.

Pipelines when sample data must be read

If sample rows must be read through an intermediary, an orchestrator such as Apache Airflow (Apache-2.0) or Dagster (Apache-2.0) can run the scans on a schedule and keep credentials in one place. Airbyte's platform repository is Elastic License 2.0, which matters if you plan to offer it as a managed service. Use these only where a pipeline already exists, since a sampling job adds another system to secure and patch.

Gateways for controlled access to samples

An API gateway such as Apache APISIX or Kong, both Apache-2.0 licensed, can sit in front of the catalog API so that scanners and users reach metadata through authenticated, logged routes. This helps when auditors ask who read sample values. It is optional for a single-team deployment, and it adds configuration you must maintain, so decide on it early rather than retrofitting it after rollout.

Deployment patterns for a self-hosted catalog in regulated networks

A PII scan is only as trustworthy as the catalog that holds its results and the network paths it uses to reach source systems. The cards below cover the hosting choice, the bare metal stack, scanner connectivity and access configuration that a regulated team should settle before the first classification run.

Kubernetes on your own cluster

OpenMetadata documents production deployments on Kubernetes across AWS EKS, Azure AKS, Google GKE and on-premises clusters. For PII work, the question is whether the cluster sits inside the same control boundary as the databases being scanned, and whether your platform team already operates the ingress, secrets and backup services the catalog depends on. Confirm which cluster your data protection policy allows for metadata that describes personal data columns.

Bare metal prerequisites

Without Kubernetes, the OpenMetadata bare metal guide requires Java 21, PostgreSQL 15 or later or MySQL 8.0.42 or later for the metadata store, and Elasticsearch 9.x or OpenSearch 3.x for search. Version drift in any of these is a common upgrade blocker, so pin them in your change records and confirm support for each version with the vendor before build. Ask your team who patches the database and search tier on the same schedule as the catalog.

Sizing minimums are a floor

The bare metal documentation gives minimum server sizing of 2 vCPUs, 4 GB RAM and 20 GB storage, with the database and search services each needing at least 2 vCPUs, 2 GB RAM and 30 GB storage. Those figures only start a trial. Production sizing depends on the number of tables and columns, how many rows the sample-based scanner reads, profiling frequency and retention of scan history. Ask for a sizing estimate based on your estate's inventory, not the minimums.

Scanner network access to sources

Auto-classification runs as an agent or external workflow that connects to each database service and, when sample data is enabled, reads row values. Map which subnets, ports and service accounts the scanner needs, and whether it may run from a segment that reaches production databases. Agree with the data owners whether sampled values ever leave the source network, and confirm that external sample data storage, which OpenMetadata has deprecated, is not part of the design.

Single sign-on as a separate decision

DataHub documents OIDC single sign-on for self-deployed DataHub Core, where the frontend must be configured by the operator. OpenMetadata access control is a distinct configuration path, so treat identity integration as its own design item. Decide which identity provider issues catalog roles, who may view sampled values or classification evidence, and how stewards are onboarded and removed. Confirm each option with the vendor documentation for the version you deploy.

Backup and upgrade path

The catalog stores PII tags, classification decisions and review history, so losing it means losing evidence of what was found and who approved it. Plan backups of the metadata database and search index on the same cadence as your other regulated systems, and test restores. Upgrades also carry breaking changes, such as the OpenMetadata 2.0 changes to embedding credentials and ingestion images, so read release notes before each move.

How a PII discovery rollout moves from pilot to reviewed records

The sequence starts with a source inventory and ends with scheduled classification runs that stewards review and privacy owners accept. Each phase produces a named artifact so the client can see what is decided and what is still open.

Inventory sources and agree the scope
DeployOpen works with the client's data owners and privacy lead to list the database services, warehouses and BI sources in scope, record which systems hold customer, employee or patient data, and name an owner for each. The deliverable is a source inventory with an owner and a data classification of its own. The decision needed from privacy and data owners is which systems are in the first pilot and which are excluded until their legal basis and access rules are confirmed. The step closes when the scope is signed off in writing.
Deploy OpenMetadata and connect one pilot service
DeployOpen installs OpenMetadata in the client's own infrastructure, following the supported deployment path the client's platform team prefers, and connects one database service from the inventory. Metadata ingestion runs first so tables and columns are visible before any classification is attempted. The deliverable is a working catalog with one ingested service, SSO configured, and a runbook for the ingestion workflow. The step is done when the platform team can reproduce an ingestion run without help.
Enable sample ingestion and run the first classification
With the pilot service connected, DeployOpen enables sample data ingestion on the auto-classification agent and runs classification with the default confidence threshold. Column name rules and entity recognition on row values both produce candidate PII.Sensitive tags, and the run log shows which engine produced each tag. The deliverable is a first results set for the pilot service, with the raw counts kept for comparison in later runs. Before running on production data, the client confirms in writing that sample values may be read and stored in the chosen location, since external sample storage is deprecated.
Tune rules and thresholds against a labeled sample
The privacy team labels a sample of columns as personal, sensitive or not personal, and DeployOpen measures how many tags the rules and entity recognition got right and wrong. Column name rules that fire on generic names such as reference or code are tightened or excluded, and the confidence threshold is adjusted per engine if the false positive rate is too high. The deliverable is a tuning record listing each rule change, the threshold used and the labeled sample it was tested against. The step is done when the privacy owner accepts the error profile in writing.
Map PII tags to confidentiality labels
DeployOpen configures tag mappings so that applying a personal data tag also applies the client's confidentiality classification, for example mapping Personal Data.Personal to Data Classification.Confidential. The client's information security owner decides which tags map to which labels and whether any tag should trigger a stricter label than the default. The deliverable is a mapping table approved by security and privacy, with a test showing the derived label appears on a sample column. The step is done when the mapping has been reviewed against the classification policy.
Set up the steward review workflow
DeployOpen defines who reviews new and changed PII tags, how a steward accepts, rejects or escalates a tag, and where the decision is recorded inside the catalog. Stewards are assigned per dataset from the source inventory, and reviews are tracked so that unreviewed tags are visible to the privacy team. The deliverable is a written review procedure, a steward assignment list and one completed review cycle on the pilot service. The client decides who has final authority over disputed tags and how long unreviewed tags may remain before escalation.
Schedule recurring runs and hand over
DeployOpen schedules classification runs for the pilot service and any further services the client approves, and documents how new tables, schema changes and failed runs are detected. The platform team receives the runbook, upgrade notes and the tuning record, and the privacy team receives the review procedure and the list of records that depend on the catalog. The deliverable is a handover pack and a scheduled run history. The step closes when the client's team has run one scheduled classification without DeployOpen involvement, and any upgrade of OpenMetadata is agreed as a separate piece of work.

What a regulator or auditor will ask to see from discovery

This table lists the evidence a supervisory authority, auditor or internal assurance team is likely to request from a PII discovery run, the component that produces each item and where the evidence usually breaks down. Use it to decide what the catalog must retain before the first scan runs. DeployOpen prepares this evidence, but auditors and supervisory authorities judge whether it is sufficient.

Evidence item	Produced by	What reviewers check
Personal data columns with tags and timestamps	OpenMetadata auto-classification run history and column tag records	Each tagged column, the tag applied (PII.Sensitive or Personal Data), the scan date and the source service it belongs to
Classification rules and threshold settings with change history	Auto Classification Agent or external YAML workflow configuration, kept in version control	Column name rules, the entity recognition confidence threshold (default 80) and who changed it and when
Tag mapping configuration	OpenMetadata tag mappings linking Personal Data tags to Data Classification labels	That applying a personal data tag also applies the confidentiality label the policy requires, and the mapping is current
Reviewer decisions and rejected false positives	Catalog review workflow and task history, exported on a schedule	Named reviewer, decision date, reason for each rejection and evidence that rejected tags were not silently reapplied
Ownership assignments per dataset	Catalog ownership records linked to tables and columns	A named accountable owner for each dataset holding personal data, and proof the owner confirmed the assignment
Lineage to downstream extracts	Table, column and pipeline lineage captured through connectors or OpenLineage events	Whether personal data flowing into analytics, exports or third-party transfers is traced to its downstream copies, and where lineage is missing
Scan scope and coverage record	Connector and service configuration plus run logs	Which databases, warehouses and schemas were in scope, which were excluded and why, and when unscanned sources were last reviewed

How to tune confidence without hiding real personal data

The confidence threshold is the setting that decides how much personal data the catalog finds and how much noise it creates, and a high value can quietly hide the columns you most need to see. In OpenMetadata, the NLP-based entity recognition over sample row values uses a confidence threshold that defaults to 80. Raising it reduces false positives such as product codes that look like account numbers, but it also drops genuine matches where the values are short, unusual or written in a local format. A high threshold produces a clean report that is incomplete, and nothing in the report shows what was dropped. Treat the threshold as a trade-off between precision and recall that privacy staff must accept in writing, not as a technical default.

Measure both error types before you rely on the output. Take a sample of columns from each source, have privacy staff label every column as personal, sensitive or not personal, then run auto-classification with the threshold you intend to use and compare the results. Count false positives, which drive reviewer workload, and misses, which are the risk. Misses matter most in free-text fields, national identifiers, and columns named with abbreviations the column name scanner does not recognize. Confirm with your own data whether the scanner's regex rules or the entity recognition engine found each miss, because the fix differs: a missing name rule is a configuration change, while a low-scoring value match may need a lower threshold for that source only.

Tuning is per source, and it must be repeated after every catalog upgrade. A threshold that works on a customer table may fail on a support ticket export or a legacy system with different encodings. Keep the test sample and the settings for each service under change control, so the same test can be rerun. OpenMetadata 2.0 made Dynamic Sampling the profiler default, which changes what sample rows the catalog reads, so a result from 1.x or an earlier 2.x build cannot be assumed to hold. Ask your implementation partner to rerun the labeled sample after each upgrade and record the change in false positives and misses, and confirm with the OpenMetadata release notes which other defaults changed in your target version.

What do buyers ask before hiring help for PII discovery?

What drives the cost of a PII discovery project with OpenMetadata?

Cost follows the size and shape of the data estate: how many database and warehouse services need connectors, how many tables and columns sit in scope, and whether sample data is read for entity recognition. Software licensing is not the main line item because OpenMetadata is released under the Apache License 2.0. The recurring costs are the infrastructure for the metadata database and search service, the time stewards spend reviewing flagged columns, and upgrade work. Ask any partner for a written scope that lists connector count, schema count, sampling rules, review capacity and the upgrade cadence it will support, so the quote can be compared line by line.

How long does automated PII discovery take to deploy and run?

Duration depends on the number of sources, how much access approval is needed from database and platform owners, how many classification rules are tuned, and how long the review cycle takes for the privacy team. Deployment of the catalog itself is only one part of the work. The fixed-scope written agreement should state phases, deliverables and dates before work starts, and it should separate the first full scan from the ongoing scan schedule. Ask the partner to name the sources it expects to be blocked on and who in your organization is responsible for unblocking them.

What do our data owners and privacy team need to provide?

Data owners need to confirm which systems are in scope, who is accountable for each dataset, and which columns they already know contain personal data. The privacy team needs to agree on the PII categories to tag, the mapping from those tags to confidentiality labels, and the rule for who signs off a classification before it is treated as final. Engineers need read-only database access and a service account with a defined scope. Ask the partner to list these inputs in the scope document with a named owner for each one.

Can an internal team run PII discovery without an implementation partner?

Yes, a team with data platform and catalog skills can install and operate OpenMetadata, but the work is larger than a single deployment. The team must set up connectors, tune confidence thresholds to limit false positives, write mapping rules, schedule scans and keep the review queue moving. Bare metal installs need Java 21, a supported database and Elasticsearch 9.x or OpenSearch 3.x, and the team must patch them. Weigh the internal capacity against the number of connectors and the review load. A partner can take over part of the build, the tuning or the operations, and the written scope should say which part.

How are OpenMetadata upgrades and breaking changes handled?

Upgrades need a plan because the 2.0 release made breaking changes, including moving embedding credentials into an llmConfiguration block, a new Databricks auth format, rebuilding custom ingestion images for Python 3.12, and migrating Great Expectations to 1.3 before upgrading. Maintenance releases from 2.0.1 to 2.0.5 arrived roughly every one to two weeks, so the cadence matters. Ask the partner how it tests upgrades in a non-production copy, how it records classification rule changes so results can be compared before and after, and whether upgrade support is included in a retainer or priced separately.

Which costs are one-time and which recur after the first scan?

One-time costs usually include discovery workshops, connector configuration, the first classification run, rule tuning and the initial mapping of PII tags to confidentiality labels. Recurring costs include hosting and storage for the catalog and its search service, scheduled scans, steward review time, staff time for new sources and upgrades. Whether support or continuous testing is part of the plan is a separate decision. Ask the partner to split the quote into one-time and recurring lines and to state which items depend on your estate growing.

Will sampling row values create privacy or security risk?

It can. Entity recognition over sample data reads real row values, so the sampling setting, the service account permissions and the storage location of sample data all need a decision before scanning starts. OpenMetadata has deprecated external sample data storage and plans its removal in 2.1, so a design built on that feature needs rework. Ask the partner to document what is sampled, how many rows, who can view results and how the catalog itself is access controlled through its security configuration.

What to ask an implementation partner before signing

Ask the partner to name every source system they will connect to the catalog and which OpenMetadata connector each one uses, then confirm which of those connectors can run the auto-classification workflow against sample data. Ask who creates the Auto Classification Agent or the external YAML workflow, who owns its credentials, and how the schedule is set for each database service. The answers should map to a written inventory of databases and warehouses, with a named owner for each. A partner who cannot list the connectors before scoping is not ready to estimate the work.

Sample data is the question most likely to create a privacy problem. External sample data storage is deprecated in OpenMetadata and planned for removal in 2.1, so ask where the partner will keep sampled values during classification, how long they persist, and who can read them. Ask whether sampling will run inside your network with no copy leaving it, and whether the partner has tested the upgrade path that removes external storage. Also ask how they will handle the OpenMetadata 2.0 breaking changes, including the move of embedding credentials into an llmConfiguration block, the Python 3.12 rebuild of ingestion images and the Great Expectations 1.3 migration, and what happens to classification history during each upgrade.

Ask how the confidence threshold, which defaults to 80, will be tested against a labeled set of columns that your data owners have confirmed as personal or not. The partner should show precision and recall results per data domain, the false positive examples they reviewed, and the change log for each threshold adjustment. Next, ask how PII tags map to confidentiality labels through tag mappings, such as Personal Data.Personal to Data Classification.Confidential, and who approves that mapping. Finally, ask who reviews flagged columns, how reviewer decisions are recorded in the catalog, and what evidence package you receive at close: configuration exports, test results, the review log and the open exceptions list.

How DeployOpen scopes and runs PII discovery

DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For PII discovery, that means agreeing in writing which sources, classifiers and review steps are in scope, then building and documenting the catalog so the organization's own stewards can run it after handover.

How we scope

Inventory the databases and warehouses in scope and confirm which connectors the catalog will need.
Agree whether column name rules, sample data recognition or both will run, and which sources may be sampled.
Define the written review path for PII tags, including who confirms, who overrides and who approves label mapping.

How we build

Deploy OpenMetadata in the client's infrastructure, configure auto-classification agents and set confidence thresholds per source.
Test classification against a labeled sample of known columns to measure false positives before tags reach production.
Configure tag mappings from PII tags to confidentiality labels, and document each rule and its reason in the runbook.

How we hand over

Deliver the configured catalog, rule set, threshold settings and mapping table with written operating procedures.
Train the named data owners and stewards on reviewing tags, handling disputes and recording decisions.
Document the upgrade path for OpenMetadata releases, including breaking changes and required migration steps before upgrading.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Re-run classification scans and review drift in tags when new sources, schemas or sampling rules are added.

Automated PII discovery and classification with an open source data catalog