Open source data catalog for GDPR records of processing: the short answer
A controller that runs customer, HR, marketing and analytics systems across several clouds and data centers must keep Article 30 records of processing and be able to show accountability under Article 5(2). The hard part is rarely the document itself. It is finding every system that holds personal data, knowing who owns it, and tracing where it flows. A self-hosted catalog such as OpenMetadata can hold that inventory: connected sources, owners, classifications and table, column and pipeline lineage, all kept inside the organization's own infrastructure. OpenMetadata is released under the Apache License 2.0, so the catalog software itself carries no license fee, and its classifications and lineage give the records a factual base to draw from.
The legal content of the records, however, stays with the controller. Article 30(1) requires the controller to record its contact details, purposes, categories of data subjects and personal data, recipients, third-country transfers and, where possible, erasure time limits and a general description of Article 32 security measures. A catalog can surface many of these fields, but someone accountable must decide the lawful purposes, confirm the data categories and approve the entries. Article 24(1) also expects the measures to be reviewed and updated where necessary, so the catalog needs named stewards and a review cycle. The GDPR requires no audit, attestation or certificate for this. Certification under Article 42 is voluntary and does not reduce the controller's responsibility.
Do not assume the Article 30(5) exemption applies. Organizations with fewer than 250 employees are exempt only if the processing is occasional and carries no risk to individuals and includes no special category or criminal offence data. Confirm this with counsel or the data protection officer before deciding to skip the records, because a catalog that already exists makes keeping them cheap and the exemption rarely covers a catalog's worth of processing. Start by checking which of your systems hold special category data, then decide the scope of the first inventory.
Which GDPR Article 30 and Article 5(2) requirements the catalog must support
This table maps each Article 30 record element and the accountability duties in Articles 5(2) and 24(1) to the OpenMetadata object that holds it and the evidence an assessor would ask for. Read it as a configuration checklist: each row names the setting to build and the gap to check. The main takeaway is that the catalog holds the inventory and its upkeep, while the legal record itself still needs a named controller owner and a review cycle.
| Requirement | OpenMetadata object to configure | Evidence to keep |
|---|---|---|
| Article 30(1)(a) controller contact details | Domain owner and team entries, plus a glossary term per controller entity | Current contact record with a named owner and last review date |
| Article 30(1)(b) purposes of processing | Glossary term per processing activity, linked to tables, dashboards and pipelines | Approved purpose description and the assets tagged to it |
| Article 30(1)(c) data subject and personal data categories | Classification tags for personal data on tables and columns, mapped to a data subject category tag | Tag history and reviewer sign-off on each classified column |
| Article 30(1)(d) recipients | Lineage edges to downstream dashboards, exports and external systems, plus custom fields for recipient type | Lineage snapshots and a list of recipients with their contractual basis |
| Article 30(1)(e) third-country transfers | Custom property for transfer location and mechanism, attached to the processing glossary term | Transfer register entries and the safeguard reference for each transfer |
| Article 30(1)(f) and (g) erasure limits and Article 32 measures | Custom fields on the processing term for retention period and general security measures | Retention schedule and a summary of security controls for each activity |
| Article 5(2) and 24(1) accountability and review | Owner assignment on every processing term, review tasks on a set cadence, and version history on glossary changes | Dated review records, owner attestations and the change log exported through the REST API or Python SDK |
| Article 30(2) processor records (separate set) | A separate domain or glossary group for processing carried out for each controller, with its own owners | Processor record set kept apart from controller records, with the controller named on each entry |
Which open-source catalogs fit records of processing
A records of processing register is an internal controller record, so the catalog you pick has to hold classifications, owners and lineage privately and keep them current. The cards below compare the two metadata catalogs that fit that job, the portal option that usually does not, and the pipeline tools that feed lineage into either one.
OpenMetadata for the internal register
OpenMetadata is the strongest fit when the register must be maintained inside the catalog. It is released under Apache-2.0 and provides classifications, glossaries, owners, domains and table, column, dashboard and pipeline lineage, with more than 130 connectors. Check that your sources have connectors, that owners can be assigned per dataset and processing activity, and that the team can run it on Kubernetes or bare metal inside your own estate.
DataHub as an alternative metadata catalog
DataHub is also Apache-2.0 and covers discovery, ownership and lineage across a data estate, with v1.7.0.1 as the version supplied for this comparison. It is a credible choice when your engineering team already runs its ingestion model. Confirm which classification, glossary and ownership workflows you need out of the box, and test how a processing activity record would be exported and reviewed by your privacy team before you commit.
CKAN for published portals
CKAN is an open-source data management system for data hubs and portals, licensed under AGPL v3.0. It publishes datasets to the public or to partners, which is a different job from keeping an internal Article 30 register. Choose it only when the goal is publication, and expect that personal data fields, processing purposes and transfer details would need to be modelled and kept out of public view.
Airflow and Dagster for lineage events
Apache Airflow and Dagster orchestrate the pipelines that move personal data between systems, so their run metadata can feed lineage into the catalog. Lineage events help you see where data flows, but they do not produce a records of processing entry on their own. Someone still has to map each pipeline to a purpose, a legal basis owner and a recipient category. Confirm which lineage integration your pipeline tool supports before you rely on it.
Fit for a controller's internal register
Article 30(1) asks each controller to record contact details, purposes, categories of data subjects and personal data, recipients, transfers and, where possible, erasure time limits and a general security description. A catalog can hold most of these as structured metadata, but the register also needs an accountable owner and a written export. Test whether your catalog can produce a complete, dated export that the supervisory authority could receive on request.
Why CKAN usually does not fit
Records of processing are internal accountability evidence, and CKAN is built to publish datasets outward. Using it as the register risks exposing internal processing details and mixing governance workflows with open data workflows. The Article 5(2) duty to demonstrate compliance is easier to meet with a system designed for internal ownership, review history and access control. Reserve CKAN for a separate publication step if you have one.
How classification, ownership and lineage feed the Article 30 register
The register is assembled from catalog evidence, but each entry still needs a named owner who signs it off. In a typical flow, OpenMetadata ingests metadata from source databases and warehouses through its connectors, runs auto-classification to tag columns that look like personal data, and places those tags in a classification such as a personal data tag set. A data steward then reviews the tagged tables and confirms or rejects each tag before the tables are linked to a processing activity. The steward's decisions are the audit trail. Confirm with your own team which connectors cover your systems of record, since connector coverage varies by source type and version.
Once a dataset is tagged and assigned an owner, the processing activity entry can be drafted from catalog objects: the owning business unit, the systems and tables in scope, the categories of data subjects and personal data, and the recipients drawn from downstream pipelines and dashboards. Lineage is the strongest input here. OpenMetadata records table, column, dashboard and pipeline lineage and accepts OpenLineage events from orchestrators, so you can show where personal data actually flows, including extracts that leave the original system. Check that your orchestrators emit lineage events for every pipeline that moves personal data, because a missing edge produces a recipient list that looks complete but is not.
Lawful basis, specific purposes and retention periods are legal decisions that the catalog cannot make. Article 30(1) asks for the purposes and, where possible, erasure time limits, and those must reflect the controller's documented reasoning. The catalog can hold these as glossary terms or custom fields on each entry, and it can show which datasets lack a purpose at all. Ask counsel or the privacy office to approve purpose wording and retention rules before they are loaded, and record who approved each change. Article 5(2) expects you to demonstrate this accountability, so the approval history matters as much as the entry itself.
How to configure the catalog for a small or large controller
Article 30(5) exempts organizations with fewer than 250 employees from the record-keeping duty, but only for processing that is occasional. Once processing is likely to result in a risk, is not occasional, or includes special category or criminal offence data, the exemption falls away. A small controller running payroll, customer onboarding or health-related services usually has to keep records for those activities, even if the headcount is low. The catalog can be kept small: one OpenMetadata instance, a handful of sources and a single owner for each processing activity. The EDPB's guide for small business says records of purely occasional activities need not be kept, so the first task is to sort activities into occasional and non-occasional before configuring anything.
- Scope the sources narrowly: Connect only the databases, SaaS exports and file stores that hold personal data, and record the rest as out of scope in the catalog notes.
- Create one domain per business function: Use domains such as HR, sales and support so each Article 30(1) entry has a clear home.
- Define classifications for personal and special category data: Set up classification tags for personal data and special category data so flagged assets feed the record directly.
- Assign one accountable owner per activity: Record a named owner for each processing activity, even when that person also holds other roles.
- Export the record in writing: Confirm that the catalog can produce a dated, written export of each record for the supervisory authority on request, as Article 30(3) and (4) require.
Confirm with your data protection lead whether any activity falls under the occasional exemption before you decide the catalog's scope, because that decision drives every later configuration choice.
A larger controller usually has many business domains, dozens of source systems and several data teams writing to the same warehouse. The catalog must hold ownership and classification for thousands of assets, so the configuration work centers on delegation and consistency. OpenMetadata documents ingestion from more than 130 connectors across warehouses, BI tools, pipelines and ML platforms, which means the connector inventory can grow quickly. Define a standard glossary and classification set once, then let each domain owner maintain their own processing entries, so that records stay consistent across the organization.
- Model domains and sub-domains: Mirror the organizational structure in OpenMetadata domains so each team sees and maintains only its own records.
- Standardize the glossary and classifications: Publish one personal data taxonomy that every domain uses, which keeps Article 30(1) categories comparable across the estate.
- Link processing records to lineage: Use table, column and pipeline lineage to show where personal data flows into recipients and third-country transfers.
- Schedule ingestion and profiling: Plan recurring ingestion so new personal data sources appear in the catalog before the next record review.
- Review access and permissions: Limit who can edit legal fields, such as purposes and retention, to the data protection office and named owners.
Identify which domains hold special category data or cross-border transfers first, because those records carry the highest review burden.
On bare metal, OpenMetadata requires Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later, and Elasticsearch 9.x or OpenSearch 3.x. The documented minimum server sizing is 2 vCPUs, 4 GB RAM and 20 GB storage, with the database and search services each needing at least 2 vCPUs, 2 GB RAM and 30 GB storage. These are minimums for a trial or light deployment, not a production sizing. A records-of-processing catalog that holds lineage, profiling results and audit history will grow beyond the minimums, so size the database and search nodes against the expected number of assets and ingestion frequency, and confirm the sizing with the vendor documentation for the release in use.
- Install Java 21 on the application host: Verify the runtime version before installing OpenMetadata.
- Provision PostgreSQL 15 or later: Keep the catalog database on a backed-up, access-controlled instance separate from source systems.
- Provision Elasticsearch 9.x or OpenSearch 3.x: Search indexes hold metadata about personal data assets, so apply the same access controls and encryption as the database.
- Plan storage above the minimums: Start from the documented 20 GB and 30 GB figures for trial sizing, then add capacity for lineage history and profiling results.
- Separate environments: Run a non-production instance for connector testing before pointing ingestion at production sources.
Ask the implementation partner to document the exact versions, sizing assumptions and backup procedure for your environment before go-live.
OpenMetadata documents production deployments on Kubernetes, including AWS EKS, Azure AKS, Google GKE and on-premises clusters, as well as Docker for local trials. Kubernetes suits larger controllers that already run a shared platform, because it supports separate environments, rolling upgrades and centralized secrets handling. It also adds operational responsibility: the cluster, the database, the search service and the ingestion workers each need their own resource settings and monitoring. For a controller with strict data residency, the cluster should run in the organization's own accounts or data centers, so the catalog's metadata never leaves the controlled estate.
- Choose the cluster location: Deploy on EKS, AKS, GKE or an on-premises cluster that sits inside the EU or in the jurisdiction your records must cover.
- Manage secrets outside the chart: Store database and connector credentials in the cluster's secret store, not in values files.
- Plan ingestion workers separately: Run ingestion on its own schedule and resource pool so profiling jobs do not starve the user interface.
- Set backup and restore for the database and search index: Confirm that restoring the catalog returns both the metadata and the search index to a consistent state.
- Pin and test upgrades: Test each OpenMetadata release in a staging cluster, because 2.0 introduced breaking changes such as an llmConfiguration block for embedding credentials.
Confirm with your platform team which cluster, storage class and backup tooling the partner will build against, and get the upgrade approach in writing.
What auditors and supervisors ask to see and which component produces it
This table maps the evidence a supervisory authority or external assessor typically requests about GDPR records of processing to the component that produces it. Read it as a checklist for designing the catalog: each row names the artifact, the component that generates it, and where gaps usually appear. DeployOpen prepares this evidence but does not issue audit reports, attestations or certifications.
| Evidence requested | Producing component | Where gaps usually appear |
|---|---|---|
| Written Article 30(1) register with version history | Register export from OpenMetadata, with the controller's own sign-off recorded outside the catalog | Register exported without a dated version, or edits made in spreadsheets that the catalog does not reflect |
| Named owner for each processing activity and dataset | OpenMetadata ownership assignments and steward workflow | Owners who have left the organization, or datasets with a team name instead of an accountable person |
| Classification results for personal data categories | OpenMetadata classifications and auto-classification results, with reviewer decisions | Automated tags accepted without review, or confidence thresholds never documented |
| Review decisions on classification and tagging | Steward workflow tasks and their approval history in the catalog | Decisions made in chat or email, so the catalog holds no record of who approved a tag |
| Lineage graphs showing where personal data flows and is shared | OpenMetadata table, column and pipeline lineage, with OpenLineage events where pipelines emit them | Lineage stopping at manual extracts, file shares or SaaS tools that no connector covers |
| Catalog change logs and access records | Audit trail of the catalog application and its database, retained according to the organization's logging policy | Logs rotated before the review period, or admin accounts shared among staff |
| Article 24(1) review evidence showing measures are reviewed and updated | Dated review records in the policy document set, linked to catalog review tasks | Reviews that happen but leave no dated artifact, so the organization cannot show the measures were kept current |
| Transfer and recipient entries tied to specific datasets | Register fields populated from catalog ownership and glossary terms, cross-checked with the organization's transfer assessments | Recipients listed in the register but absent from the lineage, or transfer assessments stored in a separate system |
What a data catalog alone does not satisfy
A catalog shows where personal data sits, who owns it and how it moves, which is the raw material for Article 30 records. It does not make the legal decisions behind those records or produce the other documents the GDPR requires. The cards below cover what the catalog cannot settle and what has to sit alongside it.
Lawful basis and purpose decisions
A catalog can store a lawful basis and a stated purpose against a dataset, but it cannot decide either. Those decisions belong to the controller, usually with legal and privacy input, and they must be reviewed when processing changes. Ask who approves each purpose entry, how changes are recorded, and whether a dataset without an approved purpose is flagged in the catalog. Illustrative: a marketing table tagged with a purpose that no one has reviewed still looks complete in the catalog.
Retention decisions
Article 30(1) asks for erasure time limits where possible, but the catalog only records the limit someone enters. It does not know whether a retention rule is legally justified or whether deletion jobs actually run. Check whether retention is owned by a named steward, whether it is linked to a deletion process outside the catalog, and whether the catalog is tested against a sample of real deletions.
Processor records and contracts
Article 30(2) requires processors to keep their own records, and the catalog can only describe processing that a connector or steward has documented. A processor's records, sub-processor list and contract terms are held by the processor and in contract files. Confirm which processors appear in the catalog as owners or recipients, and where the signed terms and processor records are stored and reviewed.
DPIAs under Article 35
A data protection impact assessment is a separate analysis of risk, necessity and mitigations for high-risk processing. The catalog can flag which datasets feed a high-risk use and link to the assessment, but it cannot produce the assessment. Ask how the catalog is linked to the DPIA register, who decides when a DPIA is required, and how changes in lineage trigger a review.
Breach documentation and transfer assessments
Article 33(5) requires the controller to document personal data breaches, and Article 49(6) requires documentation of the assessment and safeguards for certain transfers. Neither lives in a metadata catalog. The catalog helps identify which systems hold affected data and where it was transferred, but the incident log and transfer assessment files must be kept in a separate, controlled repository. Confirm where each record type is kept and who can access it.
Steward time and metadata quality
A catalog is only as accurate as the owners and stewards who maintain it. Connectors can populate tables and columns automatically, but purposes, owners, recipients and retention values usually need human confirmation. Budget steward hours as a recurring cost, assign a named owner for each system, and sample entries regularly to test whether the Article 30 view still matches how the systems actually run.
How much effort and cost does a GDPR records catalog take?
What does it cost to run a GDPR records catalog on open source?
The software license is not the main cost, because OpenMetadata is released under the Apache License 2.0. Cost comes from four components: infrastructure, connector and classification setup, steward time, and ongoing operations. Infrastructure is paid to your cloud provider or data center. Setup and steward time are paid to your staff, to a partner, or both. Ask any partner to split its quote into these four lines so you can see which part is one-time and which recurs.
Which parts of a GDPR records catalog are one-time and which recur?
One-time work includes deploying the platform, connecting source systems, configuring classification rules, building the glossary and assigning initial owners for each processing activity. Recurring work includes running ingestion schedules, upgrading OpenMetadata, reviewing new datasets, updating records when systems change, and keeping ownership current when people leave. The recurring share is often larger than the first build, because the record must stay accurate as the estate changes. Confirm how the quote prices each recurring item.
What drives the effort of a GDPR records catalog up or down?
Effort rises with the number of source systems, the number of connectors that must be configured, how much personal data sits in free-text or unstructured fields, and how many processing activities need a documented owner. Effort also rises when lineage must reach pipelines and reports rather than stopping at tables. Effort falls when the processing inventory already exists in a spreadsheet or register that can seed the glossary, and when a single accountable owner signs off each activity. Ask the partner which of these factors it assumed.
What should I ask for in a quote for a GDPR records catalog?
Ask for a written scope that lists the source systems in each connector wave, the lineage depth expected for each, and the classification rules to be tuned. Ask who performs steward workshops, how many are included, and who signs off the completed records. Request a separate line for upgrades and for operations after go-live. Confirm what is excluded, such as legal review of the records themselves, which remains the controller's responsibility. Ask for the deliverables in writing, including the exported record format.
Can we build a GDPR records catalog with our own team?
Yes, if your team has the skills to deploy and operate OpenMetadata, configure connectors, and maintain the database and search services it depends on. The platform documents production deployments on Kubernetes and on bare metal, with prerequisites including Java 21, a supported database, and Elasticsearch 9.x or OpenSearch 3.x. The harder part is usually stewardship, because the business owners of each processing activity must answer questions and review tags. Assess both the platform skills and the time your data owners can commit before you decide.
How long does it take to set up a GDPR records catalog?
No duration has been established for this scope, so confirm timelines with each candidate partner against your own environment. The timeline depends on the number of source systems, the state of existing processing inventories, and how quickly data owners respond during steward review. Ask for milestones tied to connector waves and to sign-off of each batch of processing activities, rather than a single end date. Most engagements are fixed-scope, so the agreed scope should state the timeline in writing before work starts.
Do we need Article 42 certification for our GDPR records?
No. Certification under Article 42 is voluntary, and the GDPR requires no audit, attestation or certificate for records of processing. Article 24(3) allows approved codes of conduct or certification to be used as an element to demonstrate compliance, but certification does not reduce the controller's responsibility. The records must still be in writing, including electronic form, and available to the supervisory authority on request. DeployOpen prepares systems and evidence for assessments but does not issue certifications or audit reports.
Does a GDPR records catalog remove the need for a legal review?
No. The catalog helps you find personal data, document systems and owners, and keep the Article 30 fields current, but it does not decide lawful bases, retention periods or transfer safeguards. Those decisions remain with the controller and its legal and data protection function. Include time for legal review in your internal plan, and confirm with any partner that its scope covers the technical records only. The Article 30(5) exemption for organizations with fewer than 250 employees should also be checked with counsel before you rely on it.
How a records of processing rollout runs in phases
The sequence moves from a scoped inventory of processing activities to a self-hosted OpenMetadata catalog that holds the Article 30(1) fields, reviewed by named stewards and kept current as the platform and processing change. Each phase closes with a check the controller can show a supervisor or auditor.
Scope the processing inventory
Agree with legal, privacy and system owners which business processes, systems and processors fall within the records, and which organization-level scope applies under Article 30(5). Record the scope decision in writing, including any exclusions and the reason for each. The phase is complete when the controller's privacy lead signs off a list of in-scope processing activities and the systems that hold them.
Deploy OpenMetadata in your infrastructure
Your platform or infrastructure team installs OpenMetadata in your own cloud account, data center or air-gapped network, using the Kubernetes or bare metal pattern your standards allow, along with its database and search services. DeployOpen can carry out the install under a written scope. The phase is complete when the instance is reachable through your single sign-on, its backups are tested, and access is limited to named roles.
Connect sources and run classification
Connect the warehouses, operational databases, BI tools and pipelines that hold the in-scope data, then run metadata ingestion and automated classification to tag columns that appear to contain personal data. Review the matches against sample data, because name-based rules and entity recognition produce different false positives. The phase is complete when every in-scope system is ingested and each classification result has been accepted, corrected or rejected by a reviewer.
Assign owners and glossary terms
Assign an accountable owner to each dataset and system, and create glossary terms for the processing purposes, data subject categories and personal data categories your organization uses. Owners confirm which purpose each dataset serves and which legal basis applies, using your own privacy taxonomy. The phase is complete when every in-scope asset has an owner and every processing activity links to an approved purpose term.
Map fields to Article 30(1) elements
Map each processing activity to the Article 30(1) elements: controller contact details, purposes, categories of data subjects and personal data, recipients, third-country transfers, erasure time limits where possible, and a general description of Article 32 security measures. Store these as custom properties, glossary entries or linked documents on the activity, and use lineage to link each activity to the systems that process the data. The phase is complete when each activity has every field filled in, or a recorded reason why a field is not yet possible to complete.
Steward review and export
Independent stewards, who did not create the entries, check each record for accuracy against the source systems and the lineage graph. The privacy function approves the set, and the approved records are exported in a written, electronic format for the supervisory authority on request. The phase is complete when the approval history is retained and a dated export can be reproduced from the catalog.
Keep records current under Article 24(1)
Schedule periodic owner attestations, lineage and classification re-runs, and reviews triggered by new systems, new processors or changes in purpose. Track OpenMetadata upgrades separately, reading the release notes for breaking changes before each upgrade, and test the records after each one. The phase is complete when the controller can show that measures are reviewed and updated where necessary, with a dated change history for each record.
How DeployOpen scopes and supports the catalog and its evidence
DeployOpen plans, deploys, integrates and documents a self-hosted OpenMetadata instance inside your own cloud accounts, data centers or air-gapped network. Work is agreed under a fixed-scope written agreement that names the connectors to be configured, the systems in the register, the classification rules, the roles that will steward each record, and the deliverables and timelines. Before work starts, ask for that scope to list which source systems feed Article 30 fields (purposes, data subject categories, recipients, transfers and erasure periods) and which fields will be completed by people rather than by the catalog. A scope that leaves those decisions open will leave gaps in the register later.
DeployOpen prepares the evidence that an assessment would ask for: the configured catalog, the classification and ownership model, lineage captures, access controls, backup and upgrade records and the documented procedure for keeping the register current. Ongoing work such as OpenMetadata upgrades, connector maintenance and periodic review of ownership is available as a retainer, which matters because the catalog changes with each release and with each new system. Confirm with DeployOpen how upgrade testing, rollback and change records are handled, and confirm which OpenMetadata version your team will be supported on at go-live.
DeployOpen does not issue audit reports, attestations or certifications. The GDPR requires none of these, and certification under Article 42 is voluntary and issued by accredited bodies, so a supplier's work product cannot stand in for your own accountability under Article 5(2). DeployOpen also does not give legal advice on what the register must say for your organization. Your data protection officer, legal counsel or supervisory authority contact should confirm the purposes, legal bases and retention periods, and your team owns the register content and its sign-off.
