Open source data catalog for EU AI Act data governance

Open source data catalog for EU AI Act data governance: the short answer

A provider or deployer of a high-risk AI system will eventually have to show how its training, validation and testing data was governed. A self-hosted catalog such as OpenMetadata or DataHub can hold much of that evidence: where each dataset came from, who owns it, how it moved through pipelines (table, column and pipeline lineage), which quality tests it passed and how it is classified. Those records map onto the topics in Article 10(2): design choices, collection processes and origin, preparation such as labeling and cleaning, assumptions, availability and suitability, bias examination and identified data gaps. Both platforms are released under Apache License 2.0, so the catalog runs in the organization's own infrastructure and the metadata stays there. The catalog does not decide whether a dataset is representative, free of errors or suitable for the setting of use under Article 10(3) and (4). Those judgments, the bias examination and the choice of mitigation measures remain with the organization and its domain experts, and the catalog only records them. The 2026 amendments change the frame. Regulation (EU) 2026/1744 deleted Article 10(5) and moved the rules on processing special categories of personal data for bias detection into Article 4a, with its own safeguards. Under the amended timetable, the high-risk rules for Annex III systems apply from 2 December 2027, so the catalog work has time to be built, but not to be left until the last quarter before that date.

Start with a scoping decision: list every AI system in scope, then the training, validation and testing datasets each one uses, and confirm which of those obligations apply to your role and system category. Next, check that the chosen platform can capture lineage from the pipelines that produce those datasets, since lineage is where the origin story is either proven or lost. For OpenMetadata, confirm the OpenLineage integration and the connectors your pipelines need against the current release notes before you commit. Assign a named owner and a steward to each dataset, and treat the catalog's own change history as part of the evidence. Ask your implementation partner to show how bias-related and special-category data would be tagged, restricted and deleted under your own GDPR and Article 4a procedures, because the catalog must reflect those controls without becoming a store of sensitive data itself. The standards picture is still moving. CEN-CENELEC reported in May 2026 that prEN 18283, on managing bias, was still in development, so any mapping you build today should be treated as a working draft to revisit when that standard and the relevant parts of Regulation (EU) 2024/1689 as amended are settled. Confirm with legal counsel how Article 42(1) presumption of compliance applies to your system before relying on it, and keep the evidence you produce independent of any single vendor or tool version.

Mapping Article 10 requirements to catalog functions and configuration

This table maps each Article 10 data governance need to the catalog function that supports it and the configuration a self-hosted OpenMetadata deployment would need. Read each row as a requirement, the catalog capability that addresses it, and the setup decisions that make the evidence usable. The main takeaway is that the catalog produces documentation and lineage evidence, but the design choices, bias analysis and data gap conclusions still have to be made and recorded by the provider or deployer.

Requirement	What the function must do	How to configure the catalog
Origin and design documentation (Art. 10(2))	Record design choices, collection process, origin, assumptions and intended use for each training, validation and testing dataset	Create a glossary for dataset terms, require descriptions and a named owner on every dataset entity, and use custom intake forms for design and assumption fields
Preparation and quality (Art. 10(2), 10(3))	Show how data was labeled, cleaned and checked for errors, completeness and statistical properties	Attach data quality tests and profiling to each dataset, schedule freshness checks, and link test results to the dataset record for review
Lineage from source to training set (Art. 10(2))	Trace each training, validation and test set back to source systems and every transformation applied	Enable table, column and pipeline lineage, ingest orchestrator metadata through OpenLineage events, and confirm which transformation jobs are not yet captured
Bias examination and data gaps (Art. 10(2), 10(4))	Hold bias review results, mitigation measures and identified gaps against the dataset they describe	Use classifications and domains to group datasets by AI system, and record bias and gap findings through custom intake forms linked to each dataset
Special category data for bias detection (Art. 4a)	Identify special category data, restrict access, record deletion once bias is corrected, and document the GDPR processing reasons	Apply sensitive classification tags, restrict access with catalog policies and SSO groups, and track deletion tasks as governed records outside the catalog
Log retention link (Art. 12(1), Art. 19(1))	Keep dataset versions and lineage history so that system logs can be matched to the data used over the system lifetime	Retain dataset version history and lineage snapshots, and set retention to at least the period the operator's logging policy requires, confirmed with counsel

Which self-hosted catalogs fit AI training data governance

Article 10 asks for documented origin, preparation and gaps across training, validation and testing data, so the catalog has to reach the systems where that data is built. The options below differ mainly in how deep their lineage and connector coverage go and whether the goal is internal governance or a published dataset. Confirm connector support for every source before choosing.

OpenMetadata for internal estates

OpenMetadata is released under the Apache License 2.0 and maintained by Collate, which also offers commercial enterprise capabilities and premium support. It lists more than 130 connectors for warehouses, BI tools, pipelines and ML platforms, and it covers table, column, dashboard, pipeline and metric lineage through OpenLineage events. Its REST APIs and Python SDK let teams attach dataset documentation from training pipelines. Check whether your feature stores, labeling tools and annotation exports have a supported connector or need a custom ingestion job.

DataHub for metadata-heavy platforms

DataHub v1.7.0.1, published September 3, 2026, is licensed under Apache-2.0 and is a common choice where a broad metadata graph across many tools matters. It fits when the team already runs a modern data stack and wants lineage and ownership to follow it. Ask the implementation partner to confirm which ingestion sources, lineage capture methods and API surfaces your training data pipelines need, and whether the team can operate its search and graph components.

Pipeline tools as lineage sources

Apache Airflow (Apache-2.0, version 3.3.2 published September 17, 2026) and Dagster (Apache-2.0, version 1.13.26 published October 9, 2026) define how training and validation datasets are built, so they are the strongest source of transformation lineage. A catalog only records what these tools emit or what a connector reads. Confirm that DAG or asset metadata, transformation code references and dataset versions reach the catalog, because lineage that stops at the warehouse will not show how a training set was prepared.

CKAN for published datasets

CKAN is licensed under the GNU Affero General Public License v3.0 and is built for data hubs and portals rather than internal lineage. The ckanext-dcat extension provides DCAT-AP v1, v2 and v3 schemas and RDF endpoints, which helps when a dataset description must be published or shared with other agencies. Use CKAN only where a dataset is released outside the team. For internal training data with personal or sensitive content, a catalog with lineage and access controls is usually the better fit, so confirm which datasets are public before deciding.

Matching the platform to the estate

Choose by where training, validation and testing data is created and stored. Teams with many warehouses, BI tools and ML platforms usually start with OpenMetadata or DataHub, while teams whose datasets are built mainly in orchestrated pipelines should check orchestrator lineage first. A mixed estate may use one catalog for internal governance and CKAN for release. Whatever the choice, confirm each source connector, the lineage depth required for Article 10(2), and who maintains custom ingestion code after go-live.

Deploying the catalog inside your own infrastructure

OpenMetadata documents production deployments on Kubernetes, including AWS EKS, Azure AKS, Google GKE and on-premises clusters. For an EU AI Act data governance program, a cluster is usually the better fit when the organization already runs container platforms with standard patching, backup and access controls, because the catalog then inherits those operating practices rather than introducing a new stack. The catalog holds metadata about training, validation and testing datasets, so the cluster location and the people who can reach it are part of the governance design.

Managed or on-premises clusters: EKS, AKS and GKE place the control plane with a cloud provider, while on-premises clusters keep every component in your own data center.
Stateful dependencies: Plan the metadata database and search service with persistent volumes, snapshots and tested restores, not only the application pods.
Network boundaries: Decide which subnets, identity groups and ingress paths can reach the catalog API and the UI.
Upgrade path: Track the OpenMetadata release notes for breaking changes before each upgrade, including ingestion image changes.
Operations ownership: Name the team that owns cluster patching, secrets rotation and log retention.

Confirm with your platform team which cluster service, region and node pools will host the catalog, and who holds administrative access to the cluster itself.

Evidence assessors and auditors ask for, and which component produces it

This table maps the evidence an assessor is likely to request for training, validation and test data governance to the catalog component that produces it and the points to confirm before relying on it. Read it as a checklist for the catalog design: each row is an artifact that needs an owner, an export path and a retention decision. DeployOpen prepares this evidence, but auditors and notified bodies assess it and decide whether it is sufficient.

Evidence item	Component that produces it	What to confirm
Dataset inventory with named owners	OpenMetadata or DataHub asset records with owner and domain fields, populated from connectors	Every training, validation and test dataset has a registered owner and the owner's role is documented
Lineage exports for training, validation and test sets	Table, column and pipeline lineage in OpenMetadata via OpenLineage events, or DataHub lineage	Lineage reaches from raw source to each model's dataset version, and gaps in pipelines are listed
Data quality test results	OpenMetadata data quality tests, profiling and freshness checks stored against each dataset	Test definitions are versioned, failures have documented remediation, and results are retained for the review period
Data origin and preparation records	Dataset descriptions, glossary terms and custom metadata capturing collection method, labeling and cleaning steps	Documentation covers the Article 10(2) topics in the organization's own words and is reviewed before each model release
Classification and glossary records	OpenMetadata classifications, glossary terms and PII tags linked to datasets	Sensitive attributes are tagged, and any Article 4a processing is recorded separately in GDPR records of processing
Access logs and change history	Catalog audit history plus database and pipeline logs, exported to the organization's log store	Logs record who changed metadata and who accessed sensitive datasets, and whether they are immutable
Retention of logs under Articles 12 and 19	Central log store with retention rules set by the provider or deployer, outside the catalog itself	Retention period meets the Article 19(1) minimum of six months unless other law requires longer, and the scope of Article 12 logging is confirmed for each system

What the catalog cannot prove on its own

A self-hosted catalog records where data came from, who owns it and how it moved. It does not decide whether a dataset is fair, representative or lawful to use for bias detection. The cards below cover the decisions and evidence that remain with the provider, deployer and their stewards, and what to check before relying on catalog records as Article 10 evidence.

Bias examination and mitigation decisions

Article 10(2)(f) and (g) require documented examination of training, validation and testing data for possible biases, plus the measures taken to mitigate them and the gaps found. A catalog can store the test results, owners and linked datasets, but the method, thresholds and the decision to accept residual risk must be written and signed by named people. Check that the catalog records which bias review applied to which dataset version, and who approved the outcome.

Representativeness and setting-of-use judgments

Articles 10(3) and 10(4) ask whether data is relevant, sufficiently representative, as far as possible free of errors and complete, and whether it reflects the geographic, contextual, behavioral or functional setting of use. Profiling statistics show completeness and distribution, not whether a population matches the deployment setting. Ask who signs off on that match and which evidence, such as sampling notes or population comparisons, is attached to each dataset.

Special category data under Article 4a

Where special category personal data is processed to detect and correct bias, Article 4a sets conditions, including pseudonymization, strict and documented access controls, no transfer to other parties and deletion once the bias is corrected. A catalog can tag these columns and show who can see lineage, but it does not enforce deletion or prove that synthetic or anonymized data would not suffice. Confirm the deletion trigger, the records of processing entry and the access review cadence before the bias work starts.

Conformity assessment is not the catalog

Most Annex III high-risk systems use internal-control conformity assessment under Annex VI, and notified bodies are involved only for some biometric systems and Annex I products under Article 43. The catalog supplies inputs to that assessment, such as dataset documentation and lineage, but it is not the assessment, a technical file or a certification. Ask the provider which parts of the technical documentation the catalog covers and who owns the conformity decision inside the organization.

Metadata accuracy depends on stewardship

Lineage and ownership are only as good as the connectors, ingestion schedules and the people who review them. Automated lineage can miss pipelines built outside the connected tools, and classifications can drift after schema changes. Set a named steward per dataset, a review cycle, and a check that lineage gaps are listed rather than hidden. Without that routine, the catalog will describe the estate as it was when it was last scanned.

Logs that support the record

Article 12(1) requires high-risk systems to technically allow automatic event recording over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Catalog lineage can point to the datasets and pipeline runs behind a model version, but it does not replace operational logs. Confirm where the logs are retained and that dataset identifiers in the catalog match the versions recorded in the logs.

How the catalog is rolled out for a high-risk AI system

The rollout sequence moves from scoping the AI systems and their data sets to a catalog that holds lineage, owners, sensitivity tags and quality evidence that compliance can walk through. The sequence is built around the 2 December 2027 date for Annex III high-risk rules, but the order matters more than any calendar date.

Scope the AI systems and their data sets
Start by listing each AI system that may fall under the high-risk rules, the training, validation and testing data sets behind it, and the teams that build and operate them. Confirm with legal counsel whether each system is in Annex III or in an Annex I product, because the applicable date differs. The output is a scoped inventory with one named business owner per system and one per data set, plus a note on which Article 10 obligations apply to each set. Treat the 2 December 2027 date for Annex III systems as a planning reference to confirm against the current text, not a deadline DeployOpen commits to.
Deploy the catalog in your infrastructure
DeployOpen installs OpenMetadata or DataHub in the customer's own cloud account, data center or air-gapped network, with the database, search service, SSO and backups sized to the scope agreed in the first phase. The team confirms the production prerequisites for the chosen version, including Java and the database and search service versions named in the vendor documentation. The check that closes this step is a working instance with access control tested against the roles the governance team has defined, and a restore test on the backup.
Connect sources and pipelines with lineage
Connect the warehouses, feature stores, labeling tools and orchestration jobs that produce each training, validation and testing set, so that a dataset can be traced back to its origin and forward to the models that consumed it. Where pipelines run on Airflow or Dagster, the lineage is captured through the connector or the OpenLineage integration, and gaps where lineage is entered by hand are recorded as such. The output is a lineage graph for each in-scope data set and a list of unresolved hops that the data owners must document.
Bootstrap ownership and the glossary
Assign a data owner and a steward to every in-scope data set, then build the business glossary and domains so that terms such as label, sample, source and population are defined once and linked to the assets they describe. Owners confirm the design choices, collection process and known assumptions in the catalog entry, which maps directly to the documentation topics in Article 10(2). The check is that every in-scope asset has an owner, a steward, a description and at least one linked glossary term.
Tag personal and special category data
Apply classifications for personal and special category data to the in-scope sets, using a mix of column rules and reviewed sample-based detection. Where special category data is processed for bias detection, record the legal basis, the safeguards and the access list in the catalog, and link them to the relevant GDPR records of processing, as the rules introduced by Article 4a require. The output is a classification report reviewed by the data protection officer, with each tag mapped to a confidentiality label and each exception documented.
Set quality tests and bias examination records
Define data quality tests for relevance, completeness, error rates and statistical properties, and attach them to the assets so that failures are visible to owners. Record the examinations for bias, the mitigation measures and the identified data gaps against the affected data set, so the evidence sits next to the lineage and ownership. The check is a test run on each in-scope set with results stored in the catalog and a named owner assigned to every failing test.
Run an evidence walkthrough with compliance
Before the catalog is treated as evidence, walk compliance, the AI system owners and internal audit through a sample data set from source to model, showing its owner, lineage, classifications, quality results and bias records. Agree which exports or screenshots the auditors will receive and where the logs described in Articles 12 and 19 will be retained. The output is a signed walkthrough record and a list of gaps with owners. DeployOpen prepares the evidence and controls, while independent assessors decide whether they meet the requirement.

Questions buyers ask before engaging a partner

Does a data catalog make our AI system compliant with the EU AI Act?

No. A catalog does not make a system compliant on its own. Article 10 sets obligations on data governance for training, validation and testing data of high-risk AI systems, and a catalog is one way to hold the evidence: where datasets came from, who owns them, how they were prepared and how they flow into models. Compliance also depends on the provider's quality management, documentation, risk management and conformity assessment, which sit outside the catalog. Confirm with your legal and compliance teams which obligations apply to your system and role, and with the independent assessor you use which evidence they will accept. DeployOpen prepares systems and evidence for assessment but does not issue compliance determinations.

What drives the cost and effort of a catalog for AI Act data governance?

OpenMetadata is released under the Apache License 2.0, so the software license is not the main cost. The effort is driven by:

Connectors: OpenMetadata lists more than 130 connectors, but each source still needs configuration, credentials and scheduling.
Lineage depth: table, column, dashboard, pipeline and metric lineage take more work than table-level lineage, especially where pipelines are custom.
Stewardship time: owners, glossary terms and dataset documentation must be written and kept current by people in the business.
Infrastructure: bare metal minimums are 2 vCPUs, 4 GB RAM and 20 GB storage for the server, with the database and search services each needing at least 2 vCPUs, 2 GB RAM and 30 GB storage, and production sizing will be larger.
Upgrades: OpenMetadata 2.0 includes breaking changes, so upgrades need planning.

Ask any partner for a fixed-scope quote that lists connectors, lineage scope, stewardship support and upgrade handling separately.

What do our data owners and legal team need to provide?

The work depends on people who know the data and the regulation. Data owners must confirm dataset origin, collection processes, preparation steps such as labeling and cleaning, known assumptions and gaps, and the setting in which the system is used. Legal and privacy teams must confirm the basis for any processing of special categories of personal data, including the conditions introduced as Article 4a by Regulation (EU) 2026/1744, and how that processing is recorded in GDPR records of processing. Security must agree access controls for sensitive datasets. Ask your partner to plan these reviews as named inputs with dates before technical configuration starts, because a catalog cannot document decisions that nobody has recorded.

Can our own team run the catalog after the engagement?

Yes, if the team has the skills and time to operate it. Running OpenMetadata needs people who can manage Java 21 services, a supported database (MySQL 8.0.42 or later, or PostgreSQL 15 or later), Elasticsearch 9.x or OpenSearch 3.x, scheduled ingestion, backups and upgrades. Ingestion images must be rebuilt for Python 3.12 under OpenMetadata 2.0. Stewardship must be owned by business teams, not only by platform engineers. Ask a partner to hand over runbooks, a connector inventory, a lineage coverage report and a documented upgrade procedure, and to state which parts of operations would remain under a support retainer, if you choose one.

How do the 2026 amendments and dates affect our planning?

Regulation (EU) 2026/1744 amended the AI Act on 8 July 2026, was published on 24 July 2026 and entered into force on 27 July 2026. As amended, the high-risk rules in Chapter III Sections 1 to 3 apply from 2 December 2027 for Annex III systems and from 2 August 2028 for Annex I product systems. Article 10(6) limits data requirements to testing data for systems that do not train models. Because the Commission's harmonised standard on managing bias, prEN 18283, was still in development as reported in May 2026, plan for requirements to be clarified during the project. Confirm dates and applicability with counsel, and build the catalog so that dataset records can be updated as guidance changes.

Does DeployOpen issue AI Act certification or audit reports?

No. DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms, and prepares systems, controls and evidence for compliance assessments. It does not issue audit reports, attestations or certifications. Conformity assessment under the AI Act is generally an internal-control process for most Annex III systems (Annex VI), and notified bodies are involved for some biometric systems and Annex I products (Article 43). Independent auditors and certification bodies make their own determinations. Ask any partner, including DeployOpen, to state in writing which deliverables are evidence packs and which decisions remain with your organization or an assessor.

How do we know the catalog keeps the AI training data trail complete?

Lineage only helps if it is captured from the pipelines that prepare training, validation and testing data. Ask how the partner captures lineage: through native OpenMetadata connectors, through OpenLineage events from orchestrators such as Airflow or Dagster, or through custom ingestion. Confirm which datasets are in scope, how model training runs link back to versioned datasets, and how gaps are reported. Article 12 and Article 19 also require automatic logging for high-risk systems, and the catalog does not replace those logs. Retention for logs is at least six months under Article 19(1), unless other law provides otherwise. Ask how logs are kept separate and how long they are retained.

How DeployOpen scopes, deploys and supports the catalog and its evidence

DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source catalogs inside the client's own cloud accounts, data centers or air-gapped networks. For EU AI Act data governance, the work centers on turning dataset documentation and lineage into evidence that a compliance team can review and an independent assessor can examine.

How we scope

Engagements are fixed-scope, with the training, validation and testing data sets in each high-risk system listed in writing before work starts.
The written scope names the connectors, pipelines and data stores whose lineage must be captured for Article 10 documentation.
Scope also records who owns each data set, which personal data tags apply and which systems are excluded from the first phase.

How we build

We deploy OpenMetadata or DataHub in the client's infrastructure and configure ingestion for the agreed sources.
Table, column and pipeline lineage is wired from orchestration tools so origin and preparation steps are traceable for each data set.
We test lineage completeness, ownership assignments and access controls against the scoped data sets before sign-off.

How we hand over

Deliverables include the configured catalog, dataset documentation templates, lineage diagrams and a runbook for stewards and operators.
Documentation maps each catalog record to the Article 10 topics it supports, so reviewers can trace evidence without reading the platform.
Handover covers administration, backup procedures and the upgrade steps the client's team must follow for future catalog releases.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainers can cover catalog upgrades, connector updates and periodic checks that ownership and lineage records remain current.
Confirm with DeployOpen which assessment evidence is in scope, since DeployOpen prepares evidence but independent auditors issue any assessment outcome.

Open source data catalog for EU AI Act data governance