Data catalog for the public sector

Data catalog for the public sector: why agencies need two different tools

Agencies usually need two catalogs, not one. The first is an internal governance catalog, where staff in ministries, departments or municipalities find datasets, trace lineage, assign owners and manage classifications. OpenMetadata is the open-source option most often considered for this job. The second is a published data portal, where the public and other agencies find datasets that the organization is obliged or willing to release. CKAN is the usual fit here. Many agencies run both, because the internal catalog holds drafts, restricted datasets and operational lineage that should never appear on a public site, while the portal publishes only what has been approved for release. The internal catalog deserves more care than its name suggests. Once it connects to HR, benefits, tax, licensing or case-management databases, it holds metadata about citizens' data: where personal fields sit, who owns them, which systems feed which reports and who has asked for access. That metadata can reveal sensitive structures even when the records themselves are not copied in. Hosting location, network segmentation, SSO integration and role-based access to the catalog are therefore design decisions, not afterthoughts. Confirm with your data protection officer and security team whether catalog metadata falls under the same classification as the source data, and set the access model before the first connector is turned on.

DeployOpen plans, implements, hardens and documents self-hosted platforms such as OpenMetadata and CKAN inside the agency's own cloud accounts, data centers or air-gapped networks. That work produces configuration baselines, access designs and operating runbooks that an auditor can review, but DeployOpen does not issue audit reports, attestations or certifications. Ask any implementation partner to show how they would separate the internal and public environments, how they would document where catalog metadata is stored and who can administer it, and which upgrade path they would follow for each product. Those answers tell you more about readiness than the feature list does.

Which public sector rules shape the catalog build

This table maps each public sector rule to what it means for hosting, encryption, access control and portal accessibility in a catalog build. Read each row as a question to put to legal counsel and the agency's security team before choosing a hosting model. Applicability depends on the agency's mandate, the data it holds and the jurisdiction, so counsel confirms each row before design starts.

Rule or control set	What it means for the catalog	Evidence an assessor will ask for	Where it usually fails
NIST SP 800-53 Rev. 5 baselines (low 149, moderate 287, high 370 controls)	The baseline chosen for the catalog sets the control count and the hosting and logging depth. Agencies typically map the catalog into an existing system security plan.	System security plan entries mapped to the selected baseline, with the OSCAL profile used and inherited controls named	Catalog is treated as a low-impact tool while its metadata reveals sensitive system and dataset names
FISMA and Inspector General assessment	Federal agencies must assess system security, and Inspectors General review that work independently and report to OMB. The catalog becomes part of that reviewed inventory.	Current authorization status, control assessment results and plan of action records for the catalog	Catalog is left out of the agency system inventory, so it is never assessed
FedRAMP scope for cloud-hosted catalogs	A cloud-hosted catalog that stores federal information for an agency, or is offered as a shared service, may fall under FedRAMP. A single-agency system on cloud infrastructure may fall outside scope. Confirm which case applies.	Written scope determination and, where in scope, the authorization package of the cloud service provider	Team assumes a commercial cloud region is compliant without checking authorization level or boundary
FBI CJIS Security Policy v6.1 (effective 26 June 2026)	If the catalog holds criminal justice information, CJI in transit and at rest outside physically secure locations needs FIPS 140-3 validated modules or AES with 256-bit keys. Vulnerability scans run at least monthly.	Encryption configuration showing validated modules, key lengths and monthly scan reports	Metadata records and sample values are not treated as CJI, so search indexes, caches and backups stay unencrypted
32 CFR 2002.14(h)(2) and NIST SP 800-171 for CUI	Federal agencies use NIST SP 800-171 to set safeguards for CUI on non-federal systems, unless the CUI category sets other rules. A catalog that describes CUI datasets may need those safeguards.	Mapping of catalog components to SP 800-171 requirements and a list of CUI categories recorded in the catalog	Dataset descriptions, column names and sample data reveal CUI but are kept outside the protected boundary
WCAG 2.1 Level A and AA under the ADA Title II rule (28 CFR 35.200)	Public-facing portal pages and mobile apps from state and local entities must meet WCAG 2.1 AA from 26 April 2027 for entities serving 50,000 or more people, and from 26 April 2028 for smaller entities and special districts. This applies to portals built on CKAN or similar software.	Accessibility conformance report from an independent test, with remediation tracked per page template	Accessibility is tested only on the home page, while dataset pages, filters and embedded charts are never checked
NIS2 applicability to public administration	NIS2 covers central government entities regardless of size and lists public administration as a high-criticality sector. Member States may extend it to local administration. It excludes national security, public security, defense and law enforcement. Confirm the status of the agency and whether the catalog supports covered services.	Entity scoping memo from legal counsel and, where covered, the risk management and incident reporting records for the systems that depend on the catalog	Agency assumes NIS2 does not apply because it is local, without checking Member State extension or the exclusions

When OpenMetadata fits internal governance and when CKAN fits the open data portal

The first decision for a public agency is whether the goal is internal governance or a published data portal, because the two are different products with different users. OpenMetadata is built for discovery, lineage, ownership, glossary and classification across the agency's own data estate. It lists more than 130 connectors for warehouses, BI tools, pipelines and ML platforms, captures table, column, dashboard, pipeline and metric lineage, and exposes REST APIs and a Python SDK so that stewards and engineers can automate metadata. Its audience is the agency's data owners, analysts and auditors. CKAN, by contrast, is a data management system for publishing datasets to the public. Its DataStore holds structured resource data with a read-only user and a Data API for filtering and SQL queries, and the ckanext-dcat extension supplies DCAT-AP v1, v2 and v3 schemas and RDF endpoints. Its audience is citizens, researchers and other publishers. Many agencies will need both, and the architecture should say which system is the system of record for each dataset.

The licenses differ in a way that matters for procurement. OpenMetadata is released under the Apache License 2.0, which permits internal use and modification without a copyleft trigger. CKAN is licensed under the GNU Affero General Public License v3.0. If the agency runs a modified CKAN and offers it to the public over a network, the AGPL obligations to make the corresponding source of those modifications available apply, and the agency's legal team should review what that means for custom extensions, themes and any contractor code. Confirm this interpretation with counsel before the design is fixed, and record the decision in the architecture file so the obligation is not rediscovered at the next upgrade.

Choosing CKAN also commits the agency to a release cadence. The CKAN project targets about two minor releases a year, supports the latest patch of the current minor and the previous minor, and drops older patches once a newer one ships. Minor releases may include backwards-incompatible changes, so an upgrade plan with a test environment is part of the scope. For OpenMetadata, confirm the upgrade path before go-live, because the 2.0 release notes list breaking changes such as the move of embedding credentials into an llmConfiguration block and the rebuild of custom ingestion images for Python 3.12. Ask any implementation partner to show how each platform's upgrades were handled in comparable environments.

Which catalog option fits each agency situation

When the goal is governance across the agency's own systems, such as finding owners, tracing lineage, classifying personal data and maintaining a glossary, OpenMetadata is the usual starting point in this set of options. It covers table, column, dashboard and pipeline lineage, glossaries, classifications and data quality tests, and it provides REST APIs and SDKs for automation. The main work is connecting sources and curating metadata, not publishing datasets to the public.

Connector coverage: OpenMetadata lists more than 130 connectors for warehouses, BI tools, pipelines and ML platforms, so confirm each of the agency's source systems is on the list or can be reached through a custom ingestion route.
Bare metal prerequisites: Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later, and Elasticsearch 9.x or OpenSearch 3.x.
Sizing floor: The documented minimum server is 2 vCPUs, 4 GB RAM and 20 GB storage, with the database and search services each needing at least 2 vCPUs, 2 GB RAM and 30 GB storage. Production sizing will be higher.
Deployment options: Kubernetes on managed or on-premises clusters, or bare metal without Kubernetes.
Access model: Confirm how SSO, roles and domain ownership map to the agency's identity provider before loading data.

Ask the implementation partner which connectors need custom work and who will own the search and database services after go-live.

Platforms that fit public sector catalogs and why

Public sector catalogs usually split into two jobs: an internal catalog that governs data across agencies and their systems, and a published portal that exposes datasets to the public or to partner bodies. The cards below cover the open-source platforms that fit each job, along with the licensing, release and identity checks to run before choosing one.

OpenMetadata for internal governance

OpenMetadata fits an internal catalog that needs lineage, ownership, glossaries and classifications across warehouses, BI tools and pipelines. It is released under Apache-2.0 and maintained by Collate, which also offers commercial enterprise capabilities and premium support, so the agency should confirm which features its deployment needs are in the open-source edition. The current release line is 2.0.5, and the 2.0 release changed breaking items such as embedding credentials and Python 3.12 ingestion images, so confirm the upgrade path before go-live.

DataHub for large connector estates

DataHub is an Apache-2.0 metadata platform that suits agencies with many source systems and a need for broad metadata ingestion and search. Its v1.7.0.1 release is the reference point for this assessment. The team should compare its ingestion model, search backend and access controls against OpenMetadata using the same set of agency sources, and confirm the operational footprint each option requires.

CKAN for a published data portal

CKAN is an open-source data management system for data hubs and portals, licensed under AGPL v3.0. Public bodies such as the US catalog.data.gov and the Canadian open.canada.ca/data run on it, which makes it a practical choice when the goal is publishing datasets rather than governing internal estates. Confirm the AGPL obligations with legal counsel before modifying the code, and plan for the two minor releases a year and the patch support window.

ckanext-dcat for DCAT-AP metadata

The ckanext-dcat extension provides DCAT-AP v1, v2 and v3 schemas, RDF endpoints and an RDF harvester for importing from other catalogs. It is the component that makes CKAN portal records exchangeable with national and European catalogs. Confirm which DCAT-AP version your national or sector profile requires, and test harvesting against the target catalog, since the extension is supported on CKAN 2.10 and 2.11 and your CKAN version must align with it.

ckanext-saml2auth for federated login

ckanext-saml2auth is a community extension that adds SAML 2.0 login to CKAN, which may suit agencies that authenticate staff through a federated identity provider. Treat it as an option to validate, not a default: confirm that its maintenance status, the SAML profile it implements and its handling of attribute mapping and role assignment meet your identity provider's requirements. Test session logout and deprovisioning before rollout.

Hosting and sovereignty decisions

Whichever platform wins, the agency decides where the catalog, its search index, its database and its harvested metadata live. Confirm that the hosting environment, administrator access and backup locations match the data residency and sovereignty rules that apply to the agency. Ask the implementation partner to document the hosting boundary, the operator roles and the audit logs they produce, so the agency can evidence its controls without relying on the vendor's assurances.

How a public sector catalog rollout moves from scope to operations

The sequence turns a set of agency datasets into a governed catalog and public portal that can be tested, hardened and operated under a defined patch policy. Each phase closes with a written output that the next phase depends on.

Map datasets, owners and applicable laws
The team inventories candidate datasets and sorts each one as published, shared under agreement, or internal only. Legal, records and data owners confirm which statutes, open data rules and sector obligations apply to each class, including whether any dataset falls under accessibility or cloud authorization requirements. Your own counsel and compliance staff make these determinations; DeployOpen documents them. The output is a scope register listing each dataset, its owner, its classification and the governing rules, signed off by the business owner before any build work starts.
Select hosting and the security baseline
Using the scope register, the team chooses where the platform runs and which control baseline the environment must meet. Federal and defense-linked programs point to NIST SP 800-53B baselines (low, moderate or high), while criminal justice data points to the FBI CJIS Security Policy. Confirm with your authorizing official or assessor which baseline and cloud authorization level apply, and whether sovereign hosting in your own cloud account or data center is required. The output is a documented hosting decision and a control baseline mapped to the platform components, with gaps logged for remediation.
Agree the metadata model and DCAT-AP profile
Publication staff and the catalog team agree which metadata fields each dataset must carry, which DCAT-AP version the portal will expose, and which controlled vocabularies and licenses apply. CKAN's ckanext-dcat extension supports DCAT-AP v1, v2 and v3 schemas, so the version choice affects extension configuration and downstream harvesters. Confirm the target version with the reuse or interoperability body your agency reports to. The output is a versioned metadata specification with a field-by-field mapping and sample records reviewed by the publication team.
Build pilot connectors and run the first harvest
Engineers configure a small set of source connectors and either harvest records from existing catalogs through the RDF harvester or import DCAT feeds into CKAN. For internal governance, OpenMetadata connectors pull schemas, owners and lineage from warehouses and pipelines. Pilot datasets are chosen to cover each classification class from the scope register. The output is a pilot catalog in a non-production environment where every harvested record is checked against the agreed specification, with rejected records and their causes logged.
Model sharing agreements and access roles
For data shared between agencies, the team records each agreement as metadata: parties, purpose, permitted use, retention, and the datasets it covers. Access roles are defined for publishers, stewards, reviewers and consumers, and single sign-on is configured where the agency requires it. Plugins such as ckanext-saml2auth exist for SAML-based sign-in, but its fit must be verified against your identity provider. The output is an approved agreement register linked to dataset records and a tested role matrix showing who can publish, edit, view or download each class.
Test accessibility of the portal
Before launch, the portal and its custom templates are tested against WCAG 2.1 Level A and AA, the level the ADA Title II rule sets for state and local government web content. The rule's compliance dates depend on the entity's population size, so confirm your applicable date with counsel. Testing combines automated scans with manual review by people using assistive technology, and findings are tracked to closure. The output is an accessibility test report with a remediation log and a sign-off from the portal owner.
Harden production and write runbooks
The production environment is built to the chosen baseline, with the CKAN DataStore's separate write and read-only database users, encrypted connections, and monitoring for Solr, Redis and PostgreSQL. Engineers write runbooks for backup and restore, incident handling, user offboarding and dataset takedown. Restore procedures are rehearsed, not just documented. The output is a hardened production environment, a restore test record and a runbook set approved by the security owner.
Operate with a defined patch policy
In operations, the team follows CKAN's release policy: it supports the latest patch of the current minor version and the previous minor version, and patch releases do not change the schema or add features, while minor releases may contain breaking changes. The team applies patches on a schedule agreed with the agency, tests minor upgrades in staging first, and monitors the versions of OpenMetadata, Solr and PostgreSQL the same way. The output is a patch calendar, a tested upgrade record for each release and an ongoing evidence folder for compliance reviews.

Integration points and deployment patterns for sovereign hosting

A CKAN or OpenMetadata deployment for a public body touches identity, databases, search, file storage, inter-agency harvesting and retention controls. Each of these is a decision the agency makes before the catalog goes live, and each one shapes what evidence it can show later.

Identity through SSO or SAML

Staff editors, stewards and read-only reviewers should sign in through the agency's existing identity provider, not local catalog accounts. CKAN has a community SAML2 extension (ckanext-saml2auth), so confirm its maintenance status and supported CKAN version before relying on it. For OpenMetadata, confirm the supported SSO providers and how group claims map to catalog roles. Check how deprovisioning removes access to publish and edit rights.

Database and search tiers

CKAN needs PostgreSQL 12 or newer and Solr 9 (recommended) or Solr 8, with Redis for caching. OpenMetadata needs Java 21, MySQL 8.0.42 or later or PostgreSQL 15 or later, and Elasticsearch 9.x or OpenSearch 3.x. Agencies should size these tiers for search and backup separately from the application, place them on hardened hosts, and confirm the versions their security baseline allows.

Object storage and DataStore separation

Resource uploads and the DataStore are separate stores. Uploads belong in object storage the agency controls, and the DataStore lives in its own PostgreSQL database with a write user and a read-only user. Decide where each sits, who can reach the read-only role, and whether storage encryption keys are held inside the agency boundary, since these choices determine whether citizen data stays in sovereign infrastructure.

XLoader for file loading

XLoader is the officially supported tool for loading uploaded or linked files into the DataStore, and DataPusher is legacy. Agencies should run XLoader as a controlled worker with restricted network egress, so that linked files from external URLs are fetched only from approved domains. Test it with the file types and sizes the publishing team actually uses, and confirm the version against the CKAN release in use.

RDF harvesters for inter-agency catalogs

ckanext-dcat provides DCAT-AP v1, v2 and v3 schemas, DCAT RDF endpoints and an RDF harvester for importing from other catalogs. Inter-agency sharing depends on which DCAT-AP version the partner agencies publish. Confirm the version on both sides, the harvest schedule, and whether harvested records carry their source agency so that ownership and provenance survive the import.

Backup and logging tied to retention rules

Backups, search indexes and audit logs each have their own retention needs. Agency record schedules and any applicable law set how long catalog metadata and access logs must be kept, and the backup policy must match them without keeping restored copies of withdrawn data. Confirm restore tests, log destinations inside the agency boundary, and who approves deletion. Do not assume a default retention period.

Evidence auditors and oversight bodies ask a public catalog to show

This table lists the evidence a public sector catalog typically needs to produce, which component generates it, and which agency role should sign off. DeployOpen prepares and organizes this evidence, but independent assessors and auditors form the opinion. Use it to agree evidence ownership before the build starts, not after an assessment request arrives.

Evidence item	Produced by	Agency sign-off
System inventory and owners	OpenMetadata ownership, domains and service entries	Data owner per domain and CIO or CISO office
Data classification and labels	OpenMetadata classifications and glossary terms, reviewed against the agency's classification policy	Data governance lead and privacy officer
Lineage for published figures	OpenMetadata table and column lineage, with OpenLineage events from pipelines	Statistics or open data owner for each published dataset
DCAT-AP metadata completeness	CKAN dataset records exported through ckanext-dcat, checked against the DCAT-AP profile in use	Open data coordinator and records manager
Access control records	SSO sign-in logs, CKAN and OpenMetadata role assignments and periodic access reviews	Identity owner and business owner of each role
Vulnerability scanning results	Scheduled scans of the catalog, database and search hosts, with tracked remediation tickets	Security operations lead and system owner
Patch and upgrade records	Change tickets, release notes reviewed for breaking changes, and staging test results	Change advisory board and service owner
Backup restore tests	Documented restore of the metadata database, search index and CKAN DataStore, with timings recorded	IT operations manager and records officer

What agencies ask before scoping a public sector catalog

What drives the cost of a public sector data catalog?

The main cost drivers are the number of source connectors, the number of datasets that need metadata, the profile work for DCAT-AP, the accessibility testing of any public-facing portal, and the hosting model. The software licenses for OpenMetadata (Apache License 2.0) and CKAN (GNU Affero General Public License v3.0) carry no license fee, so the spend is in setup, curation, stewardship time, infrastructure and operations. Setup and connector work is largely one-time, while hosting, patching and stewardship recur. Ask any partner for a quote that separates these components, lists the connectors and datasets in scope, and states who is responsible for legal review and data owner sign-off.

Can a small agency team run a data catalog internally?

A small team can run a catalog internally if it has an engineer comfortable with the platform's stack, a named owner for metadata quality and a clear upgrade routine. OpenMetadata's bare metal prerequisites include Java 21, a PostgreSQL 15 or later or MySQL 8.0.42 or later database, and Elasticsearch 9.x or OpenSearch 3.x, and CKAN needs Python, PostgreSQL, Solr and Redis. Each component is another service to patch, back up and monitor. Agencies often start with an internal owner for the platform and bring in an implementation partner for the initial build, integrations and runbooks, then decide what stays in house. Confirm your staffing against the operations scope before committing.

How long does a public sector catalog rollout take?

The rollout length depends on scope more than on the software. The factors that extend it are the number of source systems, whether lineage must reach column level, how many datasets need DCAT-AP metadata reviewed by a person, whether a public portal must meet WCAG 2.1 Level A and AA, and how long legal review and data sharing agreements take. No duration is fixed by the platform itself. Ask a partner for a timeline built from a written scope with named milestones, and confirm which approvals from your own agency sit on the critical path. Fixed-scope engagements agree deliverables and timelines in writing before work starts.

What must the agency provide before a catalog project starts?

The agency needs to provide named data owners for each dataset, a legal contact who can review publication rights and sharing terms, and access to the identity provider for single sign-on. It also needs access to source systems and their credentials, a decision on the hosting location, and staff time for glossary and classification review. For a public portal, the agency should confirm its accessibility obligations under the ADA Title II rule, which requires WCAG 2.1 Level A and AA for state and local government web content from 26 April 2027 for entities serving 50,000 or more people. For CKAN, ask whether the ckanext-saml2auth extension fits your identity setup. Confirm each dependency in the scope document.

Is CKAN or OpenMetadata required for open data publication?

Neither is strictly required, but they serve different jobs. CKAN is a data management system built to power public data hubs and portals, and it is used by sites such as catalog.data.gov. OpenMetadata is a metadata catalog for internal discovery, lineage, ownership and governance. Many agencies use OpenMetadata internally and CKAN for publication, and the ckanext-dcat extension supports DCAT-AP v1, v2 and v3 schemas with RDF endpoints for publishing. Choose CKAN when the goal is a published portal, OpenMetadata when the goal is internal governance, and confirm with your data governance lead whether a single platform is enough for both.

How are OpenMetadata upgrades handled given the 2.0 breaking changes?

Upgrades need a planned, tested path because OpenMetadata 2.0 includes breaking changes. The release notes list moving embedding credentials into an llmConfiguration block, a new Databricks auth format, rebuilding custom ingestion images for Python 3.12, and migrating Great Expectations to 1.3 before upgrading. A sound upgrade process includes a staging environment, a review of each connector configuration, a backup of the database and search index, and a rollback plan. Maintenance releases for 2.0 have followed about every one to two weeks, so agree in the scope how often patches are applied and who signs off. Ask a partner how they test upgrades and document changes.

Questions to ask a partner before a public sector catalog contract

Start with the hosting question, because it decides everything after it. Ask whether the partner has deployed and operated open-source catalog platforms inside government networks, and which network types: a government cloud tenancy, an agency data center, or an isolated environment with no outbound internet access. Ask who holds administrator credentials, where backups are stored, and how a DCAT-AP profile change or a CKAN upgrade is tested inside that boundary. Then ask how they map NIST SP 800-53 baselines to the catalog. A credible answer names the baseline the agency has selected (the control counts for low, moderate and high are set out in NIST SP 800-53B), shows which controls are inherited from the hosting platform, and identifies which are configured in OpenMetadata or CKAN, such as access control, audit logging and authentication. Ask for the mapping as an editable spreadsheet or OSCAL artifact, not a slide.

Next, test how the partner handles metadata standards and accessibility. For a published portal, ask how they will set up DCAT-AP profiles through the ckanext-dcat extension, which supports DCAT-AP v1, v2 and v3 and the RDF harvester, and who signs off the profile version the agency commits to. Ask how sharing agreement metadata is modeled, such as which fields record the parties, purpose, legal basis, retention and expiry, and whether those fields are exposed for internal lineage or only for publication. For accessibility, ask how they test WCAG 2.1 AA on the portal, including the automated tools, the manual keyboard and screen reader checks, and the theme changes they will make, and whether the agency receives a written test report. Ask how AGPL obligations for CKAN are documented, since CKAN is licensed under the GNU Affero General Public License v3.0. The partner should explain what source code changes must be made available when the portal is run as a network service, and how custom extensions are tracked.

Finally, ask for a written fixed scope and what is excluded. Confirm in writing which systems are in scope, which connectors or harvest sources are configured, how many datasets are migrated, which training is included, and who owns each upgrade. Ask how OpenMetadata and CKAN upgrades are scheduled: OpenMetadata publishes frequent maintenance releases and has documented breaking changes in its 2.0 release, and CKAN supports its latest patch and the previous minor version. Ask how each upgrade is tested, how long the agency is protected after a release stops receiving fixes, and whether ongoing support is a separate retainer. Confirm what is excluded, such as audit preparation, legal review of agreements, or certification work, which independent auditors and certification bodies perform. A partner who cannot state the exclusions in writing is not yet ready for a contract.

Data catalog for the public sector