Why an open source data lakehouse fits public sector analytics
An agency or shared service team that must analyze data across departments, keep records readable for decades and avoid lock-in to one warehouse vendor has a specific problem. A self-hosted lakehouse on Apache Iceberg, queried with Trino or Spark, can meet hosting and records needs, but only when the hosting, identity and evidence design is settled before the first table is loaded. Iceberg is licensed under the Apache License 2.0 and is read by several engines at the same time, so the data stays portable if the query engine changes. The decisions that determine whether the platform is approvable sit around the format, not inside it.
Public sector differs from commercial analytics in four ways. Records retention is set by law and schedule, so snapshot expiry, deletion and legal holds must be designed into the tables. Shared tenancy means one platform serves several agencies whose data may carry different legal basis, so row and column access must be enforced in the catalog and query layer and tested per agency. Publication to open data portals adds a second release path with its own review step. Finally, the compliance regime attaches to the data, not the software: CJIS Security Policy applies to criminal justice information, 32 CFR 2002.14(h)(2) applies to controlled unclassified information on non-federal systems, and NIS2 applies to public administration entities in the EU, with exclusions for national security and law enforcement. Confirm with your legal and records teams which regime governs each dataset before you choose a hosting model.
Trino's Iceberg connector supports Hive Metastore, AWS Glue, JDBC, REST and Nessie catalogs, and it supports time travel and schema evolution. Trino's authentication options include LDAP, OAuth 2.0, Kerberos and JWT, and its access control can be file-based, Apache Ranger or Open Policy Agent. Agree the catalog, identity provider and policy engine with your security team first, because these choices decide whether an auditor can trace who queried which agency's data. Platform choice follows from that design: Trino, Spark and a self-hosted object store fit when the team can run them, and a partner can carry the build, hardening and handover if it cannot.
Which requirements shape the public sector lakehouse build
This table maps each public sector requirement to the source that creates it and to what it changes in hosting, keys, scanning and portal design. Read it as a starting checklist: confirm which rules apply to your agency, jurisdiction and data before fixing the architecture.
| Requirement | What it asks for | What it means for the build |
|---|---|---|
| NIST SP 800-53 baselines | Low, moderate and high baselines select 149, 287 and 370 controls and enhancements respectively (SP 800-53B, Release 5.2.0) | Choose the baseline from the highest sensitivity of data in the shared platform. Map each control to a lakehouse component (storage, catalog, engines, access control) and keep that map as the evidence index. |
| FISMA and OMB Circular A-130 | Federal systems carry agency security programs, and Inspectors General independently assess them and report to OMB | Expect independent assessors to test the lakehouse as part of an agency system. Keep system boundaries, owners and continuous monitoring records current and accessible to reviewers. |
| FedRAMP scope under OMB M-24-15 | Cloud services that store or process federal information on behalf of an agency fall in scope, unless used only for one agency and not offered as a shared service | A shared, multi-agency lakehouse on cloud infrastructure is likely in scope. Confirm with the authorizing official whether the service needs FedRAMP authorization before choosing a hosting provider. |
| FBI CJIS Security Policy v6.1 | FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys for CJI at rest and in transit outside physically secure locations, plus vulnerability scanning at least monthly | Pin encryption to validated modules in object storage, catalog and query engine connections. Schedule authenticated scans at least monthly across Trino, Spark, catalog and storage nodes, and record remediation. |
| 32 CFR 2002.14(h)(2) for CUI | Federal agencies use NIST SP 800-171 to set protection requirements for controlled unclassified information on non-federal systems, unless the CUI category prescribes other safeguards | When contractors or partner agencies host CUI in the lakehouse, the safeguards follow SP 800-171 requirements. Confirm the CUI category first, because some categories set their own safeguards. |
| ADA Title II, WCAG 2.1 A and AA | State and local government web content, including contracted content, must meet WCAG 2.1 Level A and AA from 26 April 2027 for entities serving 50,000 or more people, and from 26 April 2028 for smaller entities and special districts | Open data portals and dashboards built on Superset or Metabase must meet these levels. Include accessibility testing in acceptance, and confirm your entity's population category and date. |
| NIS2 for EU public administration | Central government entities are covered regardless of size, public administration is a high-criticality sector, Member States may extend coverage to local administration, and national security, defense and law enforcement are excluded | Confirm whether your entity is covered and whether national law extends it. Covered entities need documented risk management, incident handling and supplier security, which shape logging, access review and vendor terms for the platform. |
How hosting rules decide where the lakehouse can run
The hosting decision starts with classifying each dataset by the regime that governs it, not with choosing a cloud or data center. The same agency may hold data under the federal control catalog in NIST SP 800-53 Rev. 5 (Release 5.2.0 is current), under FISMA and OMB Circular A-130 as the laws behind that catalog, under the FBI CJIS Security Policy for criminal justice information, under 32 CFR 2002.14(h)(2) for controlled unclassified information on non-federal systems, or, for EU-facing bodies, under the national transpositions of NIS2. Each regime sets different encryption, scanning and boundary requirements, so a single lakehouse often needs zones with different hosting rules. Map each dataset to its regime, its baseline (low, moderate or high under SP 800-53B) and its owner before the platform architecture is drawn.
Cloud hosting depends on FedRAMP scope. FedRAMP covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency, and systems used only for one agency's operations on cloud infrastructure that are not offered as a shared service fall outside it. Self-hosting Trino, Iceberg and object storage in the agency's own cloud account, operated only for that agency, usually changes which assessment applies compared with a shared SaaS lakehouse. That change does not remove the agency's obligations under its own authorization boundary. Ask your authorizing official which boundary the platform sits in, and whether any managed component such as a hosted catalog or support tunnel would make it a shared service.
Confirm each determination with your authorizing official, legal counsel and the relevant standard or agreement, not with the platform vendor. Ask counsel whether any CJI, CUI or EU personal data may leave a given zone for backup, support or vendor telemetry, and ask the security team to document the result. Open source components do not carry a compliance status of their own, so the hosting boundary, not the license, determines what evidence the agency must produce. Keep the classification register and the boundary decisions as controlled records, because they will be the first items an assessor requests.
Which open source platforms fit a shared public sector lakehouse
A shared agency lakehouse needs a table format that several engines can read, a query layer with strong authentication and access control, and governance tooling that agencies can run themselves. The cards below cover the components that meet those needs and the licensing points to review before an agency commits to a stack.
Apache Iceberg as the table format
Iceberg is the open table format that keeps records readable over time. It is licensed under Apache 2.0 and supports schema evolution without rewriting tables, time travel to exact historical snapshots, and rollback to an earlier table state. For records that must be reproduced years later, the snapshot model matters as much as the file format. Confirm which catalog your agencies will share (Hive Metastore, AWS Glue, a REST catalog or Nessie) and who operates it, because the catalog decides which engines can see which tables.
Multi-engine access for mixed agency tools
Agencies rarely standardize on one analytics tool. Iceberg lets engines such as Spark, Trino, Flink, Presto, Hive and Impala work on the same tables at the same time, so one agency can query with Spark jobs while another uses Trino or a BI tool. This avoids copying records into each agency's own warehouse. Test each engine against the catalog and the access policy you intend to run, since version support differs between engines.
Trino for federated query and access control
Trino is the query layer most agencies will use for ad hoc SQL across Iceberg and other sources. It is licensed under Apache 2.0, supports Iceberg format versions 1 and 2 (version 3 is experimental), and offers authentication by password file, LDAP, OAuth 2.0, Kerberos, certificates and JWT. Access control can be file-based or delegated to Apache Ranger or Open Policy Agent. Fault-tolerant execution is off by default, so enable it deliberately for long batch jobs and plan the spooling storage it requires.
Spark for large batch processing
Spark handles heavy transformation and batch workloads, such as rebuilding statistical tables or reconciling large program datasets, that interactive SQL engines do not run well. It reads and writes Iceberg, so its outputs remain visible to Trino and other engines. Ask the implementation partner how Spark jobs will be scheduled, which identities they run under, and how their credentials are stored, since batch jobs often hold broad write access that interactive users do not.
Orchestration with Airflow or Dagster
Pipelines that load and transform agency data need scheduling, retries, dependency tracking and an audit trail of runs. Apache Airflow (Apache 2.0, version 3.3.2) and Dagster (Apache 2.0, version 1.13.26) both fit this role. Airflow has the broader operator ecosystem; Dagster models data assets directly. Choose one, document the run history it produces, and confirm where its metadata database is hosted, because that database becomes part of the evidence trail.
Catalog and lineage with DataHub or OpenMetadata
Agencies need to know which dataset holds which records, who owns it and where a published figure came from. DataHub (Apache 2.0, version 1.7.0.1) and OpenMetadata (Apache 2.0, version 2.0.5) both provide discovery, ownership and lineage. Compare them against your identity provider integration, your connector coverage for the systems in your estate, and the effort to keep ownership fields current across agencies.
Superset for agency dashboards and Metabase with license limits
Apache Superset (Apache 2.0, version 6.1.0) provides agency dashboards with row-level security and SSO integration, which suits shared internal reporting. Metabase is AGPL outside its enterprise directory, and that directory uses the Metabase Commercial License, so confirm which features your agency needs and whether they fall under the commercial terms before standardizing on it. Review Dify and Airbyte licensing too: Dify's modified Apache license has conditions on multi-tenant hosting, and the Airbyte platform repository uses the Elastic License 2.0. Both need legal review before they enter the stack.
Zones, tenancy and trust boundaries for cross-agency access
A shared lakehouse only works if agencies can share infrastructure without seeing each other's data by accident. The cards below cover the zone layout, tenancy model, identity, key management, publication path and retention decisions that set those boundaries. Settle them before the first agency onboards, because changing them later means moving data and rewriting access rules.
Separate raw, curated and published zones
Keep three distinct zones on the object store: raw landing data as received, curated data that has been validated and conformed, and published data that is safe to release. Each zone gets its own bucket or prefix, its own write path and its own read grants. Raw zones usually hold the most sensitive records, so restrict them to ingestion service accounts and a small set of stewards. Ask the implementer to show which identities can write to each zone and how a promotion between zones is logged.
Per-agency schemas with row and column policies
Give each agency its own namespace, such as a schema or Iceberg catalog namespace, so ownership and grants are visible in one place. Shared reference tables can live in a common namespace with read-only access. Within a table, use row filters and column masks for fields that only some agencies may see, such as personal identifiers or case details. Trino supports file-based access control and integration with Apache Ranger or Open Policy Agent, so confirm which option your governance team will maintain and test it with real cross-agency queries.
Shared identity provider with agency group mapping
Use one identity provider for staff sign-in across the query engine, catalog, notebooks and dashboards, and map agency groups from the directory to the roles that grant table access. Trino supports LDAP, OAuth 2.0, Kerberos, certificates and JWT, so the choice depends on what your agencies already run. Confirm how deprovisioning works: when a person leaves an agency, their group membership and any cached tokens must stop granting access. Ask who owns the group mapping and how often it is reviewed.
Encryption keys and FIPS-validated modules
Where criminal justice information or controlled unclassified information is stored, the CJIS Security Policy v6.1 expects FIPS 140-3 validated cryptographic modules, or FIPS 197 AES with at least 256-bit keys, for data in transit and at rest outside physically secure locations. Decide who holds the key material, whether keys sit in an agency-run key management service, and how rotation is performed without breaking table reads. Ask the implementer to document which components use validated modules and which do not, because the certificate status of a library does not transfer to the whole stack.
Dedicated publication pipeline with human approval
Publishing to an open data portal should be a separate job from analytics, with its own service account and an explicit approval step before anything leaves the published zone. The pipeline should run a disclosure check, such as suppressing small cells and confirming that masked columns are absent, then record who approved each release. Treat the portal as an external boundary: a release that is later found to contain personal data is hard to recall. Ask how a published dataset is withdrawn and how that withdrawal is recorded.
Backup and retention tiers by zone
Set retention separately for each zone and for the table snapshots within it. Raw data may need a longer legal hold, curated data may follow an agency records schedule, and published data may need to be kept as a public record once released. Iceberg snapshots and time travel keep older versions until they are expired, so retention settings must include snapshot expiry and orphan file cleanup, not just deletion of current rows. Confirm the backup target for the object store and catalog, test a restore into an isolated environment, and ask who signs off on each schedule.
How a public sector lakehouse rollout runs in phases
The sequence moves from knowing what data the agencies hold and which rules apply, to a hardened shared platform, a tested pilot with two agencies, a publication path to open data portals, and a handover the agency's own team can operate and upgrade.
Classify data and map regimes
DeployOpen works with each participating agency's data owners, records functional datasets, their sensitivity, the laws and policies that govern them, and any retention obligations. The output is a classification register that assigns each dataset to a zone and names the control baseline it must meet, such as a NIST SP 800-53 baseline, CJIS Security Policy obligations or CUI handling rules where they apply. The step closes when the agency data owners have signed off the register and open questions are logged with an owner.
Design governance and tenancy
The team defines the access model: which agency roles can read, write or publish each zone, how cross-agency sharing is granted and revoked, and how identities from the agency directories are federated into Trino authentication. It also sets the catalog design, including namespaces per agency, the Iceberg catalog choice and naming standards. The output is an access model and catalog design document, which the agencies' security and records teams review before any build starts.
Build the hardened platform
Engineers deploy Trino, the Iceberg catalog, object storage and supporting services inside the agency-approved hosting environment, then map each hardening measure to the agreed control baseline. Work covers encryption in transit and at rest with the key lengths the applicable policy requires, logging, vulnerability scanning cadence, backup and restore tests, and separation of administrative access. The output is a security baseline mapping that lists each control, its implementation and its evidence location, ready for the agency's assessor.
Run the two-agency pilot
Two agencies migrate a defined set of tables from their current stores into Iceberg, using schema evolution and snapshot-based validation to confirm row counts, types and key totals against the source. Testers then run the access model from each agency's perspective, including attempts that should fail. The output is a set of migrated tables with reconciliation records and a tested access report, and the pilot closes when the agencies accept both in writing.
Build the open data publication pipeline
Engineers build the path from curated tables to the open data portal, including approval gates, de-identification checks, metadata export to the catalog, and scheduled refreshes. Publishable outputs are tested against WCAG 2.1 Level A and AA for any web content the pipeline generates or the portal presents, using both automated checks and manual review with assistive technology. The output is a publication runbook covering approvals, rollback and accessibility evidence, and the step closes when a dataset has been published and withdrawn once in a controlled test.
Hand over operations and upgrades
DeployOpen trains the agency platform team on daily operations, access requests, table maintenance such as snapshot expiry and compaction, backup restores and monitoring. It delivers an operations runbook and an upgrade runbook that cover Trino and Iceberg version changes, testing in a non-production environment and rollback steps. The step is complete when the agency team has run a restore and an upgrade rehearsal under observation, and the open items list has an owner for each entry.
Evidence assessors and auditors ask for from the lakehouse
This table lists the evidence items an assessor or auditor is likely to request from a public sector lakehouse, the component that produces each item, and where it usually falls short. Read it as a checklist to design logging, catalog and scanning before the first assessment window opens. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or certification.
| Evidence item | Producing component | What to confirm before assessment |
|---|---|---|
| Access control configuration exports | Trino access control (file-based, Apache Ranger or Open Policy Agent) and catalog grants | Exports are dated, versioned in change control, and show cross-agency roles mapped to named data sets |
| Authentication logs | Trino event logs and the identity provider (LDAP, OAuth 2.0, Kerberos or certificates) | Logs record user, source and result, and are retained for the period the agency's records schedule requires |
| Iceberg snapshot history for records integrity | Iceberg table metadata and snapshot list, queried through time travel | Snapshot expiry and orphan file cleanup policies are documented so that retained records are not removed early |
| Vulnerability scan reports at the required frequency | Scanner in the CI pipeline and against running hosts, with exported reports | Scan cadence matches the baseline the agency adopts, such as the monthly scanning CJIS Security Policy v6.1 requires where CJI is in scope |
| Encryption configuration showing key length and module validation | Object storage, Trino TLS settings and key management system configuration | Keys are AES-256 where the baseline requires it, and the cryptographic modules show FIPS 140-3 validation status from the vendor |
| Lineage and catalog ownership records | DataHub or OpenMetadata catalog, with ownership and lineage captured from pipelines | Every published data set has a named owner, a classification and lineage back to its source |
| Accessibility test results for published portals | CI pipeline accessibility tests and manual review of the portal before publication | Results map to WCAG 2.1 Level A and AA, and the agency confirms which compliance date applies to its population size |
| Change and configuration history for the platform | Git repositories, CI logs and deployment records for Trino, catalogs and pipelines | Every production change links to an approved request, a test result and a rollback plan |
Keeping long-term public records readable with open table formats
An Iceberg snapshot is the unit of reproducibility, and it is also the unit of retention. Each committed change to an Apache Iceberg table creates a snapshot, and time travel lets an analyst query the table as it stood at a given point, which is what an agency needs when a published statistic or a benefits decision has to be reproduced years later. Schema evolution without rewriting tables lets the columns change as programs change, while the older snapshots still describe the data as it was recorded. Because the data files are Parquet on object storage that the agency controls, the records do not depend on one vendor's proprietary format or on a warehouse that might be retired. Confirm with your records office and your catalog team which snapshots must be queryable, for how long, and who may roll a table back.
Keeping a snapshot is not the same as meeting a retention schedule. A snapshot that is still referenced keeps the data files it points to, so a record that was deleted in the current table can still be read through an older snapshot until that snapshot is expired. Removing the data for a privacy request or an expired retention period therefore takes three steps: delete the rows, expire the snapshots that reference them under a documented policy, and then clean up orphan files that no snapshot or table metadata still references. Plan for these steps in the operating model, assign an owner to each, and test them on a copy of a real table before a deletion request arrives. Record the expiry settings and each run so the agency can show when data left the platform.
Retention rules are a legal decision, so the platform should implement them rather than define them. Ask the records office which categories of record must be preserved unchanged, which may be deleted after a defined period, and which need a legal hold that overrides expiry. Then agree how time travel windows, snapshot expiry and orphan file cleanup map to each category, and who approves an exception. Keep the catalog and access logs in the same retention regime as the data they describe, because an auditor will ask for both. Confirm these rules in writing before the first production load, since changing them later means reprocessing tables that already carry history.
What agencies ask about cost, effort and running an open lakehouse
What drives the cost of a public sector open lakehouse?
Cost comes from four main groups: staff time, storage, compute and assessment effort. Staff time covers platform engineering, data engineering, security configuration, and the time agency data owners spend on classification and access decisions. Storage and compute are recurring and scale with data volume, retention periods and query concurrency. Classification and assessment effort is often the largest one-time item when data includes criminal justice information, controlled unclassified information or personal data. Ask any partner for a quote that separates one-time work (design, migration, hardening, evidence preparation) from recurring work (hosting, patching, upgrades, table maintenance and support), and that states which party pays for each line. No cost figure is given here, so request a written scope before comparing options.
How long does it take to deliver an open lakehouse for an agency?
The timeline depends on the number of source systems, the data classifications involved, the number of participating agencies, and how quickly decisions are made on access and retention. No duration figure is supplied for this work, so do not rely on a generic estimate. Ask for a written scope that lists phases, deliverables, dependencies on agency staff and the approvals each phase needs, such as security authorization or policy sign-off. Engagements are typically fixed-scope, so the agreed timeline should be tied to that written scope rather than to a verbal target.
What must the agency provide before an implementation partner starts?
The agency carries the decisions that no implementation partner can make on its behalf. These include named data owners for each dataset, a decision on which identity provider will authenticate staff and partner users, and written access policies for each agency and role. The agency also needs to confirm which laws and frameworks apply to each dataset, such as FISMA-related controls, CJIS Security Policy, or 32 CFR Part 2002 requirements for CUI, and set retention schedules for records. Without these inputs, the partner cannot configure access control or evidence collection correctly, and the schedule will slip. Agree these inputs in the scoping phase and record who signs off each one.
Can an internal agency IT team run an open lakehouse on its own?
Some agencies run the platform with an internal team, and others use a partner for build and then run it internally. The practical question is whether the team has skills across several areas: Kubernetes or infrastructure operations, Trino or Spark tuning, Iceberg table maintenance such as snapshot expiration and orphan file cleanup, catalog and identity integration, backups and upgrades. Trino's documentation covers deployment on Kubernetes with Helm, and it requires a 64-bit Java 25 runtime, minimum 25.0.1, so the team must keep the runtime current. Assess coverage honestly for the people on call, not only the people who built the system. A partner can cover build, hardening, documentation and handover, and can provide ongoing support under a retainer if the agency wants that.
How do compliance regimes change the effort for a lakehouse?
Each regime adds specific work. Under the FBI CJIS Security Policy, version 6.1 took effect on 26 June 2026 and requires FIPS 140-3 certified modules or FIPS 197 AES with keys of at least 256 bits for CJI in transit and at rest outside physically secure locations, plus vulnerability scanning at least monthly. A CUI project under 32 CFR 2002.14(h)(2) points to NIST SP 800-171 for non-federal systems. A defense contractor storing CUI in a cloud service must use a FedRAMP Authorized service at the Moderate baseline or higher, under 32 CFR 170.17(c)(5). A federal system built on NIST SP 800-53 Rev. 5 uses baselines in SP 800-53B that select 149, 287 or 370 controls and enhancements for low, moderate or high. Each baseline increases documentation, testing and monitoring effort. Confirm the applicable regime with your legal and security leads before scoping.
Does DeployOpen certify that the lakehouse is compliant?
No. DeployOpen does not issue audit reports, attestations or certifications. It prepares systems, controls and evidence for compliance assessments, and independent auditors, assessors or certification bodies make the determination. For example, under FedRAMP, authorization decisions are made through the program's process, not by an implementation partner. Ask a partner to describe what evidence it will produce, such as configuration baselines, access control records, encryption settings, scan results and runbooks, and which assessor will review each item. Agree in writing who owns each control after handover.
What recurring work should an agency budget for after go-live?
Recurring work falls into several groups. Operations include patching and upgrading Trino, the catalog service, the object storage layer and the BI tools, plus monitoring and on-call coverage. Data work includes Iceberg table maintenance such as compaction, snapshot expiration and orphan file cleanup, and access review as staff and agency roles change. Security work includes monthly vulnerability scanning where CJIS applies, key rotation and periodic review of access. Support can be provided through a retainer, and the terms, response expectations and included upgrades should be written into the agreement. Ask each partner to list these items separately in its quote so the agency can compare recurring costs over the life of the platform.
How DeployOpen scopes and supports a public sector lakehouse
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the agency's own cloud accounts, data centers or isolated networks. For a public sector lakehouse, we start from the hosting rules, records obligations and access model that apply to your agency, then build and hand over a platform your own team can run.
How we scope
- Written scope covers hosting location, data classifications and the compliance baseline your agency must evidence.
- We confirm table format, catalog, query engines and object storage choices against your records and portal requirements.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any work starts.
How we build
- We deploy Iceberg tables, a catalog, Trino and object storage inside your infrastructure, with no data leaving your environment.
- Cross-agency access control is configured with the authentication and policy options your agency has approved.
- Security testing and configuration hardening are carried out against the controls your assessors will review.
How we hand over
- Documentation covers architecture, access model, retention settings, backup procedures and runbooks for table maintenance.
- Your operations staff are trained to run upgrades, snapshot expiry and access reviews on the delivered platform.
- Compliance evidence is packaged for your auditors, who issue any assessment outcome, not DeployOpen.
How we support
- Ongoing support, upgrades and continuous security testing are available as a retainer after handover.
- Upgrades are planned against open-source release cycles and tested in a non-production environment first.
- Support scope, response expectations and service terms are agreed in writing with the retainer.
