Open source data lakehouse for healthcare

Open source data lakehouse for healthcare: why the build differs from other industries

A health data team usually inherits claims extracts, EHR exports and research datasets that sit in separate stores, each with its own access rules. The question is whether a self-hosted lakehouse can bring them together under HIPAA business associate terms. It can, provided the build starts from the regulation: the HIPAA Security Rule applies to every electronic protected health information (ePHI) store the platform creates, receives, maintains or transmits, so the lakehouse, its catalog, its query engine and its logs are all in scope. The short platform answer is Trino or Spark reading Apache Iceberg tables on self-hosted object storage, with a catalog and an access control layer chosen for the audit trail they produce, not only for query speed. Several features of healthcare data shape the design. Identified and de-identified zones must be separated in storage, catalog and grants, because a re-identification risk in one zone should not spread to the other. Under 45 CFR 164.312, unique user identification, automatic logoff and encryption are the technical safeguards the platform must address, and the access model must reflect the care team's need to know. Also, there is no HIPAA certification to aim for: HHS states that no Security Rule standard requires certification, so the evidence you assemble (access reviews, audit logs, risk analysis, business associate contracts) is what an OCR review would examine. Check with counsel which of your data flows actually make you a covered entity or business associate. Before procurement, confirm three things with your compliance lead and your platform vendor. First, which catalog (for example Iceberg REST or Hive Metastore) and which authorization option (file-based, Apache Ranger or Open Policy Agent, as Trino documents) will record who queried which table. Second, whether the query engine can enforce row- and column-level policies for each zone. Third, whether the implementation partner will sign business associate terms that flow down to subcontractors, as 45 CFR 164.314 requires. Note that HHS has proposed a Security Rule revision, which as of October 2026 has no final rule, so design for the current rule with room to add controls.

Which HIPAA and health data requirements shape the lakehouse design

This table maps each health data requirement to the design decision it forces on a lakehouse: zones, catalog access, key management and incident reporting. Read each row as a question to answer in the architecture before any engine is installed. The main takeaway is that access control, audit and partner terms must be designed into the catalog and storage layers, since they cannot be added later as a reporting layer.

Requirement	What it asks for	Lakehouse design consequence
Unique user identification (45 CFR 164.312(a)(2)(i))	Each user of ePHI has a unique name or number, so activity can be traced to one person.	Ban shared query accounts. Bind Trino or Spark identities to the identity provider and map every table read to a named user in the catalog logs.
Emergency access procedure (45 CFR 164.312(a)(2)(ii))	A documented way to reach ePHI during an emergency, with the access itself reviewed afterward.	Define a break-glass role for the identified zone with approval, time limits and automatic review. Test it in the lakehouse runbook, not only on paper.
Automatic logoff and encryption (45 CFR 164.312(a)(2)(iii), 164.312(a)(2)(iv), 164.312(e)(2)(ii))	Automatic logoff and encryption are addressable, so the entity decides and documents the approach and any alternative.	Set session timeouts in query clients and BI tools. Encrypt Parquet data at rest in object storage and in transit between engines, catalog and clients, and document why each choice fits the risk analysis.
Key management for encrypted zones	No single standard names a key product, but encryption specifications and the risk analysis require controlled keys and a record of who can use them.	Separate keys for the identified and de-identified zones. Keep key custody outside the engine admins' sole control and log every key use. Confirm rotation and recovery steps with the key service and your security team.
Business associate contract terms (45 CFR 164.314(a)(2)(i))	Vendors and partners handling ePHI must comply with the Security Rule, flow terms to subcontractors and report security incidents, including breaches of unsecured PHI.	Document which platform components touch ePHI, who operates them and which subcontractors have access. Include the implementation partner and any hosting provider in this inventory and in the contract chain.
Incident reporting to the covered entity	Security incidents and breaches must reach the covered entity through the contract's reporting path, on the timing the contract sets.	Feed Trino query logs, catalog audit events and object storage access logs into a monitored store. Agree the escalation path and evidence package with the covered entity before go-live.
2025 proposed Security Rule revision (90 FR 898)	HHS proposed stronger requirements in January 2025. As of October 2026 no final rule has been published, and the Unified Agenda projects final action for July 2027.	Build to the current rule and keep the design modular, so controls such as asset inventories, network segmentation and stronger audit can be added without rebuilding zones. Track RIN 0945-AA22 and reassess at each release.
GDPR Article 32 (security of processing)	Controls appropriate to the risk, including access restriction, pseudonymization and encryption, with regular testing of their effectiveness.	Restrict identified data to named care or research roles. Pseudonymize patient keys before data reaches the research zone. Schedule access reviews and security testing against the lakehouse.
GDPR Article 34 and the CNIL Hopital Prive de la Loire case	CNIL fined the hospital EUR 500,000 on 3 September 2026 under Articles 32 and 34, citing external access without VPN or multi-factor authentication and access control that ignored the care team's need to know.	Require VPN or private access plus multi-factor authentication for every external or remote session. Grant access by care-team relationship and purpose, not by job title alone. Prepare breach notification workflows that meet Article 34 timing for affected individuals.
NIS2 health sector scope (Annex I)	Healthcare providers, EU reference laboratories, medicinal product R&D and certain device manufacturers are listed as sectors of high criticality for EU entities.	Confirm with counsel whether your entity falls in scope. If it does, add NIS2 incident reporting and supply chain obligations to the zone design, catalog ownership and partner terms.

How claims, clinical and research data flow into identified and de-identified zones

Start by fixing the flow before any table is created. Claims extracts, EHR feeds, FHIR resources and HL7 v2 messages land first in a raw zone that mirrors the source as received, with the transport metadata (sender, message type, receipt time) kept alongside. Identified clinical data then moves into an identified namespace, and only a governed transformation produces the de-identified namespace used for research and broad analytics. Each step is a separate Iceberg namespace with its own write role, so no analyst or pipeline can write to the identified zone through the de-identified path. Keep the raw zone restricted as well, because it often holds the identifiers before any transformation has run.

Separate namespaces, or separate catalogs where the risk justifies it, make access control enforceable rather than a matter of view definitions. Trino's Iceberg connector reads through a configured catalog (iceberg.catalog.type selects the metastore, such as Hive Metastore, AWS Glue or a REST catalog), and its access control can be file-based, delegated to Apache Ranger or Open Policy Agent, or driven by authentication such as LDAP, OAuth 2.0 or JWT. Map each zone to a catalog and a role, then test that a research account cannot reach the identified catalog at all. Schema changes are the usual source of drift: Iceberg supports schema evolution without rewriting tables, but a new HL7 or FHIR field still needs a mapping decision and a de-identification review before it reaches the research zone.

Before ingestion, confirm with each source vendor which message versions, FHIR profiles and HL7 segments are actually emitted, how deletions and corrections are signaled, and whether identifiers such as patient IDs can be replaced with tokens at the source. Ask for sample extracts and a data dictionary, and agree how late or amended records are handled, since these affect Iceberg snapshot and merge design. Also confirm which fields the vendor considers identifiers under HIPAA and whether the business associate contract covers the transfer. Written answers to these questions become the input to the de-identification method and the access matrix, and they should be signed off by the privacy officer before the first load.

Which open source platforms fit a healthcare lakehouse and why

A healthcare lakehouse is a set of roles, not a single product. Table format, query engine, orchestration, ingestion, dashboards and metadata each carry different licensing, access control and evidence obligations, and the choices in one layer constrain the others. The cards below set out what each role does for PHI-bearing analytics and what to confirm before committing.

Apache Iceberg as the table format

Iceberg is the layer that makes historical results reproducible. It is licensed under the Apache License 2.0 and supports time travel to exact historical snapshots and rollback to an earlier table state, which matters when a clinical or claims report must be re-run as it stood on a past date. Check that your engines and catalog support the format versions you need, and decide early how snapshot retention interacts with your records schedule and with any erasure obligations, since old snapshots keep deleted rows until they expire.

Trino as the query engine

Trino queries data in place and needs its own catalog and storage around it. It is licensed under Apache-2.0 and reads Iceberg tables through its Iceberg connector. Access control can be file-based, delegated to Apache Ranger or implemented with Open Policy Agent, and authentication supports LDAP and certificates, which fits environments that cannot rely on public cloud identity. Confirm that column and row restrictions for identified data are enforced by the chosen policy engine and tested, not assumed.

Fault-tolerant execution needs spooling

Trino's fault-tolerant execution is off by default and needs a spooling exchange manager. Enabling a retry policy is a deliberate choice. The TASK policy, recommended for large batch queries, requires an exchange manager that spools intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS. In a self-hosted healthcare estate, that spool becomes another store holding query intermediates, so it needs the same encryption, retention and access controls as the lakehouse itself. Ask your partner where the spool lives and how it is purged.

Airflow or Dagster for orchestration

Orchestration determines who can see and rerun data movements. Apache Airflow (Apache-2.0) and Dagster (Apache-2.0) both schedule and monitor pipelines as code, which gives you version-controlled, reviewable jobs that handle de-identification steps in a defined order. Choose based on your team's skills and the granularity of lineage you need. Either way, confirm that run logs do not capture identifiers in plain text and that pipeline credentials are held in a secrets store, not in job definitions.

Airbyte and its Elastic License caveat

Airbyte's platform repository is under the Elastic License 2.0, which is not an OSI open source license. That license restricts providing the software to third parties as a hosted or managed service, which does not usually restrict an internal health system, but legal review should confirm it against your use and any contracts with partners. It is also worth reviewing whether the connectors your source systems require are under permissive terms before adopting them. Treat Airbyte as a licensed component to be cleared, not a default.

Apache Superset for dashboards

Superset provides governed dashboards on the same tables, under Apache-2.0. It connects to query engines such as Trino, so clinical and operational dashboards read the lakehouse rather than a copied extract. Healthcare use depends on row-level security, SSO and audit of who viewed which dashboard, and these features need to be configured and tested for each dashboard group. Confirm that exports and scheduled reports do not leave the controlled zone.

DataHub or OpenMetadata for lineage

A catalog gives auditors a traceable path from a published figure back to its source. Both DataHub and OpenMetadata are Apache-2.0 metadata platforms covering discovery, lineage and ownership. For HIPAA and research sharing, the catalog should record which tables hold ePHI, the owner of each dataset and which zone it belongs to. Confirm that the catalog itself does not store sample values containing identifiers, since catalog metadata often is widely readable.

Deployment patterns for hospitals, payers and research networks

A healthcare lakehouse fails or succeeds on a small set of placement and separation decisions made early. The cards below cover where the platform runs, how identified and research data are kept apart, how the Trino cluster is laid out, and how keys and research access are handled, so each choice can be reviewed against HIPAA obligations and your own risk file.

On premises or own cloud account

Decide whether the lakehouse runs in your data center or in a cloud account you control. Either way, the business associate and covered entity relationship defines who can administer storage, query engines and logs. Ask where backups, snapshots and Spark or Trino spill files land, since copies of ePHI often end up outside the primary zone. Confirm that every location is covered by your risk analysis and by any business associate contract that applies.

Separate identified and research zones

Keep identified patient data in a restricted zone and publish research extracts from a separate zone with de-identified or limited data sets. Separate storage buckets, catalogs and query access groups make the boundary visible to auditors and reduce the blast radius of a credential leak. Ask how a record moves from identified to research zones, who approves each move, and whether the transform is logged.

Trino coordinator and worker layout

A Trino cluster has one coordinator and one or more workers, and the documentation recommends a dedicated coordinator for larger clusters. For healthcare, run separate clusters or resource groups for identified and research workloads so a heavy research query cannot starve clinical reporting. Confirm that the coordinator is the only node that clients authenticate against, and that worker nodes sit on a segment without direct user access.

Kubernetes with Helm as an option

Trino has a Helm-based Kubernetes guide, which can suit hospital platform teams already operating clusters. The trade-off is that the cluster, network policies, secrets and node pools become part of the evidence you must maintain. Confirm which Kubernetes distribution your organization supports, how images are sourced and scanned, and whether the cluster can run without internet access for air-gapped or restricted segments.

Key management for encryption

The HIPAA Security Rule treats encryption of ePHI as an addressable specification, so the decision must be documented even when encryption is applied. Decide who holds the keys for object storage encryption, catalog credentials and backups, and whether keys sit in an HSM or a self-managed key service. Ask how key rotation interacts with Iceberg snapshots and backups, and who can revoke access to a research copy.

Research sharing without copying identified data

Researchers often need to join cohorts across institutions without receiving identified rows. Options include publishing aggregate or de-identified tables, running approved queries inside your environment, or granting time-limited read access to a research catalog. Each option changes what leaves your network and what a later breach would expose. Ask which data sharing agreements and privacy reviews apply before any extract is created, and how sharing is logged.

How a healthcare lakehouse rollout runs, phase by phase

The sequence moves a healthcare lakehouse from scoped data sources to a documented, handed-over platform, with each phase closing on a named output that the covered entity can review. Durations depend on the scope agreed in writing before work starts.

Scope data sources and analyze risk
DeployOpen works with your data, compliance and clinical informatics leads to inventory claims, clinical, research and reference sources, including any FHIR and HL7 feeds, and records which ones carry ePHI. Each source is tagged with its owner, sensitivity, retention obligation and whether the data is identified, limited or de-identified. The step closes with a written data inventory and a risk analysis that feeds the HIPAA Security Rule risk analysis your organization maintains, with the scope confirmed in the signed statement of work.
Design catalog and access with HIPAA mapping
The team designs the table format and catalog choice, the storage layout for de-identified and identified zones, and the authentication and authorization model covering unique user identification, automatic logoff and emergency access. Each design decision is mapped to the relevant standard and implementation specification in 45 CFR 164.312 and to the administrative and physical safeguards in Appendix A to Subpart C. The output is a design document with a control mapping that your security and privacy officers review and approve before any build begins.
Run a pilot on de-identified data
The first build loads a limited set of de-identified datasets into the lakehouse, with Trino or Spark queries, dashboards and a documented process for re-identification controls. Engineers test schema evolution, time travel and ingestion from representative sources, and your analysts validate query results against source reports. The pilot closes with a test report listing defects, performance observations and any gaps in the de-identification approach that must be resolved before identified data enters the platform.
Onboard identified zone under business associate terms
Identified ePHI moves into the restricted zone only after the business associate contract is executed and confirms the terms required by 45 CFR 164.314, including Security Rule compliance, flow-down to subcontractors and incident reporting. DeployOpen configures network isolation, encryption, role-based access and break-glass procedures to match the design, and onboarding is limited to the data owners and care or research teams with a documented need to know. The step ends with an access review signed by the data owner for each group that has been granted access.
Validate audit logging
Engineers enable query, access and administrative logging across the catalog, query engines, object storage and the BI layer, then forward logs to a protected, centralized store with retention set by your policy. Test cases simulate normal access, denied access, privileged changes and emergency access, and the team confirms each event appears with the user, time, object and action recorded. The output is an audit logging validation report that maps logging coverage to the audit controls your compliance team tracks.
Hand over with runbooks and evidence
DeployOpen delivers operational runbooks for ingestion, table maintenance, snapshot expiration, backup and restore, access requests, incident response and upgrades, along with architecture diagrams and the control evidence collected during earlier phases. Your engineers run a supervised exercise, such as a restore test or an access revocation, to confirm they can operate the platform without DeployOpen. The handover is complete when the signed acceptance records the runbooks, evidence package and open items, and ongoing support, if any, is agreed separately as a retainer.

Evidence auditors and regulators ask for from the lakehouse

This table maps the evidence a HIPAA assessor, HHS Office for Civil Rights reviewer or customer security team is likely to request to the lakehouse component that produces it. Read it as a checklist of what to design for before go-live, not as a list of certifications. DeployOpen prepares this evidence, while independent auditors and certification bodies issue any formal reports.

Evidence	Component that produces it	What the reviewer checks
Access control configuration	Trino access control (file-based, Apache Ranger or Open Policy Agent) and catalog-level grants	Who can reach identified versus de-identified zones, and whether grants match the care team's need to know
Unique user records	Identity provider integration (LDAP, OAuth 2.0, Kerberos or JWT) with Trino authentication	Each query traces to one named person, with no shared service accounts for analyst access
Audit logs	Trino query event logging, object storage access logs and catalog change history	Who queried which PHI tables, when, and whether logs are retained and protected from editing
Encryption configuration	Object storage encryption settings, TLS on Trino and catalog connections, and key management records	Evidence that ePHI is encrypted at rest and in transit, and how keys are rotated and restricted
Business associate agreement mapping	Contract register listing each vendor or hosting party with ePHI access, plus subcontractor flow-down terms	Every party that can touch the lakehouse is covered by a contract meeting the requirements in 45 CFR 164.314
Risk analysis inputs	Data flow diagrams, asset inventory of tables, zones, catalogs and engines, and threat register	The risk analysis covers every zone, copy and export path, including Spark, notebooks and BI tools
Iceberg snapshot history for report reproduction	Iceberg snapshot metadata and time travel queries, with the report query and parameters stored alongside	A figure in a past submission can be regenerated from the snapshot it used, and the snapshot is retained long enough
Incident and breach records	Alerting on audit logs, incident runbooks and records of notifications to the covered entity	Security incidents are detected, documented and reported within the timelines set in the business associate contract

How HIPAA, HITRUST and EU health rules change the build

The HIPAA Security Rule in 45 CFR Part 164 Subpart C applies to covered entities and their business associates for all electronic protected health information they create, receive, maintain or transmit. For a lakehouse, that means claims and clinical extracts in object storage, the query engines that read them, and the catalog that exposes table definitions are all in scope. The technical safeguards in 45 CFR 164.312 shape the build most directly: unique user identification, emergency access, automatic logoff and encryption are the areas where engine and catalog configuration becomes evidence.

Unique user identification: Each analyst and service account needs its own identity passed through to Trino or Spark, not a shared query account.
Audit controls: Query logs, catalog changes and table writes should be retained in a store the security team can review.
Business associate contracts: Under 45 CFR 164.314(a)(2)(i), vendors and subcontractors that touch ePHI must be bound to the Security Rule, so self-hosting does not remove the contract question.
Penalty exposure: The inflation-adjusted civil money penalty table in 45 CFR 102.3 sets per-violation ranges that make access logging worth designing up front.
Pending revision: HHS proposed a Security Rule revision on 6 January 2025, and no final rule had been published as of October 2026, so confirm which obligations will change.

Confirm with counsel which datasets are ePHI, which parts of the lakehouse are business associate functions, and how the proposed revision would affect your design.

Questions healthcare teams ask before a lakehouse build

What drives the cost of a healthcare lakehouse build?

The main cost drivers are the volume and variety of sources, the number of access zones, and how much engineering the build needs before the first dashboard. The cost falls into four groups: storage and compute nodes in your own cloud account or data center, one-time engineering for ingestion, identity and audit, recurring operations such as patching, backups and table maintenance, and support. Costs rise with more FHIR and HL7 feeds, more identified zones that need separate controls, and more concurrent users. Ask any partner for a quote that lists each group separately, states which items are one-time and which recur, and names who pays for hardware, licenses and staff time.

How long does a lakehouse rollout take for a healthcare team?

The duration depends on the agreed scope, not on a fixed template. A rollout covering one source, one de-identified zone and read-only reporting is far shorter than one covering claims, clinical and research sources, an identified zone with audit trails, and a research sharing workflow. Engagements are fixed-scope, so the timeline is agreed in writing before work starts. Ask for milestones tied to specific deliverables, such as source onboarding, access model testing and audit log review, and for the assumptions behind each one, including how long your data owners need for approvals.

What does our team need to supply for a lakehouse project?

Three roles are needed from your side before the build starts. Data owners must define which datasets exist, who is allowed to use them and how long they are kept. Your privacy officer must approve the de-identification approach, the identified zone rules and the research sharing terms. Clinical informatics staff must explain how FHIR resources and HL7 messages are actually populated in your source systems, including local code sets and known data quality gaps. Ask the partner to list these inputs and their expected time commitment in the scope document, so they are scheduled before engineering starts.

Can an internal team run a self-hosted healthcare lakehouse?

An internal team can run it if it has skills in object storage, query engines, identity integration, Kubernetes or VM operations and security logging. The operating work is continuous: patching engines, rotating credentials, checking backups, maintaining tables and reviewing access logs. Trino, for example, runs as a coordinator and workers and requires a 64-bit Java 25 runtime, minimum 25.0.1, so the team must track its upgrades. Many organizations split the work, with the internal team owning data use and the partner providing upgrades, documentation and support under a retainer. Decide this before the build, and write the handover and runbooks into the scope.

Does a lakehouse replace our EHR analytics tool?

Usually it does not replace the EHR, and it may not replace the reporting tool built into it. A lakehouse is an analytical store that receives copies of data from source systems, including EHR exports, claims feeds and research datasets. Operational workflows, clinical documentation and order entry stay in the source system. Some organizations retire overlapping reports over time, while others keep the EHR reporting for clinical staff and use the lakehouse for cross-source analysis and research. Confirm with clinical informatics which reports must remain in the EHR, and which data feeds the lakehouse, before scoping the work.

Do we need HIPAA certification for a self-hosted lakehouse?

No. HHS states that no Security Rule standard requires an entity to certify compliance, and private certifications do not prevent HHS from later finding a violation. The HIPAA Security Rule applies to covered entities and business associates for all electronic protected health information they handle. Compliance is shown through risk analysis, safeguards, documentation and evidence that controls work, and HHS enforces through complaints, compliance reviews and audits. If a partner or customer asks for a certificate, ask what it covers. HITRUST offers its own assessments, which are separate from HIPAA and carry their own validity periods, and DeployOpen does not issue audit reports or certifications.

Which HIPAA technical safeguards does a lakehouse need to address?

The five technical safeguard standards in 45 CFR 164.312 cover access control, audit controls, integrity, person or entity authentication and transmission security. The implementation specifications include unique user identification and an emergency access procedure, which are required, and automatic logoff and encryption, which are addressable. In a lakehouse, this means unique identities for every analyst and service account, role-based access to identified and de-identified zones, query and audit logs that record who read which tables, and encryption in transit and at rest. Ask the partner to map each control to the specific engine, catalog and storage component that enforces it.

What should a business associate agreement require for a hosted lakehouse?

Any vendor or service provider that handles ePHI for you needs a business associate contract. Under 45 CFR 164.314, the contract must require the business associate to comply with the Security Rule, pass the same terms to subcontractors that handle ePHI, and report security incidents, including breaches of unsecured PHI. For a self-hosted build, the agreement question covers any partner that can reach your data, any managed storage or backup provider, and any cloud provider hosting your accounts. Have legal counsel review the contract terms with the partner before access to production data is granted.

What to ask an implementation partner for a healthcare lakehouse

Start with the zone design, because every other answer depends on it. Ask the partner to show how the de-identified and identified zones are separated: which storage buckets or namespaces hold each, which catalog namespaces map to them, and which service accounts can write across the boundary. Ask how identifiers are removed or tokenized on the way from the identified zone to the de-identified one, who approves changes to that pipeline, and how a research extract is tested for re-identification risk before release. For access control, ask how the chosen query engine integrates with your identity provider, whether role and row-level rules are enforced in one policy layer or repeated in each engine, and how unique user identification and emergency access procedures from the HIPAA technical safeguards are implemented and tested. Ask for a sample of the access policy as code so your security team can review it before build starts.

Next, pin down audit logs and the contract trail. Ask where query, login and policy-change logs are written, how long they are retained, and how they are protected against alteration, such as write-once storage, hashing or shipping to a separate account that the platform team cannot administer. Ask who can read the logs and how that access is itself logged. On business associate terms, ask for the partner's business associate agreement template before any scoping starts, and confirm it flows the Security Rule obligations down to subcontractors that touch ePHI, including any cloud or hosting provider. Ask how the partner reports security incidents, including breaches of unsecured PHI, and what timeline it commits to in writing. Treat any claim of HIPAA certification as a red flag. HHS states there is no HIPAA certification, so ask instead which assessment the partner's evidence will support and who performs it. Finally, ask how research data sharing terms are written into the data use agreements, and how snapshot expiry and table maintenance are scheduled so deleted or expired records do not persist in old snapshots, with handover covering runbooks and the named operators who will run them.

Open source data lakehouse for healthcare