Open source data lakehouse in air-gapped networks

Why an air-gapped open source data lakehouse is a different build

Iceberg over on-premises object storage, queried with Trino or Spark, can run on a network with no route to the internet, and no cloud warehouse or cloud identity service is needed to make it work. The difference from a connected build is that nothing arrives on its own. Every engine image, JVM, connector, library, catalog component, operator and security rule set has to be selected, verified, carried across the boundary and recorded, and every later upgrade repeats that cycle. The practical shortlist for the core stack is Apache Iceberg as the table format, Trino or Spark as the query engines, Ceph Object Gateway as the S3-compatible store, and an Iceberg REST catalog or Hive Metastore for table metadata, with DataHub or OpenMetadata as optional catalog layers to confirm against the network's support requirements.

In a connected build, a missing dependency is a ticket. On an isolated network it is a delay of a transfer cycle, and a wrong version can stall a whole rollout. Before design starts, confirm three things with the network owner: how media crosses the boundary and who approves each transfer, whether Kubernetes images can be pre-pulled and pushed to a private registry (the kubeadm documentation describes listing and pulling control plane images in advance, and pointing kubeadm init at a private registry with --image-repository), and whether the catalog can authenticate users from the directory service already on the enclave instead of a cloud identity provider. Ceph Object Gateway supports Keystone and LDAP authentication and Open Policy Agent integration, which gives a path to keep identity on site. Several licensing points also change the shortlist. Some vendors reserve air-gapped deployment for paid editions, so the licence entitlement for offline use must be confirmed in writing before a component is selected. Metabase, for example, lists offline deployment only on its Enterprise plan, so a free BI tool may need replacing with Apache Superset, which is Apache-2.0 licensed. Plan for the chain of custody for every package as well, since NERC CIP-013-2 describes verifying software integrity and authenticity for vendor software and patches, and the same discipline applies to any enclave regardless of sector.

Which requirements shape an air-gapped lakehouse build

This table maps the requirements most likely to shape an air-gapped lakehouse to their source, and to what each one means for the build. Read the applicability column first: the NERC CIP rows apply only where your organization is subject to those standards, and the other rows apply through your own policies, contracts and assessor expectations.

Requirement	Source and applicability	What it means for the build
Remote access to BES Cyber Systems through an intermediate system	NERC CIP-005-7 applies only to organizations that own or operate covered bulk electric system assets	Vendor and admin access to the lakehouse must pass through a controlled intermediate system with encryption and multi-factor authentication, and active vendor sessions must be detectable and disableable.
Verification of software integrity and patches	NERC CIP-013-2 applies only where the organization is subject to it; other frameworks ask similar questions	Every container image, Python or Java package, and Trino or Spark artifact carried across the gap needs a recorded hash, signature check and approval before import.
S3-compatible object storage authentication	Ceph Object Gateway supports Keystone and LDAP authentication, multi-factor authentication and Open Policy Agent integration	Confirm which identity source the storage tier uses in your environment, and make sure it works without cloud identity services, since the gap rules out external token issuers.
Query engine authentication	Trino documents password file, LDAP, Kerberos, certificates, JWT and OAuth 2.0 options	Choose an authentication method your isolated directory can serve. Test it in the enclave, because OAuth 2.0 or JWT flows often depend on an external identity provider.
Table and column access control	Trino supports file-based access control, Apache Ranger and Open Policy Agent	Decide where row and column policies live and how they are audited. Confirm each option's behavior with the vendor before choosing one.
Automatic event logs for high-risk AI systems	EU AI Act Article 12(1) requires technical logging capability over the system's lifetime; Article 19(1) requires keeping logs for at least six months unless other law requires otherwise	If an AI assistant queries the lakehouse and is in scope as a high-risk system, query and retrieval logs must be captured and retained on the isolated network. Confirm applicability with counsel.
Offline installation of platform components	Kubernetes kubeadm supports pre-pulled images and a private registry; GitLab and Wazuh publish offline guidance	Plan a mirrored registry, package repositories and documentation inside the enclave. Confirm each component's offline install path with its vendor before scoping.
Licensing for air-gapped use	Vendors such as Metabase and Mattermost place air-gapped deployment in paid editions	Check the license edition for every tool that will run disconnected, including visualization and catalog components, before committing to a design.

Which open source platforms fit an isolated lakehouse and why

An air-gapped lakehouse is a set of parallel components, each of which must work without internet access, without cloud identity and with every package imported across the boundary. The cards below cover the table format, query engine, storage, visualization and catalog layers, and flag where air-gapped support needs confirmation in the vendor's own documentation before it is written into a design.

Apache Iceberg table format

Apache Iceberg is licensed under Apache-2.0 and defines how tables are laid out as data and metadata files in object storage. Its core features, including schema evolution without rewriting tables, time travel to exact snapshots and rollback to an earlier state, matter in an isolated network because they let teams reproduce a report and recover from a bad load without restoring from an external backup service. Check which catalog you will run inside the boundary, since the table format alone does not provide access control or discovery.

Trino query engine

Trino is licensed under Apache-2.0 and reads Iceberg tables through its Iceberg connector, with Hive Metastore, AWS Glue, JDBC, REST and Nessie catalogs. For isolated networks, its authentication options include password file, LDAP, Kerberos, certificates and JWT, and its access control can be file-based, Apache Ranger or Open Policy Agent. Plan for a Java runtime and a Kubernetes or VM deployment that you can build from mirrored images, and confirm that every connector you use runs without outbound calls.

Ceph Object Gateway storage

The Ceph Object Gateway (RGW) provides S3-compatible and Swift-compatible APIs on self-hosted storage, so Iceberg data files and metadata sit on hardware you own. It supports LDAP and Keystone authentication, multi-factor authentication and Open Policy Agent integration, which lets storage access follow the same identity source as the rest of the enclosure. Ask the implementation team to validate the authentication path end to end, because storage identity is often the first place a cloud dependency appears.

Apache Superset visualization

Apache Superset is licensed under Apache-2.0 and gives analysts dashboards and SQL exploration over Trino. Because it is Apache-2.0, it can be deployed inside the enclave without a commercial license for that step, but you still need to confirm that any extensions, map tiles or font and asset downloads are mirrored locally. Test single sign-on against your directory service, since Superset must reach your identity provider from inside the isolated network.

Metabase licensing caveat

Metabase is AGPL licensed outside its enterprise directory, but its vendor pricing page lists air-gapped deployment only on its Enterprise plan. Treat this as a licensing decision before design starts: a team that assumes the community edition covers an isolated network may need to buy a commercial plan or choose a different visualization tool. Confirm the current plan terms with the vendor in writing, since plans change.

DataHub or OpenMetadata catalog

DataHub and OpenMetadata are both licensed under Apache-2.0 and provide discovery, lineage and ownership across the lakehouse. In an isolated network, the catalog is also where you record which package versions and table definitions were imported, which supports the software integrity checks that NERC CIP-013-2 expects of supply chain processes. Confirm that the connectors and search indexes they depend on can be built and updated from local mirrors, and that the data they collect never leaves the boundary.

How to get Kubernetes, images and packages across the gap

Kubernetes is the usual runtime for Trino, Spark and supporting services, so the cluster itself has to be built without internet access. With kubeadm, the approach is to list the control plane images on a connected staging host with kubeadm config images list, pull them with kubeadm config images pull, then move them into the enclave and point kubeadm init at the internal registry with --image-repository. The cluster build is only as repeatable as the staged set, so the team keeps the list, the pulled images and the CNI and CSI add-ons together as one versioned bundle.

Staging host: A connected build environment that the team controls, with a documented procedure for what enters the enclave.
Private registry: An internal registry that holds every control plane, network and storage image the cluster references.
Add-ons: Network plugin, ingress and storage drivers are often missed in the first list and must be staged the same way.
Upgrades: Each Kubernetes minor version needs its own image set, so the upgrade path is planned as a repeatable transfer.
Ownership: The platform team owns the bundle, the registry and the audit trail of every import.

Confirm with the Kubernetes distribution or vendor which additional images, certificates and binaries their install method requires beyond the kubeadm list.

How the air-gapped rollout moves from assessment to handover

The sequence turns an isolated network into a working lakehouse in stages, with each phase producing a signed-off artifact so that the boundary, the software supply chain, the catalog and the operating procedures are agreed before data moves in.

Assess the boundary and agree requirements
DeployOpen and the client's security, network and data owners map the enclave boundary, the classification of each data set, the users and roles, the table format and engines, and the controls the accreditation authority expects. Network teams document what crosses the gap, in which direction, and through which approved transfer media. The output is a written requirements baseline and a boundary diagram that the client approves. The step is done when every data flow in the diagram has a named owner and an approved path.
Build the staging and artifact pipeline
In a connected staging zone, DeployOpen assembles every container image, operating system package, Java runtime, Python wheel, Helm chart and model file the platform needs, and records their versions and checksums in a bill of materials. Kubernetes control plane images can be pulled in advance with kubeadm, and the platform components are mirrored into a private registry. The client's supply chain reviewers verify integrity and authenticity before anything is approved for transfer. The output is a signed transfer bundle with a manifest that the isolated side can verify independently.
Build the isolated platform
Inside the enclave, the team installs the Kubernetes cluster from the private registry, deploys Ceph Object Gateway for S3-compatible object storage, and brings up the query engines, starting with Trino and its Iceberg connector and any Spark jobs the workloads need. Every component is configured to resolve only internal names and to pull nothing from public repositories. The output is a cluster where a connectivity test to any external address fails as designed and each service passes its health checks. The team records the configuration in version control held inside the enclave.
Integrate the catalog and local access control
The team deploys an Iceberg catalog that the enclave can run without cloud identity, such as a REST catalog or a Hive Metastore, and connects it to Trino. Authentication is set up against the client's directory service, which may be LDAP or Kerberos, with certificates for service-to-service calls. Authorization uses Trino's file-based rules, Apache Ranger or Open Policy Agent, as the client decides. Access tests confirm that each role sees only its own schemas and that the audit trail records queries by user. The output is a signed access matrix and a passing test suite for every role.
Import and validate the data
Approved source extracts cross the gap through the agreed transfer process and are loaded into Iceberg tables in a landing zone. The team reconciles row counts, checksums and schema against the source manifests and runs the defined data quality checks. Data owners review the results and sign off on each data set before it moves to a curated schema. The output is a validation report per data set and a record of every import, including who approved it and when it crossed the boundary.
Write the operational runbook
DeployOpen documents the procedures the client's own operators will run: backups of the catalog and object storage, table maintenance such as snapshot expiry and compaction, certificate rotation, monitoring and alert handling, user onboarding, and the procedure for approving and importing each package. The runbook also covers incident response for a failed transfer. The output is a runbook reviewed by the client's operations lead, and a training session that shows each procedure is performed by the people who will own it after handover.
Rehearse the first upgrade across the gap
Before handover, the team runs a full upgrade cycle in a non-production copy of the enclave. A new release set is prepared in staging, verified, carried across using the same transfer process, and applied, with rollback tested if the upgrade fails. The rehearsal records how long each step took in the client's own environment, which the client can use to plan future releases. The output is a completed rehearsal log, a tested rollback procedure, and a handover package with the runbook, configuration, bill of materials and access matrix.

How upgrades and security updates cross an air gap

Every upgrade and security patch for the lakehouse crosses the gap as a controlled transfer, not as a live download. The transfer cycle runs in four stages: new releases are pulled and verified in a connected staging environment, scanned for known vulnerabilities, signed and packaged, then imported into the isolated network through the approved media or data diode process. Only after import does the release move into a non-production enclave that mirrors production. Container images, Helm charts, Java and Python packages, and the Ceph and Iceberg artifacts all follow this path. The Kubernetes control plane images and the Ceph packages should be pulled in advance so the import list is complete before the window opens.

Trino and Apache Iceberg do not share a release calendar, so each import needs its own compatibility check. Trino's Iceberg connector reads and writes tables through a catalog, and a Trino upgrade can change how it handles table metadata, so confirm the Trino version against the Iceberg format versions your tables use (Trino supports format versions 1 and 2, with version 3 experimental). Check the Iceberg library version against the Trino release, and run a test that writes, compacts and time-travels on a copy of production tables before any production import. Ask your implementation partner to document this matrix per release. Confirm the current Trino release line and the Iceberg release with your partner, since both move independently.

Each import needs a rollback plan written before the transfer. Keep the previous signed images, chart values and package sets in the enclave so the prior state can be redeployed within the change window, and snapshot the Iceberg catalog metadata before any schema-affecting change, because Iceberg rollback applies to table state rather than to the engine binaries. Maintain an import register that records the package name, version, hash, signature check result, scan report, approver, test evidence and the date it reached production. That register is also the evidence you will need for supply chain controls such as NERC CIP-013-2, which asks for verification of software integrity and authenticity, and it lets an auditor trace which release ran on any given date.

What auditors and assessors ask for from an isolated lakehouse

This table maps the evidence an assessor usually requests from an air-gapped lakehouse to the component that produces it and the purpose it serves. Read each row as a question to answer before the assessment starts, not after. DeployOpen prepares systems, controls and evidence for compliance assessments, but auditors and certification bodies issue the actual reports, attestations and certifications.

Evidence	Producing component	Purpose for the assessor
Import manifests and checksums	Transfer process across the gap (media log, hash files, signed package lists)	Shows that every package, engine and dependency entering the isolated network was verified for integrity and authenticity, the control behind NERC CIP-013-2 supply chain expectations
Access control policy exports	Trino access control (file-based, Apache Ranger or Open Policy Agent) and object storage IAM policies	Shows who can read, write or administer each schema and table, and that the policy matches what was approved
Iceberg snapshot history	Iceberg table metadata and snapshot logs in the catalog	Lets analysts reproduce a past report or dataset exactly, and shows what changed and when, using time travel to exact historical snapshots
Trino query audit logs	Trino coordinator event listener output, stored on local log infrastructure	Records who queried which data and when, a core input when reviewing access to sensitive tables
Change records for upgrades	Versioned change tickets, release notes and offline upgrade bundles for each component	Demonstrates that each upgrade was planned, tested in a non-production copy, approved and traceable to a specific package version
AI usage logs where Articles 12 and 19 apply	Gateway and application logs from any model or assistant running in the enclave	Where a high-risk AI system is in scope, Article 12(1) requires automatic event logging over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise

What do buyers ask before hiring a partner for an air-gapped lakehouse?

What drives the cost of an air-gapped lakehouse build?

The biggest drivers are the size of the boundary, the number of engines and connectors in scope, how many systems must be imported across the gap, and how much compliance evidence the network owner requires. Object storage capacity and compute nodes are hardware or cloud costs paid to your infrastructure provider, while the engineering work to design, install, harden and document the stack is paid to the implementation partner. Ask for a written scope that separates one-time build work from recurring items such as upgrade cycles, table maintenance and support retainers. Hardware and licensing lines should be itemized so you can compare them with your own procurement.

How long does an air-gapped lakehouse project take?

Timing depends on scope and the boundary, so there is no fixed duration that applies to every network. Key variables include how many data sources must cross the gap, whether a Kubernetes cluster and object storage must be built from scratch, how many approval steps the security authority requires for each transfer, and whether the team has already staged an offline package set. Ask the partner to tie each phase to a written milestone and to list which steps depend on your approvals or hardware delivery. Treat any single duration offered before a scope review as a placeholder.

What does our team need to provide before work starts?

Expect to provide network diagrams for the isolated enclave, an inventory of data sources and their owners, the required retention and access rules, and named contacts for security, change approval and the data owners. You will also need access to the physical or logical environment, an agreed process for moving software packages and images across the boundary, and a decision on which identity source the platform will use. Confirm early who approves imports and who signs off on evidence, because those reviews often set the pace more than the build itself.

Can our internal staff run the lakehouse after handover?

Yes, if the handover is planned that way from the start, but it depends on how many people you assign and what skills they already have. Operating the stack requires people who can manage Kubernetes or the chosen deployment method, object storage, the table catalog, query engines and access policies, plus a process for importing upgrades. Ask the partner for runbooks, architecture records, restore tests and a training plan, and agree in writing which tasks the partner performs under a retainer after handover. Your team should also rehearse an offline upgrade before it is needed in production.

Does the lakehouse work without internet access?

The core components can be deployed without internet access, but only when every container image, package, model artifact, documentation set and rule update is staged in advance on the isolated side. Kubernetes supports offline cluster builds by pulling control plane images beforehand and pointing installation at a private registry. Trino and Iceberg run on your own storage and catalog, so they need no external cloud identity once authentication is configured for your directory. Ask the partner to list every external call each component makes, including telemetry, license checks and update checks, and to show how each is disabled or proxied.

Do paid editions or license terms limit an air-gapped deployment?

Yes, some vendors reserve air-gapped deployment for paid tiers, so the open-source option you select must be checked against the feature you need. Metabase offers air-gapped deployment only on its Enterprise plan according to its pricing page, and Mattermost lists it under Enterprise Advanced. Confirm the license of each component, including any enterprise directories or proprietary modules, before you design around them. Your legal and procurement teams should review the written scope and the license inventory together so there are no surprises after the build.

What does an implementation partner hand over as compliance evidence?

A partner can prepare configuration records, access control policies, logging settings, backup and restore evidence, and a register of imported software with integrity checks, which your assessors can review. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports, attestations or certifications, which remain the role of independent auditors and certification bodies. Ask the partner to map each evidence item to the control requirements your regulator or framework specifies, and to confirm who maintains the evidence after handover.

Questions to ask a partner before signing an air-gapped scope

Start with how artifacts cross the gap, because every package, container image, model file and rule set that enters the isolated network is a supply chain decision. Ask the partner to describe the transfer procedure in writing: who builds the bundle, how hashes and signatures are recorded, and where the manifest is checked before import. Ask how the Kubernetes control plane images are pulled and pointed at a private registry, how Trino, Spark and Iceberg dependencies are mirrored, and which step produces the evidence that supports NERC CIP-013-2 style verification of software integrity and authenticity. Ask whether a second person reviews each import. A partner who cannot show that record for the last transfer has not run one.

Next, ask how identity works with no cloud provider in the path. Trino supports password file, LDAP, Kerberos, certificate and JWT authentication, and access control can be file-based, Apache Ranger or Open Policy Agent. Ask which of these the partner would use, where the directory service lives, how service accounts for the catalog and the object store are issued and rotated, and how a revoked user is removed from the engine, the catalog and the storage layer. Also ask how multi-factor authentication is provided for administrative access, since the answer for the Ceph Object Gateway differs from the answer for the query layer. Confirm each path with your own identity team rather than accepting a general statement.

Then ask how upgrades are tested and who keeps the offline runbook current. A safe upgrade plan moves a new release through a staging copy of the lakehouse first, using the same mirrored artifacts and the same catalog. Ask how the partner validates Iceberg table format compatibility and Trino fault-tolerant execution settings before a version change, and what the rollback path is. Ask who owns the runbook, how often it is exercised, and what happens when the partner's engineers are unavailable. Finally, ask what they do when a vendor places its air-gapped feature in a paid edition, as Metabase, Mattermost and Rocket.Chat do for air-gapped deployment. The right answer names the licence or edition decision, the cost owner, and the open-source alternative that remains usable. Get that decision in writing before the scope is signed.

How DeployOpen plans, builds and supports this lakehouse

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own air-gapped networks. For an air-gapped lakehouse, that means agreeing the boundary, the import path and the evidence trail in writing before any component is moved across it.

How we scope

Map which analytics workloads, source systems and users sit inside the isolated network and which must never cross it.
Agree the table format, catalog, query engines and object storage in writing, with versions and licensing confirmed for each component.
Define the software import process, including how packages, images and rules are verified before they enter the enclave.

How we build

Deploy on-premises object storage, the catalog and access control without cloud identity, using the client's own directory and policy tools.
Stage container images, Helm charts and language dependencies in a private registry and mirror, so the cluster builds without internet access.
Test the lakehouse end to end inside the isolated network, covering time travel, schema changes, access rules and failure recovery.

How we hand over

Deliver runbooks for the import workflow, upgrade rehearsals across the gap and table maintenance on Iceberg tables.
Document configuration, access control decisions and the evidence each control produces for auditors to review.
Train the operators who will run the platform so the client team can perform imports, restores and routine queries unaided.

How we support

Support is available as a retainer covering patching, upgrade planning and changes to the import pipeline and access rules.
Run restore tests and backup verification on a schedule agreed in the written scope.
Re-test the platform after significant changes, with findings and remediation tracked in the client's own records.