Why an open source data lakehouse for SaaS companies changes the analytics decision
Consider a SaaS platform team whose product analytics, customer-facing dashboards and internal reporting all run on a consumption-priced cloud warehouse. Query volume grows with every tenant and every dashboard refresh, the bill follows usage, and the team has limited control over how tenant data is separated or how it could be moved later. A self-hosted lakehouse on Apache Iceberg tables, queried with Trino, can serve product and embedded analytics on infrastructure the company controls. Iceberg gives the team one table format that several engines can read and write at the same time, so the choice of query engine does not lock the data into one vendor. The short answer on platforms is a set of separate roles. Iceberg tables hold the data in object storage the company runs. Trino handles interactive SQL across those tables and can use a catalog such as an Iceberg REST catalog or Hive Metastore. Spark or ClickHouse is added for specific workloads, such as large batch transformations or sub-second customer-facing queries. Apache Superset or Metabase serves embedded dashboards, and each has its own licensing terms that must be checked before a dashboard reaches a customer. DeployOpen plans, builds and supports this stack inside the client's own cloud accounts. It prepares systems and evidence for compliance assessments, but it does not issue audit reports or attestations, and that remains the role of independent auditors.
Before the team commits, it should confirm three things in its own environment: which tenant isolation model it needs (shared tables with row-level policies, or separate tables or catalogs per tenant), which latency the customer-facing dashboards must meet, and who will run table maintenance, backups and upgrades after handover. Those answers decide whether Trino alone is enough or whether a second engine is justified. The team should also ask how the current warehouse exports data and whether the exported format can be read directly by an Iceberg engine, because that determines how difficult an exit would be. The rest of this page works through each of these decisions.
Which requirements a SaaS lakehouse has to meet and what they mean for the build
This table maps the requirements a SaaS company faces when it builds a lakehouse to the source that creates each one and what that means for the architecture. Read each row as a question to answer before design starts, and use the confirm column to decide which items need legal, audit or vendor input. Most rows affect catalog, access control and evidence design more than raw compute.
| Requirement | Source | What it means for the build |
|---|---|---|
| Per-tenant data separation | Trino security docs: file-based, Apache Ranger or Open Policy Agent access control; Iceberg catalog design | Decide whether tenants share tables with row-level policies or get separate schemas or buckets. Confirm that the chosen query engine enforces tenant filters at the access control layer, not only in application code. |
| Mixed batch and real-time reads on the same tables | Apache Iceberg core features; Trino Iceberg connector supports INSERT, UPDATE, DELETE and MERGE | One table format can serve dashboards and batch jobs, but freshness depends on the ingestion path. Confirm the write pattern and commit frequency with your streaming and engine teams before sizing. |
| Cost control against consumption pricing | Trino deployment docs (coordinator plus workers); Trino fault-tolerant execution needs object storage spooling | Cost shifts from per-query billing to compute nodes, storage and staff. Ask for a model that shows how query volume and concurrency change node count, and confirm where fault-tolerant spooling data lands. |
| Feeding embedded analytics | Trino and Iceberg read path; Apache Superset and Metabase licensing sources in the pillar | Customer-facing queries need predictable latency and per-tenant isolation. Confirm with counsel which embedded BI license applies before choosing the visualization layer. |
| SOC 2 Trust Services Criteria for customer assurance | AICPA 2017 Trust Services Criteria; SOC 2 Type 1 and Type 2 report definitions | Common criteria CC1 to CC9 apply to every SOC 2 report. Build access logs, change records and backup evidence from the start. Confirm with your auditor whether Type 1 or Type 2 is requested and which categories are in scope. |
| NIS2 if the company is a cloud or managed service provider in scope | NIS2 Annex I; Commission Implementing Regulation (EU) 2024/2690 | If in scope, the platform is a high-criticality service with security measures set by the implementing regulation. Confirm your sector classification and national transposition with legal counsel before scoping controls. |
| GDPR Article 28(3) processor terms when processing customer personal data | GDPR Article 28(3) and Article 32 | Your contracts must bind you to documented instructions and Article 32 security measures. Confirm that the lakehouse, backups and any deletion workflow are covered in the data processing agreement and sub-processor list. |
| EU Data Act switching rights | Data Act Articles 23, 25 and 50; Article 29 on switching charges (charges barred from 12 January 2027) | Open table formats and self-hosted storage help customers switch to another provider or on-premises infrastructure. Confirm which switching obligations apply to your service and whether your export tooling supports a documented exit path. |
| AI Act Article 12 and 19 logging if shipping high-risk AI features | EU AI Act Article 12(1) on automatic event logging; Article 19(1) on keeping logs at least six months | If an AI feature is classed as high-risk, the lakehouse may hold the event logs. Confirm the risk classification with counsel, then design retention for at least six months, unless other law requires longer. |
Which open source platforms fit a SaaS lakehouse and why
A SaaS lakehouse is a stack of separate choices: a table format, a query engine, a catalog, orchestration, and a layer that shows results to customers. Each component carries its own license and operating model, so the decision is less about a single product and more about how these pieces fit together and who runs them.
Apache Iceberg as the table format
Iceberg is the open table layer that lets several engines read and write the same tables. Its core features include schema evolution without rewriting tables, hidden partitioning, time travel to exact snapshots and rollback to an earlier state. For a SaaS product, these features make per-tenant tables and historical product metrics easier to manage. Check how your tenant layout maps to namespaces and tables, and how snapshot expiry will interact with deletion requirements. Apache License 2.0 applies to the format itself.
Trino for interactive and batch SQL
Trino is a distributed SQL engine under Apache-2.0 that queries Iceberg, Delta Lake, ClickHouse, PostgreSQL and other sources through connectors. It does not store data itself, so a Trino lakehouse needs separate object storage and a table catalog such as an Iceberg REST catalog or Hive Metastore. For large batch jobs, the TASK retry policy of fault-tolerant execution is the setting to test, since it needs an exchange manager spooling to object storage. Confirm this on your own cluster before relying on it.
Spark or ClickHouse for specific workloads
Spark suits heavy transformation, backfills and machine learning feature pipelines that run for long periods on the same Iceberg tables. ClickHouse suits high-concurrency, low-latency customer-facing queries where dashboards must answer in seconds. Many SaaS teams run both, which doubles the operational surface. Decide which workload owns which copy of the data, and confirm that the chosen engines can write the table format you have standardized on before committing.
Airflow and Dagster for orchestration
Apache Airflow, licensed under Apache-2.0, schedules and monitors pipelines as code and is widely deployed. Dagster, also Apache-2.0, models pipelines around software-defined assets, which suits teams that want lineage and freshness tracked per table. Either tool needs its own metadata database, workers and upgrade plan. Pick one per platform team rather than mixing both without a reason, and confirm how each handles retries for tenant-scoped jobs.
Airbyte and its license caveat
Airbyte is a common choice for loading SaaS sources such as billing, CRM and product event systems. The Airbyte platform repository is licensed under the Elastic License 2.0, which restricts providing the software to third parties as a hosted or managed service. If your product exposes ingestion to customers or resells it as part of your offering, legal review is required before adoption. For internal ingestion, confirm the license terms for your use case against the current license text.
Apache Superset for embedded analytics
Superset is an Apache-2.0 visualization platform that many SaaS teams evaluate for embedded dashboards inside a product. Its embedding and row-level security features must be tested against your tenant model, because a dashboard that leaks rows across tenants is a serious incident. Confirm how authentication passes tenant identity from your application to Superset, and load-test concurrency before choosing it for customer-facing use.
Metabase and its AGPL boundary
Metabase is AGPL licensed outside its enterprise directory, which uses the Metabase Commercial License. AGPL obligations can reach a modified version offered over a network, and embedding features may sit in the commercial part. Before you embed Metabase in a customer-facing product, confirm which features you use are in the AGPL portion, what source you must publish, and whether the commercial license is required. Get written legal review for this decision.
DataHub or OpenMetadata for the catalog
DataHub and OpenMetadata are both Apache-2.0 metadata platforms for discovery, lineage, ownership and governance across the lakehouse. They help answer which tenant data a table holds, where a customer's records were copied and who owns each dataset. Both need connectors to your catalog, query engines and orchestrator, and both add a service to run. Ask each candidate how it records lineage from Trino and Spark jobs before choosing.
Per-tenant separation patterns and where each one breaks down
A SaaS lakehouse has to keep one tenant's rows, files and audit trail apart from another's while still running shared pipelines and queries. The patterns below trade operational overhead against catalog load and isolation strength, and each one has a point where it stops scaling.
Table per tenant
Each tenant gets its own Iceberg tables, so isolation is enforced by object and catalog boundaries rather than by query rules. The cost appears in operations: every schema change, compaction job and snapshot expiry runs once per tenant, and a catalog holding thousands of tables slows planning and listing. Ask how table creation, schema migration and maintenance are automated across all tenants, and how the catalog performs at your expected tenant count. Each tenant's snapshot history is also a separate audit trail to retain and export.
Shared tables with tenant_id
All tenants share one table per dataset, partitioned or clustered on a tenant_id column, which keeps the catalog small and makes cross-tenant product analytics simple. Isolation then depends on access control applied at query time, so a missing filter or a misconfigured view can expose another tenant's rows. Confirm how Trino enforces row filters and column masks in your chosen version, and test them with a tenant-scoped service identity. Audit logs must record which tenant identity read which rows, not only which table was queried.
Enforced row-level access in Trino
Trino supports file-based access control, Apache Ranger and Open Policy Agent, and these can express row filters and column masks on shared tables. Policy evaluation adds latency and a dependency on the policy service, and policies written against one table layout must be re-tested after each schema change. Confirm the exact row filter and column mask behavior, supported policy syntax and decision logging in the Trino version you deploy before relying on it for tenant isolation. Test with real query shapes, including joins and aggregations.
Separate catalogs or buckets for large tenants
Large or regulated tenants can get a dedicated catalog and object storage bucket, which isolates their data, keys and retention rules and lets noisy workloads run apart. The trade-off is a growing set of catalogs and buckets to provision, monitor and back up, and cross-tenant queries must union across catalogs. Use it for a short list of tenants that justify the cost, not as the default. Keep the audit trail per bucket and per catalog, and confirm that Trino can address several Iceberg catalogs in the configuration you plan to run.
How a SaaS team moves analytics onto the lakehouse in phases
The sequence moves product and customer analytics from a consumption-priced warehouse onto open table formats in stages, with each phase producing a document, a tested control or a reconciled dataset that the next phase depends on. Tenant isolation is designed and tested before any customer data moves.
Define requirements and the tenancy model
The team agrees which analytics workloads move first, the data volumes and freshness each one needs, and how tenants are separated: one table per tenant, shared tables with tenant keys, or a hybrid. DeployOpen works with the SaaS engineering, data and security leads to write the requirements and tenancy design document, covering who owns each dataset and how a tenant's data is located and deleted. The step is done when the design is approved in writing and the open questions have named owners.
Build the landing zone and object storage
The platform team provisions self-hosted object storage in the organization's own cloud account or data center, with separate buckets or prefixes for raw, curated and tenant-scoped zones, encryption keys, network paths and backup targets. Identity for services and engineers is wired to the company's existing directory. The output is a landing zone baseline with documented access roles and a tested restore of a sample object, so the storage layer can be rebuilt if the team needs it.
Decide the catalog and table format
The team selects Apache Iceberg or Delta Lake as the table format, and a catalog that the chosen query engines can use, such as an Iceberg REST catalog, Hive Metastore or AWS Glue where that fits the environment. Engine choices for Trino, Spark or ClickHouse are tested against sample tables to confirm they read and write the same tables. The output is a catalog and format decision record, with a proof that two engines query one table and a written list of features the team must confirm with each vendor.
Ingest the first pipeline and run it in parallel
One high-value product event or customer table is ingested through an orchestrated pipeline, for example on Airflow or Dagster, and loaded into the lakehouse while the existing warehouse continues to serve reports. Each run is compared with the warehouse for row counts, key totals and null rates by tenant. Discrepancies are traced to source, transformation or load logic. The step closes when reconciled row counts hold across an agreed number of runs that the team sets with the business owners.
Pilot embedded analytics with tenant isolation tests
A small group of customers or internal users is given a dashboard or embedded view built on the lakehouse, with the BI layer such as Apache Superset or Metabase connected through row-level security tied to the tenant identifier. The security team runs tests that attempt cross-tenant reads through every path: the BI layer, direct engine access, API keys and exports. The output is a tested access policy set and a signed test report. Any failed case blocks the pilot from widening. Licensing for embedding is confirmed before customers see the product.
Cut over workloads and decommission the old warehouse
Workloads move in the order agreed in the first step, each with its own reconciliation sign-off, rollback path and owner acceptance. Consumers are repointed, scheduled jobs are retired from the old warehouse, and cost is monitored against the baseline recorded earlier. Before the warehouse contract ends, the team checks what must be exported and confirms the terms for the exit. The output is a decommissioning checklist, a documented exit path showing how data and metadata would move to another provider or on-premises infrastructure, and a final reconciliation archive.
Evidence customers and auditors ask for and which component produces it
This table maps the evidence SaaS customers, SOC 2 auditors and AI Act reviewers typically request to the lakehouse component that produces it. Read each row as a question to answer before the build starts, since most evidence gaps come from logs and retention settings that were never switched on. DeployOpen prepares the systems, controls and evidence for assessment, but independent auditors issue any SOC 2 report.
| Evidence | Produced by | Notes |
|---|---|---|
| SOC 2 access and change control evidence (CC1 to CC9 as baseline) | Identity provider, Trino access control, catalog and infrastructure-as-code change history | Security common criteria CC1 to CC9 apply to every SOC 2 examination. Confirm which Trust Services categories the report covers before collecting evidence. |
| Type 1 system description and control design | Documented architecture, data flow diagrams and control design records as of one date | A Type 1 report covers design at a point in time. Prepare the system description early, since it is the baseline a Type 2 period is tested against. |
| Type 2 operating effectiveness over a period | Continuous evidence of controls operating across the observation window: access reviews, change tickets, alert records | Auditors test operation over the period, so evidence must be collected continuously rather than assembled at the end. Confirm the observation window with the auditor. |
| Trino and catalog access logs | Trino query event logs, Iceberg REST or Hive Metastore audit logs, forwarded to a central log store | Confirm which query and catalog events each component records and how long they are retained. Logs should show which tenant schema each query touched. |
| Snapshot history and time travel records | Iceberg snapshot metadata and table history, queried through Trino or Spark | Snapshots show table state at each commit and support rollback. Expiry settings determine how long this history exists, so document them as a control. |
| Tenant isolation test results | Scheduled cross-tenant access tests run against Trino roles, views and catalog permissions | Tests should attempt reads and writes across tenant boundaries and keep the results. Re-run them after each change to access policy. |
| Backup and restore test records | Object storage replication or snapshots, catalog metadata backups, documented restore runs | A backup that has not been restored is not evidence of recovery. Keep dated restore test records with the time taken and the data verified. |
| Data retention and deletion records | Retention schedules, snapshot expiry jobs, orphan file cleanup logs and row-level delete history | Old snapshots can still hold deleted rows until they expire. Show both the deletion request and the cleanup that removed the underlying files. |
| AI Act logging for AI features | Application and model gateway logs of inputs, outputs and events, stored with a defined retention period | Article 12(1) requires high-risk AI systems to allow automatic event logging over their lifetime, and Article 19(1) requires keeping logs for at least six months unless other law applies. Confirm whether your AI features fall within high-risk categories. |
What drives cost and how the Data Act changes switching
Cost on a self-hosted lakehouse splits into recurring infrastructure and one-time or recurring people work, and the two respond to different drivers. Object storage for Iceberg data files is paid per stored volume, so retention length and how many snapshots you keep drive it up. Compute is paid for the Trino coordinator and worker nodes, which you size for query volume and concurrency. A dedicated coordinator becomes more relevant as the cluster grows, according to Trino's deployment documentation. Engineering covers the initial build, the catalog, access control and tenant isolation, and the ongoing work of table maintenance such as compaction and snapshot expiry. Ask any partner to split its quote into these lines, state who pays for each (your cloud account, the partner, or a vendor), and mark which lines are one-time and which recur. Confirm whether the quote includes an exchange manager on object storage if you plan to run large batch queries with fault-tolerant execution, since that adds storage and configuration work.
The Data Act changes the commercial risk of lock-in. Under Regulation (EU) 2023/2854, which has applied since 12 September 2025, data processing service providers must let customers switch to another provider or to on-premises infrastructure. The provider can require up to a two-month notice period, followed by a 30-day transitional period. From 12 January 2027, providers may not impose any switching charges, and until then reduced charges may not exceed costs directly linked to switching. For a SaaS company this matters in two ways: your own customers may ask how they can leave your analytics services, and your cloud warehouse contract may be harder to exit cheaply. Keeping tables in Iceberg on storage you control means a move between catalogs or engines is an engineering task rather than a data export negotiation. Check with counsel how the Data Act applies to your own service contracts, since its scope depends on the service you provide.
Before you sign, ask the partner to model three scenarios with your real query and tenant volumes: steady batch load, peak interactive load and a switch to another engine. Record the expected concurrency limits, the retention schedule for snapshots and the procedure for moving tables to another catalog. Confirm whether the partner can export Iceberg metadata and data files without vendor tooling, and whether the Iceberg tables remain readable by Spark or Trino after a change of catalog. Those answers tell you the actual cost of switching, which the Data Act now treats as a cost you should be able to keep low.
What SaaS companies ask about self-hosted lakehouse effort and cost
How does a self-hosted lakehouse cost compare with a consumption-priced cloud warehouse?
A self-hosted lakehouse replaces per-query or per-credit charges with costs you can see: object storage, compute nodes for the query engines, catalog and orchestration services, engineering time and support. The cloud warehouse bill rises with query volume, concurrency and data scanned, so the comparison depends on your workload shape. Build a model with three inputs: steady query load, peak concurrency for embedded analytics, and how much data is rescanned by dashboards. Ask any partner for a cost model that separates one-time build work from recurring infrastructure, staffing and support, and that shows how each line changes when query volume doubles. No cost figures are given here because they depend on your environment.
How long does it take to build a self-hosted lakehouse for a SaaS product?
The duration depends on scope, and the scope is agreed in writing before work starts. The main variables are the number of source systems, whether tenant isolation must be built into the table layout, how many query engines you run, whether real-time ingestion is included, and the state of your existing identity and networking. A build limited to batch ingestion into one catalog with a single engine is a very different project from one that adds per-tenant access control and embedded analytics. Ask for a phased plan with explicit deliverables and acceptance tests for each phase, not a single end date.
What does our internal team need to provide for the project?
Your team needs to provide decisions and access more than headcount during the build. Expect to supply a named owner for the data model, the source system contacts who can explain extracts and change events, the cloud or data center accounts where the platform will run, identity provider details for single sign-on, and a sign-off process for security controls. Some organizations also assign an engineer to shadow the build so knowledge stays in-house. Confirm these responsibilities in the statement of work, including who approves the tenant model and who signs off on the migration from the current warehouse.
Can our own engineers run a self-hosted lakehouse after handover?
Yes, if the team has skills in several areas and the runbooks are complete. Operating the platform involves Kubernetes or virtual machine operations, object storage administration, the catalog service, query engine tuning, scheduled table maintenance such as compaction and snapshot expiry, backup and restore testing, and upgrades. Trino, for example, runs as a coordinator and workers, and the Iceberg connector depends on a catalog such as a REST catalog or Hive Metastore, so those components need owners. Assess your team against that list before deciding between full in-house operation and a retainer for upgrades and support, which DeployOpen offers.
Does Apache Iceberg make it easier to leave a platform under the EU Data Act?
Iceberg supports portability, but it does not satisfy the Data Act by itself. Regulation (EU) 2023/2854 requires data processing service providers to let customers switch to another provider or to on-premises infrastructure, with a maximum two-month notice period and a 30-day transitional period. From 12 January 2027, providers may not impose switching charges. Iceberg helps because the same tables can be read by engines such as Spark, Trino and Flink, and the format is Apache-2.0 licensed. Your exit still depends on the catalog, access policies, and whether your contracts and any managed services preserve the data in open formats. Have counsel review the provider terms against Articles 23, 25, 29 and 50.
Does DeployOpen issue SOC 2 reports for our lakehouse?
No. DeployOpen does not issue SOC 2 reports, attestations or certifications. A SOC 2 report comes from an independent CPA firm after an examination, and it covers a system description and controls against the AICPA 2017 Trust Services Criteria. What DeployOpen can do is prepare the systems, controls and evidence the auditor will examine, such as access reviews, logging, change records, backup tests and configuration baselines for the lakehouse. Ask the auditor early which Trust Services Categories are in scope and whether a Type 1 or Type 2 report fits your timeline.
What operational risks should we check before choosing a self-hosted lakehouse?
The main risks are small files and unmanaged snapshots that slow queries and inflate storage, uncontrolled access to tenant data, weak backup coverage for the catalog metadata as well as the data files, and dependency on a few engineers. Check whether the design includes scheduled table maintenance, tenant-level access policies enforced in the query layer, tested restore procedures, and monitoring of query queues. Trino's fault-tolerant execution is off by default and needs an exchange manager for batch retries, so confirm whether your batch jobs need it. Ask the partner to show these controls in the design before you sign.
What should we ask a partner about cost and scope before signing?
Ask for a written scope that lists sources, tables, tenants, engines, access controls, environments and acceptance tests. Then ask for a quote that separates one-time work (design, build, migration, testing, documentation) from recurring costs (infrastructure, licenses if any, support, upgrades and table maintenance). Ask which assumptions the quote depends on, such as data volumes and concurrency, and what changes the price if those assumptions move. DeployOpen does not publish a price list, and a quote follows from the written scope.
How DeployOpen scopes, builds and supports a SaaS lakehouse
DeployOpen is a Chainscore Labs company that plans, implements, hardens, backs up, documents and supports self-hosted open source platforms inside the client's own cloud accounts, data centers or isolated networks. For a SaaS lakehouse, that means the Iceberg tables, the object storage, the catalog, the Trino or Spark engines and the access layer all run in accounts the company controls, and DeployOpen works inside those accounts. Most engagements start with a written, fixed scope that names the table format, catalog choice, tenancy model, engines, freshness targets, backup approach and handover deliverables. A quote follows from that written scope, so the first decision for your team is how much of the tenancy and freshness design you can specify before any build begins.
Scoping should also settle who runs the platform after go-live. Ongoing work such as engine and catalog upgrades, Iceberg table maintenance checks, backup restore tests and dependency patching can be covered by a retainer, or handed to your own platform team with a documented runbook. Ask the partner to show which of these tasks it will own, which it will teach, and how restore tests will be recorded. Because Trino's own deployment guidance expects a Java runtime and a coordinator plus workers, the upgrade calendar for those components should be agreed in the scope, not improvised after launch.
Security testing covers the data and API layers that a SaaS lakehouse exposes: the query endpoints, the tenant-scoping logic in row-level and column-level policies, the authentication path to Trino and the API gateway in front of any embedded analytics. Testing is a separate, scoped engagement, so it can be run against the design before production data arrives. Compliance readiness is a different service. DeployOpen prepares systems, controls and evidence, such as access reviews, change records and per-tenant data separation tests, that your auditor can examine for a SOC 2 or ISO/IEC 27001 assessment. It does not issue audit reports, attestations or certifications. Those come from independent auditors and certification bodies, and your evidence plan should name them early.
