How to use this open source data lakehouse requirements checklist
Your team has outgrown a proprietary cloud warehouse, or wants to avoid being locked into one, and now needs to agree what the new platform must do before anyone installs software. This checklist covers the requirements to agree before building a lakehouse: the table format, the catalog, the query engines, object storage, access control and lineage, data volumes and freshness, backups, and the team that will run it. Use it as a scope agreement, not a shopping list. For each category, agree the scope in writing, name one owner who answers for it, and record how the answer will be verified, such as a test query, a restore exercise, a documented catalog policy or a sample audit log. Any item left unanswered is a scope risk, and it should appear in the project plan with a date for closure rather than being quietly deferred to the build phase.
The checklist is vendor-neutral. It names open formats and engines because those are the options most teams evaluate, but it does not assume that any one of them is the right choice for your estate. Where a decision depends on a feature of a specific product, the checklist asks you to confirm that feature in the version you plan to deploy, because table format and engine capabilities change between releases. Start with the questions that are hardest to reverse: which table format the organization standardizes on, which catalog holds the table metadata, and who can grant and revoke access to raw and curated zones. Get those decisions recorded before you size compute or pick a dashboard tool. Treat the team question as a requirement too. A lakehouse that no one is staffed to patch, back up and tune will drift into the same operational problems it was meant to solve.
Functional requirements every open source data lakehouse must answer
This table lists the functional questions to settle before choosing table formats, catalogs and query engines for a self-hosted lakehouse. Read each row as a decision: the first column names the requirement, the middle column explains why it matters, and the last column says how to verify it before build. Engine capabilities change between releases, so confirm each one against the current vendor documentation.
| Requirement | Why it matters | How to verify |
|---|---|---|
| Table format | Apache Iceberg is Apache-2.0 licensed and lets Spark, Trino, Flink, Presto, Hive and Impala work on the same tables. Delta Lake and Apache Hudi are alternatives that may fit an existing estate better. | Confirm which engines must read and write the tables today and in the next few years, then check each format's engine support in its own documentation. |
| Catalog choice | The catalog holds table metadata and decides which engines see the same tables. Options include Hive Metastore, AWS Glue, a REST catalog, Nessie and Snowflake, and each has different operating and identity needs. | Ask the partner to name the catalog, its high availability design, backup method and which engines connect to it, then test a table write and read from each engine. |
| Query engines | Trino is a distributed SQL engine that queries data in place through connectors. Spark suits large batch and machine learning jobs, and ClickHouse suits low-latency analytical queries. Most lakehouses run more than one. | Map each workload to an engine, then run representative queries on a test cluster sized to expected concurrency and record response times. |
| Object storage | Table data sits in self-hosted object storage, which must be sized, replicated and backed up separately from the query engines. Trino needs separate storage and a catalog because it does not hold data itself. | Confirm capacity, replication, encryption at rest and bucket-level access controls with the storage team, and verify that every engine can reach the bucket. |
| Workloads and concurrency | Interactive dashboards, scheduled batch jobs and ad hoc analyst queries compete for the same compute. Concurrency limits decide whether one cluster or several are needed. | Write down peak concurrent users, longest acceptable query times and batch windows, then load test with that profile before sign-off. |
| Time travel and rollback | Iceberg supports time travel to exact historical snapshots and rollback to an earlier table state. Auditors and analysts often rely on reproducing a past report, so this needs a retention rule. | Ask for a demonstration that queries a past snapshot and rolls back a test table, and confirm the snapshot retention period in writing. |
| Schema evolution | Iceberg supports schema changes without rewriting tables, which lowers the risk of breaking downstream reports. Changes still need ownership and review. | Test adding, renaming and dropping columns on a copy of a production-like table, then check that each query engine reads the new schema. |
| Row-level changes | Trino's Iceberg connector supports INSERT, UPDATE, DELETE and MERGE on Iceberg tables. Row-level changes are needed for corrections, slowly changing data and erasure requests. | Run MERGE and DELETE tests from each engine that writes to the tables, then confirm that old snapshots and orphan files are cleaned up under the retention policy. |
Why the table format and catalog decision comes first
The table format and the catalog decide which engines can read the same tables, and changing either later means migrating data and rewriting jobs. Apache Iceberg is the format most open lakehouse designs start from. It is licensed under the Apache License 2.0, and its main repository is the Java reference implementation, with separate implementations in Python (PyIceberg), Go, Rust and C++. That breadth matters when a team needs a Python service, a Rust ingestion tool or a Spark job to write to the same tables that Trino reads. Confirm that every engine on your list has a maintained Iceberg implementation at the version you plan to run, because support for write operations and features such as row-level deletes can differ between engines.
The catalog is the second decision, and it is easy to underestimate because Trino does not hold data itself. Trino's Iceberg connector works with Hive Metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs, selected with the iceberg.catalog.type configuration key. Trino also reads and writes Parquet, ORC and Avro data files, and it needs separate object storage and a table catalog to function as a lakehouse. Choose the catalog with the engines and the warehouse in mind, not only the one currently in use. Ask whether the catalog will be reachable by every engine in the design, how it authenticates users and services, and who operates it. A catalog outage or mis-set credential stops all engines at once, so it belongs in the reference architecture and the backup plan.
Which query engine to require for which workload
Trino fits analysts and BI tools that run ad hoc SQL against Iceberg tables and other sources, such as PostgreSQL, MySQL or Snowflake, within a single query. It reads data through connectors, so the lakehouse still needs its own object storage and a table catalog, such as an Iceberg REST catalog or Hive Metastore. Set these requirements before the catalog and cluster are designed.
- Host requirements: Trino's deployment documentation calls for a 64-bit Linux system and a 64-bit Java 25 runtime, minimum 25.0.1. Confirm that your base images and patching process can keep to that minimum.
- Cluster shape: The documented model is one coordinator plus workers, with a dedicated coordinator recommended for larger clusters. A Kubernetes deployment is covered by the Helm guide in the documentation.
- Catalog choice: The Iceberg connector works with Hive Metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs, selected through
iceberg.catalog.type. Choose the catalog as part of this decision. - Fault tolerance: Fault-tolerant execution is off by default. The TASK retry policy suits large batch queries but needs an exchange manager spooling to S3, Azure Blob Storage, Google Cloud Storage or HDFS.
- Access control: Trino supports file-based access control, Apache Ranger and Open Policy Agent, and authentication through LDAP, OAuth 2.0, Kerberos, certificates and JWT, among others.
Confirm the number of concurrent interactive users and how many queries span sources outside the lakehouse before sizing the coordinator.
Apache Spark is the engine for large transformations, scheduled batch pipelines and feature preparation for machine learning, with Python, SQL, Scala, Java and R supported. Its support timeline and cluster model affect upgrade planning, so treat the version as a governed decision. Spark also needs surrounding tools to be a complete platform.
- Current lines: Spark 4.2.0 was released on July 14, 2026. The 3.5.x line has an extended LTS that ends in November 2027, with security fixes only in the final period.
- Support policy: Feature releases are planned quarterly from 4.3.0, feature branches are maintained for 6 months, and the last release in each major line is an LTS maintained for 18 months.
- Scala version: Spark 4 is pre-built with Scala 2.13 and has dropped Scala 2.12 support. Check that existing jobs and libraries build against it.
- Cluster managers: Spark offers its own standalone manager, Hadoop YARN and Kubernetes. The Apache Spark Kubernetes Operator, licensed under Apache-2.0, manages Spark applications on Kubernetes.
- Missing components: Spark has no notebook workspace, job scheduler UI, catalog or access control layer. Plan for JupyterHub, Apache Airflow and a table catalog, and confirm who owns each one.
Review your job count, runtime windows and Spark version history with the team that will maintain the jobs.
ClickHouse is the candidate when many users or embedded dashboards need fast responses against a predictable set of tables, often denormalized or pre-aggregated. It serves a different purpose from the Iceberg tables and it becomes a second store that must be governed. The requirement is to decide which copy is the source of truth and how it is loaded.
- Workload pattern: Dashboards and product features that run many similar queries at once differ from exploratory SQL. Measure peak concurrency and latency targets from your own query logs.
- Source of truth: Define whether the lakehouse or the analytical store owns each dataset. Document the rebuild path if the analytical copy is lost.
- Ingestion path: Confirm how data reaches the store, whether through batch jobs from Airflow or Dagster, or through streaming from Kafka. Each path has its own failure modes and monitoring needs.
- Federation: Trino's connector list includes ClickHouse, so some queries can span both stores. Test whether federated queries meet your latency expectations before relying on them.
- Licensing and support: Confirm the license, current release line, and support model with the project and with any partner you engage. Do not assume them from a general description.
Test concurrency at the expected peak with production-like data volumes before the store is added to the architecture.
Security requirements to agree before the first table is created
A lakehouse concentrates sensitive data in one place, and the engines, catalog and storage all become places where access can be granted or leaked. The cards below list the security decisions to settle with the data owners and security team before anyone creates a table. Each one names what to decide and how to verify the decision in the running platform.
Authentication for query engines
Decide how analysts, service jobs and BI tools prove who they are to Trino and the other engines. Trino documents password file, LDAP, Kerberos, OAuth 2.0, certificates and JWT, so the choice depends on the identity provider already in place and whether humans and pipelines need different methods. Verify that every interactive and automated path uses a named method, and that no shared password file account is used for production jobs.
Authorization model and policy engine
Choose how table and column permissions are expressed. Trino supports file-based access control, Apache Ranger and Open Policy Agent, and each implies a different place where rules are written, reviewed and audited. Agree who owns the policies, how row filters and column masks are handled, and how changes move from request to production. Verify with test queries from each role that denied tables and masked columns return what the policy says.
Encryption at rest and in transit
Object storage holds every Parquet file, metadata file and snapshot, so its encryption settings define the protection of the whole lakehouse. Decide which keys encrypt the buckets, who holds them, and whether TLS is required between clients, coordinators, workers, the catalog and storage. Confirm these settings with the storage vendor and the engine configuration rather than assuming defaults. Verify that unencrypted connections are refused and that key rotation has a tested procedure.
Secrets and service accounts
Pipelines, the catalog and the engines each need credentials to reach storage and source systems. Decide where these secrets live, which service account owns each job, and how long credentials last. Avoid sharing one account across environments, and keep catalog and storage keys out of notebooks and job definitions. Verify that every service account has a single purpose, a documented owner and a rotation schedule that the team has exercised.
Network segmentation
Decide which networks can reach the coordinator, workers, catalog, metastore and object storage, and which can reach users. Trino clusters with a dedicated coordinator and worker nodes create several surfaces that should not be open to the wider corporate network. Source connectors such as PostgreSQL or Snowflake also need explicit egress paths. Verify with a network scan from outside the approved zones that only the intended ports and paths respond.
Audit logging of queries and table changes
Decide which events must be recorded: who ran which query, which tables were written, merged or deleted, and which access was denied. Set where these logs are stored, how long they are kept and who can read them, since they may hold sensitive query text. Where the lakehouse supports AI workloads, the EU AI Act requires automatic event logs for high-risk systems. Verify by running a test delete and confirming that the record shows the user, table, time and statement.
Operational requirements: backups, table maintenance and the team that runs it
This table lists the operational duties a self-hosted lakehouse creates after go-live, what each one asks for, and how to verify it before you commit. Use it to agree ownership and test evidence with a partner before build starts. The main takeaway: backups and maintenance are only real if you have restored from them and rehearsed upgrades.
| Requirement | What it asks for | How to verify it |
|---|---|---|
| Catalog metadata backup | Regular backup of the catalog database or REST catalog state, plus the metadata pointers that let tables resolve to their current snapshots | Restore the catalog into a separate environment and confirm that table reads and time travel return expected results |
| Object storage backup and retention | Versioning or replication policy on the bucket holding data and metadata files, with a documented recovery point for each zone | Run a restore drill that recovers a deleted table prefix and reconciles row counts against a known checksum |
| Snapshot expiry | A scheduled job that expires old Iceberg snapshots according to a retention schedule agreed with compliance and legal owners | Review expiry logs and confirm that time travel windows match the documented policy, with exceptions recorded |
| Orphan file cleanup | Periodic removal of data and metadata files no longer referenced by any snapshot, run after snapshot expiry | Compare object storage listings with table file references in a test run before enabling deletion in production |
| Compaction and small file management | Scheduled rewrite of small data files and manifests, sized to the query engines in use and the ingestion pattern | Track file counts and query latency per table over time and confirm that compaction jobs complete within their window |
| Engine and format upgrades | A cadence for upgrading Trino, Spark and the table format, with compatibility checks for connectors and catalog clients | Rehearse each upgrade on a copy of production metadata and record rollback steps before the production change |
| Fault-tolerant execution in Trino | Fault-tolerant execution is off by default. A QUERY or TASK retry policy must be chosen, and TASK is recommended for large batch queries, which requires an exchange manager | Run a batch query under forced worker loss in staging and confirm that it completes with the chosen retry policy |
| Exchange manager spooling storage | Spooling storage on S3, Azure Blob Storage, Google Cloud Storage or HDFS, sized and secured as its own data store with lifecycle rules | Confirm the supported storage option and its access model with the Trino documentation and your storage team before design sign-off |
| Monitoring and alerting | Metrics and logs for query failures, queue times, compaction lag, catalog availability and storage errors, routed to the on-call owner | Trigger a test alert for each critical signal and confirm the on-call response within the agreed runbook |
| Incident ownership | A named owner for each layer (storage, catalog, engines, access control, pipelines) and an escalation path to the partner for out-of-hours issues | Walk through a tabletop incident per layer and confirm each owner can act without waiting for another team |
| Named roles and skills | Platform engineer for Kubernetes and storage, data engineer for table design and pipelines, security engineer for access control, and a data steward for lineage and retention | Check that each role has a named person, backup coverage, and written skills expectations agreed before handover |
Compliance evidence a lakehouse can produce
The requirements that matter most for evidence are time travel and snapshot history, lineage and access logs, and retention and deletion rules. Apache Iceberg lists time travel to exact historical snapshots and rollback to an earlier table state as core features, so a finance or reporting team can reproduce a figure as it stood on a given date. Confirm that the table format, engine and catalog you choose expose those snapshots to the reviewers who will need them, and that snapshot expiry is set to a documented schedule rather than left at a default nobody has reviewed. Without that schedule, the evidence you rely on may be removed by routine maintenance.
Lineage and access logs are the second source of evidence. Metadata platforms such as DataHub and OpenMetadata, both Apache-2.0 licensed, can record ownership and lineage across the estate, while query engines such as Trino support authentication through LDAP, OAuth 2.0, Kerberos and JWT and access control through file-based rules, Apache Ranger or Open Policy Agent. Decide early which system is the system of record for who queried what, how long those events are kept, and where they are stored. Retention rules and deletion workflows also need a written owner. Deleting a person's rows from an Iceberg table is a row-level operation, and the data remains in older snapshots until they expire and orphaned files are cleaned up, so the deletion procedure should state that sequence.
Where the lakehouse feeds AI workloads or automated decisions, check the system's classification under the EU AI Act. Article 12(1) requires high-risk AI systems to technically allow automatic recording of events over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Whether either applies depends on the use case and your legal review, so confirm it before the logging design is fixed. DeployOpen prepares controls, configurations and evidence for compliance assessments, but it does not issue audit reports or certifications; independent auditors and certification bodies make that determination.
How to agree requirements in a workshop
This sequence turns a loose lakehouse idea into a written set of decisions on data, formats, engines, access and operations, so the scope can be quoted and built against. Durations and effort depend on your estate and should come from a written scope, not from this page.
Inventory data sources and volumes
List every source system, its owner, the data types it produces, approximate row counts and daily or monthly growth, and any sources that are already in a warehouse. Data owners and the platform lead attend, and the output is a register with one line per source. You know the step is done when each source has a named owner and a volume estimate that the owner has confirmed in writing.
Set freshness targets per dataset
For each dataset, agree how stale the data may be before a report or product becomes wrong: hourly, daily, or near real time. Business owners and analytics leads decide, and platform engineering checks which targets need streaming ingestion and which can use batch jobs. The output is a freshness column in the register. Datasets that need second-level freshness should be flagged for separate design work, since they change the ingestion and query choices.
Decide table format and catalog
Choose the open table format (for example Apache Iceberg or Delta Lake) and the catalog that tracks tables across engines. Confirm which engines must read the same tables, which catalog types each engine supports, and whether any external platform must read the data. Record the reason for each choice, including portability and exit requirements. Done means a short decision record that engineering, security and the data owners have all signed.
Choose engines per workload
Map workloads to engines: SQL analytics, batch transformation, streaming, machine learning and interactive dashboards often need different tools. Trino, Spark and ClickHouse each fit different patterns, so confirm the concurrency, latency and data-size needs of each workload and the skills your team already has. Ask each candidate engine's documentation to confirm the connectors, catalog types and deployment model you need. The output is a workload-to-engine table with an owner for each row.
Define access and lineage needs
Specify who may see which tables, columns and rows, how identities are granted access (directory groups, service accounts, or both), and what must be audited. Also specify the lineage you need, such as which upstream tables feed a regulated report. Security, data governance and the compliance lead agree the rules. The step is complete when access rules are written per data zone and the lineage and audit log requirements are listed, with an owner for each.
Set backup and recovery objectives
Agree how much data loss and how much downtime each dataset and the catalog can tolerate, and where backups of object storage, catalog metadata and configuration are kept. Include how table snapshots and expiry settings interact with recovery and retention. Operations and the data owners sign off. The output is a recovery objective per dataset and a restore test plan that someone is responsible for running before go-live.
Name the run team
Identify who will operate the platform after launch: who patches and upgrades engines, maintains tables, responds to incidents, manages access requests and owns on-call. Include whether the internal team, a partner under a retainer, or a mix will do this work. The output is a responsibility matrix with named roles and skills gaps flagged. Ask any implementation partner to confirm the handover and support arrangements in writing before this step closes.
Sign off requirements in writing
Compile the register, decision records, workload table, access and lineage rules, recovery objectives and run-team matrix into one requirements document. Each accountable owner signs it, and changes after sign-off go through a named change process. The document then becomes the basis for a written, fixed scope and quote. You know it is done when every open question has an owner and a date set by that owner.
What buyers ask before they agree lakehouse requirements
What drives the cost of a self-hosted lakehouse and what should a quote include?
The cost is driven by the people, the infrastructure and the ongoing operation, and each one is paid for differently. Infrastructure covers object storage capacity, the compute nodes for query engines and any Spark or batch clusters, and backup copies. Engineering covers the one-time build: table format decisions, catalog setup, ingestion, access control and migration of existing reports. Recurring costs include table maintenance such as compaction and snapshot expiry, upgrades, monitoring and support. Query volume and concurrency change compute needs more than storage does. Ask any partner for a quote that separates one-time work from recurring operations, lists the environments included (development, test, production), names who runs upgrades, and states the handover deliverables in writing. DeployOpen does not publish a price list, and a quote follows from the agreed written scope.
How long does agreeing lakehouse requirements take?
The duration depends on how many source systems, data domains and regulatory zones are in scope, and on how many owners must sign off on access rules and retention. A single department with one or two sources and clear owners moves much faster than an enterprise with overlapping business units and unclear data ownership. The work is mostly decision-making: confirming the table format, the catalog, the query engines, the storage target, the freshness each consumer needs, and who approves access. Prepare inventories of sources, expected data volumes, query patterns and retention obligations before the first workshop, because those inputs shorten the cycle. No fixed duration applies to every organization, so ask a partner to propose a timeline tied to the scope and the decisions still open.
Can an internal team build a lakehouse without a partner?
Yes, an internal team can build one if it has the skills and time, but the cost of getting it wrong is usually in operations rather than in the first deployment. A workable team needs people who understand distributed query engines, object storage, identity integration, data modeling and table maintenance, and someone accountable for on-call operations after go-live. Spark, for example, does not include a notebook workspace, job scheduler UI, catalog or access control layer of its own, so the team must select and integrate those parts. Assess honestly which of these skills already exist, which would be hired and which would be bought in for a fixed-scope build. A partner can cover the gaps for a defined period, while the internal team keeps ownership of data definitions and business rules.
What does the work need from our staff?
The work needs named people from data engineering, platform or infrastructure, security, and the business domains that own the data. Platform staff provide cloud or data center access, network paths, storage and identity integration. Security staff set authentication, authorization and logging requirements. Data owners confirm definitions, sensitivity classes and retention periods, and they approve who may see which rows. Engineers review table designs, pipeline code and query performance tests. Ask for a RACI matrix at the start, with each role's expected time commitment stated in the written scope, so that delays in one team do not stall the others.
Which catalog should we choose for an Iceberg lakehouse?
The catalog choice depends on which engines you run and where your identity and metadata systems already live. Trino's Iceberg connector supports Hive Metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs, selected through the iceberg.catalog.type setting. Confirm that every engine you plan to use can talk to the catalog you pick, including Spark and any analytics tools. Evaluate how the catalog handles access control, auditing, multi-engine concurrency and backup of its metadata. A catalog is a single point of coordination, so the requirement is to decide who operates it and how it is restored. Test each candidate against your own engines and workloads before you commit.
Is Iceberg or Delta Lake safer for portability?
Apache Iceberg is licensed under the Apache License 2.0, and the format is supported by several engines, including Spark, Trino, Flink, Presto, Hive and Impala, which lets the same tables serve multiple engines at the same time. Delta Lake choices should be assessed against the same portability test: which engines you need to read and write the tables, and whether they support the features your teams depend on. Microsoft's OneLake documentation notes that it can expose Delta tables to external Iceberg readers through metadata virtualization, which shows that interoperability is improving but still depends on the platform. Write down the engines you must support for the next several years, then verify read and write behavior with a test table in your own environment. Portability also depends on keeping data in your own object storage and on exporting metadata you control.
How do we verify access control before go-live?
Verify access control by testing it with named accounts and documented expected results before any production data is loaded. Trino documents authentication through password file, LDAP, OAuth 2.0, Kerberos, certificates and JWT, and access control that is file-based, through Apache Ranger or through Open Policy Agent. Build a test matrix that lists each role, each table or column, and each row-level rule, then run queries as each identity from every engine that reads the data. Confirm that denied queries fail, that masked columns return masked values, and that the audit trail records who accessed what. Check the catalog and the object storage permissions separately, because a user who can reach files directly may bypass engine rules. Keep the test results as evidence for your security and compliance reviews.
What should the requirements document cover for lineage and backups?
The requirements document should state which lineage, backup and recovery outcomes the platform must support, and how each will be tested. For lineage, list the systems that must be connected, such as sources, pipelines and dashboards, and name the metadata catalog you expect to use. DataHub and OpenMetadata are both Apache-2.0 metadata platforms that can serve this role, and their fit should be confirmed against your sources. For backups, separate table data in object storage, catalog metadata and pipeline definitions, since each needs its own recovery point and restore test. Iceberg supports time travel to exact historical snapshots and rollback to an earlier table state, but snapshots are not backups on their own, so state how long snapshots are retained and who approves expiry. Ask for a restore test as a deliverable.
Questions that separate lakehouse platforms and partners
Ask each vendor and implementation partner how they design the catalog and governance layer before they discuss engines. The catalog decides which engines can read and write the same tables, and it is the single component most likely to lock you in. Ask which catalog they recommend for Apache Iceberg tables (a Hive Metastore, AWS Glue, a REST catalog, Nessie or a Snowflake-hosted catalog, as Trino's Iceberg connector supports them), how table-level and column-level permissions are enforced across Trino, Spark and any other engine, and whether access control runs through file-based rules, Apache Ranger or Open Policy Agent. Ask who holds the catalog credentials and where the audit trail of grants and queries is stored. A partner who cannot answer these in concrete terms has not designed the governance layer yet.
Ask how table maintenance will run after go-live, because open table formats do not maintain themselves. Ask for the schedule and owner of snapshot expiration, orphan file cleanup, compaction and manifest rewrites, and what happens to time travel and rollback windows when snapshots expire. Ask how they will test fault-tolerant execution if large batch jobs need it, since Trino's TASK retry policy requires an exchange manager spooling to object storage. On performance, ask for the test method, the data volumes and concurrency used, and the query set they will benchmark against your real workloads. Ask for a written migration and handover plan that names what is transferred, what the client team must operate, and which runbooks, dashboards and access reviews will exist at the end. Finally, ask what they will not do, such as operating the platform after handover without a retainer, supporting engine versions that have left their maintenance window, or taking responsibility for your compliance conclusions. Refusals on these points are useful information, and so are answers that deflect them. For a fuller list of questions and red flags, see the separate RFP questions topic.
How DeployOpen scopes and delivers a lakehouse requirements phase
DeployOpen treats the requirements checklist as a written, fixed-scope phase that ends in decisions, not slides. We work through table format, catalog, engines, storage, access control, freshness and the run team with your architects and data owners, then document each choice so the build can start from agreed ground.
How we scope
- Written scope lists the workloads, data domains and source systems the lakehouse requirements cover.
- Each requirement area gets a named owner on your side and a recorded decision or open question.
- Volumes, freshness targets and retention expectations are captured as figures your team confirms, not estimates we assume.
- Ongoing support after the phase is available as a retainer, scoped separately from the requirements work.
How we build
- We compare Iceberg, Delta Lake and catalog options against your engines, cloud accounts and on-premises constraints.
- Access control and lineage requirements are tested against a small proof of concept on representative tables.
- Sizing, backup and table maintenance assumptions are written down so they can be checked against real measurements.
- Security and compliance evidence needs, such as access logs and retention records, are mapped to the design.
How we hand over
- Deliverables include a signed requirements register, architecture decisions and the reference design for the build phase.
- Run ownership is documented: which team operates the catalog, engines, storage and upgrades after go-live.
- Open questions for your independent auditors or data protection officers are listed with the evidence each one needs.
- Handover sessions walk your engineers through the decisions so the reasoning stays with your organization.
How we support
- Requirements can be revisited through change requests against the written scope before the build begins.
- Post-handover support, upgrades and continuous testing are available as retainers agreed in writing.
- Security testing of the resulting lakehouse, including access paths and AI features, can be scoped as a separate engagement.
- Ask us to confirm which deliverables and response arrangements apply before the scope is signed.
