Why an open source data lakehouse for financial services starts with reproducible reports
A bank or insurer often must reproduce a regulatory report months after it was filed, with the same figures, the same logic and the same source data. An open lakehouse built on Apache Iceberg over self-hosted object storage, queried with Trino, meets that requirement when the firm controls where data sits, who can read it and what evidence each query leaves behind. Iceberg provides time travel to exact historical snapshots and rollback to an earlier table state, and Trino's Iceberg connector supports both, so a reporting team can rerun a query against the table as it stood on the filing date. The format is licensed under Apache License 2.0, and Trino under Apache-2.0, so the firm does not depend on a single vendor's warehouse to keep that history readable.
This differs from generic analytics in three ways. First, reproducibility is a control in its own right: snapshot retention, catalog history and query logs must be designed so that old snapshots are not expired before the retention period ends. Second, the platform may fall under ICT third-party rules. Under DORA, Article 30(2) requires every ICT services contract to state where services are provided and data processed, and Article 28(3) requires a register of all such arrangements, so the storage location, catalog host and any managed service behind them must be documented. Third, the scope of customer and cardholder data must be set before ingestion. If card data enters the lakehouse, PCI DSS may apply to the storage, query and admin paths, and the firm should confirm its assessor's view of that scope before the first load.
The short answer on components is Iceberg as the table format, Trino as the query engine, a table catalog such as an Iceberg REST catalog or Hive Metastore (Trino does not hold data itself), and a metadata catalog such as DataHub or OpenMetadata for lineage and ownership. Several decisions must be made before build: which catalog holds the authoritative table history, who owns snapshot expiry and retention settings, which access model applies (file-based, Apache Ranger or Open Policy Agent), where data and backups physically sit, and whether the team that runs the platform can produce a tested exit plan. Confirm each of these with your compliance and third-party risk teams first, because they determine the design more than the software does.
Which regulatory requirements shape the lakehouse design
This table maps each regulatory requirement supplied for financial services to the lakehouse design decision it tends to force. Read each row as a question to settle before build starts, not as a statement that the design conforms. Auditors and assessors decide conformity, so use the evidence column to plan what to produce.
| Requirement | Design decision it implies | Evidence an assessor will ask for |
|---|---|---|
| DORA Article 28(3): register of ICT contractual arrangements | Record the lakehouse components (object storage, catalog, query engines, orchestration) and each provider as entries in the register, including any managed service used for backup or hosting. | Current register entries mapped to each lakehouse component and its owner |
| DORA Article 28(8): tested exit strategy and transition plan | Keep table data in open formats (Iceberg or Delta Lake on firm-controlled object storage) so that a different engine or catalog can read the same files if a provider is replaced. | Documented exit plan and a test record showing tables read from a second engine or catalog |
| DORA Article 30(2): contract terms on location, change notice and data return | Confirm in the contract where data is stored and processed, and that the vendor or integrator can return data on termination or insolvency. Self-hosting in firm-chosen regions makes location easier to state. | Contract clauses on data location, change notice and data return, reviewed by legal and the ICT risk function |
| PCI DSS v4.0.1 scope | Decide early whether cardholder data reaches the lakehouse at all. Tokenize or exclude primary account numbers where possible, because any system that stores or processes cardholder data falls into scope and needs the full control set. | Scope statement and data flow diagram showing which tables, catalogs and query nodes sit inside the cardholder data environment |
| SEC ICFR IT general control areas (program development, program changes, computer operations, access to programs and data) | Put lakehouse pipelines and dbt-style transformations under version control with reviewed changes, scheduled job logs and restricted production access. Table snapshots support reproducing a reported figure as it stood at close. | Change tickets and approvals, job run logs, access review records and reproducible queries against a dated snapshot |
| FTC Safeguards Rule: MFA for any individual accessing an information system (16 CFR 314.4(c)(5)) | Require MFA on every path into the query engines, catalog, notebooks and BI tools, not only the portal. Any exception needs a written approval from the Qualified Individual. | Identity provider configuration for each query entry point and a written list of approved exceptions |
| Swift Customer Security Controls Framework v2026 (26 mandatory and 6 advisory controls) | Only applies if the firm is a Swift user. Place the Swift connector and its credentials in a restricted zone, and keep the lakehouse copy of Swift messages under the same controls as the source. | Independent assessment results and the annual attestation in KYC-SA, with evidence of controls on the lakehouse host |
| EU AI Act Article 12 (automatic event logging) and Article 19 (keep logs at least six months) | Applies only where a high-risk AI system uses lakehouse data. Log queries, retrieval and model inputs and outputs into retained storage, and confirm with counsel whether the system is in scope at all. | Log schema, retention setting of at least six months where applicable, and a record of which datasets fed each AI system |
How the storage, catalog and query layers fit together
The production environment holds the tables that feed regulatory and management reports. Its layers are object storage holding Parquet data files, Apache Iceberg tables that define schema and snapshots on top of those files, a table catalog that tells engines where each table's current metadata lives, and a query layer. Trino reads and writes these tables through its Iceberg connector, and Spark or ClickHouse can be added where batch transformations or low-latency aggregations need them. Trino does not store the data itself, so the object store and the catalog are separate systems that must be deployed, backed up and access-controlled on their own.
- Catalog choice: The Trino Iceberg connector documents Hive Metastore, AWS Glue, JDBC, REST, Nessie and Snowflake catalogs, selected with
iceberg.catalog.type. Confirm which one your environment can run and operate. - Object storage: Store data on self-hosted object storage under the firm's control, with versioning and retention settings agreed with the records team.
- Table format versions: Trino supports Iceberg format versions 1 and 2, with version 3 experimental. Pin the version your reports depend on.
- Time travel: Snapshots let teams query the table as it stood at a given time, which supports reproducing a past report.
- Fault-tolerant execution: Keep it off unless batch workloads need it, and if enabled, confirm the spooling location is inside the production trust boundary.
Before go-live, ask the implementation partner to document which catalog, metastore database and storage bucket each report depends on.
Development and test environments let analysts and engineers change transformations and models without touching reporting data. They should run the same table format, catalog type and Trino version as production, so that tests behave like production, but they must not hold a copy of live customer or account data unless that data has been masked or approved for use. The layers are the same as production, but they are deployed in a separate network segment with separate object storage buckets, a separate catalog database, and separate credentials. Promotion from development to production is then a controlled step, not a shared storage path.
- Same versions: Match the Iceberg connector behavior, Trino release and catalog type to production so that query results and table metadata are comparable.
- Separate buckets and catalogs: Use distinct storage locations and catalog namespaces so a test job cannot write to a production table path.
- Synthetic or masked data: Load masked or synthetic extracts only. Document how the masking was done.
- Spooling scope: If fault-tolerant execution is tested, point the exchange spooling to a development bucket, not to the production bucket.
- Promotion evidence: Record which build and which table definitions were promoted, so the change trail can be shown to auditors.
Ask the partner how environment separation is enforced and whether it is tested, rather than assumed from naming conventions.
A restricted zone holds personal and sensitive financial data, such as customer identifiers, account details or cardholder-related fields where these are in scope. It is a separate trust boundary: its object storage, catalog and query access sit behind their own network controls, identity provider and authorization rules. Access is granted per table or per column through the query layer, using the access control mechanisms that Trino documents, including file-based rules, Apache Ranger or Open Policy Agent. Users reach the zone through authenticated sessions, and the query engine should be configured to use one of the documented authentication methods, such as LDAP, OAuth 2.0 or certificates.
- Dedicated query cluster: Run a separate Trino cluster for the zone so that misconfiguration in general reporting cannot expose restricted tables.
- Authentication: Choose from the Trino-documented methods and confirm how service accounts are issued and rotated.
- Fine-grained rules: Define access at schema, table and column level, and test the rules with negative cases.
- Spooling in the zone: If fault-tolerant execution spools to S3, Azure Blob Storage, Google Cloud Storage or HDFS, the spool bucket inherits the zone's classification and must be encrypted and access-restricted.
- Logging: Keep query and access logs in storage the security team controls.
Confirm with your assessor which controls apply to this zone and how their evidence will be produced.
Trino's fault-tolerant execution is off by default. When enabled with a QUERY or TASK retry policy, intermediate results are written to an exchange location so that failed work can be retried. The TASK policy is recommended for large batch queries and requires an exchange manager that spools to Amazon S3, Azure Blob Storage, Google Cloud Storage or HDFS. Spooled data is a copy of query intermediates, so it is a data location in its own right, and it belongs in the trust boundary of the environment that runs the query. This is often overlooked in data-flow diagrams and in the register of where data is processed.
- Off by default: Enable it only for workloads that need retries, and document the decision.
- Retry policy: Choose QUERY or TASK based on workload size and test both.
- Spool location: Name the bucket or container used for each environment, and keep it separate between environments.
- Retention of spooled data: Confirm how spooled files are cleaned up and whether that cleanup is covered by the retention schedule.
- Encryption and access: Apply the same encryption and access rules as the source data class.
Ask the partner to show the spool location on the data-flow diagram and in the list of locations where data is processed.
Which open source platforms fit a financial services lakehouse and why
A financial services lakehouse is a stack of decisions, not a single product. The table format, query engine, orchestration, lineage and reporting layers each carry different licensing, access control and evidence obligations, so each choice should be checked against the firm's own requirements before it is built. The cards below set out what each platform does, where it fits and what to confirm.
Apache Iceberg as the table format
Iceberg is licensed under Apache-2.0 and gives engines such as Spark and Trino a shared table format. Its time travel to exact historical snapshots and rollback to an earlier table state are the features that let a bank reproduce a regulatory report as it stood on a past date. Schema evolution without rewriting tables helps when reporting definitions change. Confirm which catalog the firm will run (Hive Metastore, AWS Glue, JDBC, REST or Nessie), how snapshot expiry is set, and who owns table maintenance.
Trino as the query engine
Trino is licensed under Apache-2.0 and queries data through connectors, so it reads Iceberg tables without owning the storage. Its authentication options include LDAP, Kerberos, OAuth 2.0 and certificates, and its access control can be file-based, Apache Ranger or Open Policy Agent. For a financial firm, the key checks are that column and row policies are enforced in one place, that the Java 25 runtime is approved by the platform team, and that fault-tolerant execution is tested before long batch reporting runs depend on it.
Apache Airflow for scheduled reporting
Airflow 3.3.2 is licensed under Apache-2.0 and is used to author, schedule and monitor reporting pipelines as code. Version-controlled DAGs give auditors a record of how a regulatory extract was produced, which supports change management evidence for IT general controls. Confirm how secrets are stored, how the scheduler and metadata database are backed up, and whether the firm's change approval process covers DAG releases as well as application code.
Dagster for asset-based pipelines
Dagster is licensed under Apache-2.0 and models pipelines as software-defined assets, so each reporting table has a declared upstream dependency and a tracked materialization history. That structure is useful when a risk or capital figure must be traced to the inputs used on a given run. Dagster and Airflow overlap, so the firm should pick one primary orchestrator for regulated reporting rather than running both for the same outputs. Confirm the team's skills and the lineage export it will feed into the catalog.
DataHub or OpenMetadata for lineage
DataHub and OpenMetadata are both licensed under Apache-2.0 and cover discovery, lineage, ownership and governance across the data estate. Lineage captured here supports the question of which source fed a reported figure and who approves changes to it. Neither tool replaces data quality testing or access policy, so the firm should confirm which lineage sources each catalog ingests (Trino, Iceberg, Airflow and Dagster are the sources to check) and who maintains ownership records when systems change.
Superset or Metabase for reporting
Apache Superset is licensed under Apache-2.0. Metabase is AGPL outside its enterprise directory, which uses the Metabase Commercial License, so features a firm needs for row-level security or SSO may sit in the commercial edition. Both tools inherit the query engine's permissions only if the connection is configured for it, so the firm should confirm that analysts cannot bypass Trino policies through a shared service account. Check the license terms before exposing dashboards to customers or external parties.
Airbyte licensing for ingestion
The Airbyte platform repository is licensed under the Elastic License 2.0, which restricts providing the software to third parties as a hosted or managed service. For an internal bank or insurer running Airbyte for its own staff this is usually a different question than for a vendor, but the firm should have counsel confirm the position for its entity structure and any affiliate or outsourced use. Where that review is not complete, the firm can compare ingestion with Airflow or Dagster jobs that use Apache-2.0 connectors.
How a regulated lakehouse rollout runs in phases
The sequence moves a financial services lakehouse from agreed scope to a handed-over platform, with each phase producing a documented output that a reviewer, auditor or supervisor can check. Nothing moves into production reporting until the earlier evidence has been accepted by the owners of the data and the controls.
Scope reports and classify the data
The firm and DeployOpen list the regulatory and management reports in scope, the source systems behind each one, and the data classes they contain, such as customer identifiers, cardholder data where it exists, and internal financial ledgers. Data owners sign off on each classification and on which zones may hold which classes. The output is a scope register that maps every report to its sources, owners, classification and retention obligation. It is done when each data owner has signed the register and open questions are logged with an owner and a due date.
Design the catalog and access model
The team selects the table format, catalog and query engines against the requirements agreed in the previous step, then designs the access model. This covers who can read raw, curated and reporting zones, how fine-grained row and column rules are expressed, how service accounts are scoped, and how the identity provider is connected. Apache Iceberg and Trino are common choices here, but the catalog and authorization design must be confirmed against the firm's own identity and audit requirements. The output is a written design with a role matrix and a test plan for access. It is done when access tests show that users outside a role cannot read the restricted columns or rows.
Land source data and reproduce one report
DeployOpen builds the ingestion for the agreed sources into the raw zone, then rebuilds one existing report from the lakehouse tables and compares it with the figures produced by the current system. Time travel is used to pin the exact snapshot the report was built from, so the same figures can be reproduced later from that snapshot. Differences are traced to source, transformation or rounding and are resolved before the test is accepted. The output is a reproduction record that names the snapshot identifiers, queries, row counts and reconciliation results. It is done when the report reproduces from the recorded snapshot with every difference explained.
Configure lineage, retention and audit logging
The team connects a metadata catalog so that each reporting table shows its upstream sources, transformations and owners. Retention rules are set per data class and per zone, including how old snapshots and orphaned files are expired, and the audit logs for queries and administrative changes are routed to storage the firm controls. The retention schedule is reviewed against the firm's legal and records obligations before it is applied. The output is a lineage map, a retention schedule and a log configuration with sample entries. It is done when a reviewer can trace a reported figure to its sources and see who accessed the underlying data during the test period.
Run the parallel reporting cycle
For at least one full reporting cycle agreed with the firm, the lakehouse produces the in-scope reports alongside the existing process, and reconciliation runs for every figure that differs. Each difference is logged with a root cause, an owner and a decision on whether it is a defect in the new platform, in the legacy process, or an accepted change. Report owners and the control function review the results before any report is switched over. The output is a parallel run log and a sign-off record for each report. It is done when every report has a signed decision on the result, and no open defects remain against the reporting controls.
Hand over runbooks and operating evidence
DeployOpen delivers runbooks for routine operations, including table maintenance, snapshot expiry, backup and restore tests, access reviews, upgrades and incident response, and walks the firm's platform team through each one. The register of ICT services, exit and transition plans, and the contract terms on data location are updated to reflect the platform. The output is a handover package with runbooks, configuration baselines, test records and the evidence index that the firm's compliance and audit teams will use. It is done when the firm's operators have executed the restore and access review runbooks themselves and the package is accepted in writing.
Evidence examiners ask for and the component that produces it
This table maps each piece of evidence a bank or insurer typically needs from its lakehouse to the component that produces it and the owner who should keep it. Read it as a checklist to agree with your assessors before build, since the exact format each examiner accepts varies. DeployOpen prepares systems, controls and evidence for assessments, but independent auditors and certification bodies issue any attestation or audit report.
| Evidence | Component that produces it | What the examiner or team checks |
|---|---|---|
| Snapshot history for report reproduction | Iceberg snapshot metadata, read with time travel in Trino or Spark | Can the team rerun a regulatory report against the exact table state used on the reporting date? |
| Access grants and policy decisions | Catalog and Trino access control (file-based, Apache Ranger or Open Policy Agent) | Who could read customer or card-related columns, and who approved each grant? |
| Query and audit logs | Trino event logs and catalog audit logs shipped to central storage | Can every query on sensitive tables be traced to a named user, service account and time? |
| Lineage graphs | DataHub or OpenMetadata fed by pipeline orchestration such as Airflow or Dagster | Can the team show source-to-report lineage for a figure in a filing? |
| Retention schedules and expiry records | Snapshot expiry jobs, table maintenance logs and object storage lifecycle rules | Are retention periods defined per dataset, and is there proof that expired data was removed? |
| DORA register input for ICT services | Service inventory maintained by the firm, populated with lakehouse components, hosting locations and providers | Does each component appear with its provider, data location and whether it supports a critical or important function? |
| Exit strategy and transition documentation | Architecture records covering open table format portability, catalog export and data return steps | Can the firm move tables to another engine or host if a provider fails or the contract ends? |
| PCI DSS scope documentation, where card data is present | Network diagrams, data flow maps and access inventory for the lakehouse and its connected sources | Does the documented scope match where cardholder data is stored, processed or could affect its security? |
What buyers ask before building a lakehouse for regulatory reporting
What drives the effort and cost of a self-hosted lakehouse for regulatory reporting?
The main drivers are the number of source systems, the volume and history of data to migrate, the number of regulatory report templates to reproduce, and how much access control and lineage must be built before the first report runs. Cost falls into four groups: infrastructure (object storage, compute nodes for Trino or Spark, catalog and metadata services), one-time implementation work (design, connectors, pipelines, access policies, testing), recurring operations (patching, upgrades, backups, table maintenance), and support. Ask any partner to split a quote into these groups and to state which items are fixed scope and which are time-and-materials. Infrastructure is paid to your cloud or data center provider, while DeployOpen's work is scoped in writing before it starts.
How long does it take to reproduce the first regulatory report?
It depends on how many sources feed the report and how well the existing report logic is documented. The first reproduction usually depends on three things: getting a reliable copy of the source extracts into Iceberg tables, recreating the transformation logic in SQL or Spark, and proving that the output matches the figures already filed. Use time travel to a known snapshot as the comparison point, and agree on the tolerance for differences before work starts. Your partner should give a timeline only after the source inventory and report scope are fixed in a written statement of work.
What must the bank's own team provide before implementation starts?
Your team needs to supply four things: named owners for each source system and report, read access or extract files for the source data, the current report specifications and reconciliation controls, and decisions on classification, retention and access roles. Infrastructure access is also required, including the cloud account, network segments, identity provider integration and a change window. Without these inputs, the implementation schedule will slip because engineers must rediscover logic that already exists in spreadsheets or legacy tools. Ask the partner for a written dependency list with owners and dates.
Can our internal team run the lakehouse after handover?
Often yes, but only if the team has the skills and time to operate it. Running a lakehouse involves Trino or Spark administration, catalog management, object storage operations, table maintenance such as compaction and snapshot expiry, security patching and backup restore tests. Many firms split the work: internal staff own data products and access requests, while a retainer covers upgrades, monitoring and incident support. Ask for a handover plan with runbooks, a list of skills the operations team needs, and a test of restore and upgrade procedures before the partner leaves.
How does Iceberg snapshot expiry interact with our retention rules?
Snapshot expiry removes old table metadata and the data files that only those snapshots reference, which is how time travel windows are limited. Retention policy must therefore set two separate periods: how long reports must be reproducible through time travel, and how long the underlying records must be kept. Expiring snapshots too early can make a past report impossible to reproduce, while expiring them too late keeps deleted personal data in old files. Iceberg documents snapshot expiration and rollback as core features, so the policy should specify expiry schedules, orphan file cleanup and a record of each run. Confirm these steps with your compliance and records teams before configuring any job.
How does DORA exit planning apply to self-hosted open-source software?
DORA requires documented, tested exit strategies and transition plans for ICT services that support critical or important functions, and contracts must state where data is processed and how data is returned on termination. Self-hosted open-source software changes the shape of this work: you do not depend on a vendor's proprietary storage, which helps portability, but you still own the exit plan. Open table formats such as Iceberg let data remain readable by several engines, so the exit plan can focus on catalog export, metadata migration and tested restores. Confirm with your legal and resilience teams which ICT services support critical functions and how they appear in the register of information.
Does a self-hosted lakehouse satisfy audit or regulatory requirements by itself?
No. A lakehouse can support evidence for IT general controls, access reviews, change logs and data lineage, but it does not make the firm compliant on its own. Independent auditors and regulators decide whether controls are adequate, and DeployOpen prepares systems, controls and evidence for those assessments without issuing audit reports or certifications. Ask your auditor which control areas they will test, such as program changes, computer operations and access to data, and agree on the evidence format before the design is final.
What to ask an implementation partner before signing
Start with catalog and access design, because every later control depends on it. Ask the partner which Iceberg catalog they will run (Hive Metastore, AWS Glue, a REST catalog or Nessie are all options Trino's Iceberg connector supports through iceberg.catalog.type), who can change catalog metadata, and whether access is enforced in Trino through file-based rules, Apache Ranger or Open Policy Agent. Ask them to show, for your own roles, how a row filter and a column mask would be written and tested. Then ask how Ranger or OPA policies are versioned: stored in Git, peer reviewed, deployed through a pipeline with an audit trail, and rolled back as a unit. A partner who edits policies in a console with no history cannot show a regulator who could read which table on which date.
Ask how report reproduction is tested, since time travel is the feature regulators will care about most. A credible answer names the snapshot ID or timestamp recorded for each submitted report, a scheduled job that reruns the query against that snapshot, and a comparison of row counts and control totals with the filed figures. Ask whether the partner has tested this against the Iceberg version you will run and whether they can show a restore from a rollback to an earlier table state. Ask how lineage and retention are evidenced: which catalog (DataHub or OpenMetadata are both Apache-2.0 options) captures column-level lineage from Airflow or Dagster jobs, how snapshot expiry and orphan file cleanup are scheduled, and what record shows a retention rule was applied. Ask for the sample evidence pack before you commit to it.
Ask for DORA documentation in the form your register of information and exit planning will need. Article 28(3) expects an entry for each contractual arrangement for ICT services, and Article 28(8) expects tested exit strategies for services supporting critical or important functions. Article 30(2) expects each contract to state where services are provided and data processed, with advance notice of changes, and to cover data return on insolvency or termination. Ask the partner to draft the data location and return clauses against those articles, to name every location where code, backups and logs will sit, and to explain how a Trino cluster and its object storage could be moved to another provider. On certification, be direct: DORA issues no certification scheme, and PCI DSS assessments end with a Report on Compliance or Self-Assessment Questionnaire, not a vendor badge. Treat any promise of certification, guaranteed audit outcome or a "DORA-certified" platform as a red flag, and ask for the written handover and upgrade plan, including who patches Trino (release 484 was published on October 15, 2026, and Trino's deployment guide requires Java 25, minimum 25.0.1) and who signs off on upgrades in production.
How DeployOpen scopes and supports a financial lakehouse
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside the bank's or insurer's own infrastructure, including its cloud accounts, data centers or isolated networks. For a financial lakehouse, the work starts with a written scope that names the regulatory reports, source systems and access rules in play, and engagements are fixed-scope with deliverables and timelines agreed before work begins.
How we scope
- Inventory the regulatory reports, source systems and data domains the lakehouse must serve
- Agree which zones, retention rules and access groups apply to cardholder, customer and finance data
- Confirm where data, catalogs and logs will be hosted, which the written scope records in full
How we build
- Deploy the table format, catalog, query engine and object storage in your own environment
- Configure fine-grained access control, lineage capture and time travel settings for reporting tables
- Test restores of earlier table snapshots so report reproduction is shown to work before go-live
How we hand over
- Document architecture, configuration, data flows and the control evidence each component produces
- Provide runbooks for table maintenance, snapshot expiry, backups and access reviews for your team
- Run knowledge transfer sessions so operations staff can run the platform without DeployOpen
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, scoped separately from the build
- Plan version upgrades of query engines and catalogs with tested rollback steps in the written scope
- Prepare systems and evidence for your compliance assessments, while auditors and certification bodies issue the opinions
