Why an open source data lakehouse for energy and utilities needs a different plan
A self-hosted lakehouse on Apache Iceberg, queried with Trino and Spark, can serve both analytics and forecasting for a utility, provided CIP scoping and network separation are settled first. The platform choice matters less than the scoping decision. Smart meter interval data is piling up, grid and asset data sits in silos, and the compliance team needs a clear answer on where that data sits relative to BES Cyber Systems. Before any table is created, the utility must decide which data, pipelines and users fall inside the CIP boundary and which do not. Under CIP-002-5.1a, BES Cyber Systems are sorted into high, medium and low impact using bright-line criteria, and that sorting determines which controls apply to the systems around them.
Once scoping is agreed, the design follows from it. Analytical copies of operational data should be produced through a one-way path that the operational network can initiate and the lakehouse cannot reverse, and the team should confirm with its compliance lead how CIP-005-7 treats routable connections to any system that could affect high or medium impact assets. Object storage, the Iceberg catalog, Trino coordinators and Spark workers can then be placed in a zone the utility controls, with access through the identity provider it already runs. Short answer on platforms: Apache Iceberg for tables, Trino for interactive SQL, Spark for large forecasting jobs and an Iceberg REST or Hive Metastore catalog. DeployOpen plans, builds and documents this inside the client's own infrastructure, and the scoping document, network diagrams and evidence are delivered with it, not left for the utility to reconstruct later.
Which NERC CIP and NIS2 requirements shape the lakehouse build
This table maps the NERC CIP and NIS2 requirements most likely to touch an energy lakehouse to concrete build decisions. Read each row as a question to answer with your compliance team before data flows are designed. The main takeaway is that the first decision is whether any lakehouse feed comes from, or can reach, a BES Cyber System.
| Requirement | What it means for the lakehouse build | Evidence to prepare |
|---|---|---|
| CIP-002-5.1a impact classification | Classify each source system (SCADA historian, meter head-end, EMS feed) as high, medium, low or outside scope using bright-line criteria. Feeds from high or medium impact BES Cyber Systems bring the lakehouse into scope of later controls. | Documented impact classification of each source, with the rationale for each feed |
| CIP-005-7 Electronic Security Perimeter | Routable connections from high or medium impact assets must sit inside an Electronic Security Perimeter. Keep the lakehouse outside that perimeter and move data through a one-way, reviewed boundary rather than direct queries into operational networks. | Network diagrams showing perimeter boundaries and every approved data path |
| CIP-005-7 Intermediate System | All interactive remote access to in-scope assets must pass through an Intermediate System with encryption and multi-factor authentication. Analysts querying Trino or Spark should not reach OT systems directly. | Remote access logs and MFA configuration for the Intermediate System |
| CIP-011-3 BES Cyber System Information | Meter, topology and control data that qualifies as BES Cyber System Information needs protection on storage, in transit and in backups. Access control, encryption and disposal rules should apply to lakehouse tables, snapshots and object storage buckets. | Information protection program, access lists for BCSI tables, and disposal records |
| CIP-013-2 supply chain risk management | The plan must cover vendor incident notification, vulnerability disclosure, software integrity and authenticity checks, and vendor-initiated remote access. Each open-source component (table format libraries, query engines, connectors) needs a verified source, a version record and a named owner. | Supply chain plan, component inventory with versions, and review records dated within 15 calendar months |
| CIP-015-1 internal network security monitoring | Enforceable from 1 October 2028. Monitoring of internal network traffic inside in-scope environments will need sensors and retained logs. Plan storage for those logs now, since the lakehouse may be a natural place to keep them, subject to CIP-011-3 handling. | Confirmed applicability assessment and a monitoring architecture decision before the enforcement date |
| NIS2 Annex I energy | Energy, including electricity, district heating and cooling, oil, gas and hydrogen, is listed as a sector of high criticality. Obligations sit with the operator, so the lakehouse design should be reviewed under national NIS2 transposition rules with legal counsel. | Scope determination by the entity and national transposition checklist |
| Audit and penalty exposure | There is no CIP certificate. Entities registered as a Balancing Authority, Reliability Coordinator or Transmission Operator receive a compliance audit at least once every three years, and penalties apply per violation per day. Confirm your own registration category with NERC or your Regional Entity. | Compliance registration record and audit-ready evidence binder maintained by the entity |
Which utility data belongs in the lakehouse and which stays out
The first decision is whether a data set is used to analyze the grid or is needed to operate it. Interval smart meter reads, meter events, SCADA historian exports, outage and work management records, asset registers, weather and load forecasts and market data are analytical inputs and are well suited to an open lakehouse. Real-time control signals, protection settings and anything a BES Cyber System depends on to perform its function stay outside it. Before the first load, ask the operations and compliance teams to classify every source against the BES Cyber System inventory maintained under CIP-002-5.1a, since the impact rating of the system that produces a feed determines whether the feed can be copied out at all. Where the classification is unclear, treat the source as in scope for the operational boundary until the review is complete.
Within the lakehouse, use three zones. The raw zone holds data exactly as received from meters, historians and business systems, with ingestion timestamps and source identifiers, and is restricted to the platform team. The curated zone holds validated, de-duplicated and typed tables that analysts and forecasting jobs read. The restricted zone holds data that identifies customers, premises or the location of critical assets, such as service addresses joined to meter IDs or detailed substation topology. Access to restricted tables should be granted through Trino or Spark roles and enforced by the catalog or policy layer, with CIP-011-3 information protection requirements reviewed for any asset data that is treated as BES Cyber System Information. Confirm the classification rules with compliance before the zones are built.
Iceberg table design determines whether the meter estate stays manageable. Meter IDs are high-cardinality, so partition by date or by a time bucket rather than by meter, and rely on Iceberg's hidden partitioning so queries do not need to know the partition scheme. Sort or cluster data files by meter ID within each partition so point lookups prune files efficiently. For long retention, plan snapshot expiration, orphan file cleanup and compaction as scheduled jobs from the start, because interval data produces many small files and snapshots retain old files until they expire. Use time travel for reproducing forecast inputs and rollback for bad loads, and confirm with your engine vendors which Iceberg format version each engine reads, since the Trino connector supports format versions 1 and 2 with version 3 experimental.
How deployment constraints change by environment
Control center environments are the most restricted placement. Systems that monitor or control the grid, such as energy management and SCADA, sit within operational networks that the utility segments for reliability and security. Lakehouse components should not be placed inside that zone. In most designs, the lakehouse stays outside the operational boundary and receives only data that has passed a controlled one-way path. Trino coordinators, workers and object storage are not hosted in the control environment in a typical design, so the team should confirm with its OT security owner which hosts, if any, may sit in the zone at all.
- Placement: Analytics components run in corporate or dedicated data zones, with no query path from analytics back into control systems.
- One-way diffusion: Data leaves the OT side through a unidirectional gateway or a tightly controlled replication broker, and no inbound connection is opened to OT.
- Engine location: Trino coordinators and workers run outside the control environment and query object storage and a catalog that is also outside it.
- Host requirements: Trino requires a 64-bit Linux system and a 64-bit Java 25 runtime, minimum 25.0.1, so the hosting image must meet those requirements before any placement decision.
- Change control: Patching and image changes follow the utility's OT change process, which often has longer approval cycles than IT.
Confirm with the utility's OT security and compliance teams which zone each component may occupy and how the diffusion path is documented.
Corporate IT is the most common home for the lakehouse. Smart meter head-end data, customer information, work management and market settlement data usually already live in corporate systems, so the analytics platform can sit next to them. The main constraint is keeping the platform's trust boundary separate from general office networks and documenting which data sets came from the OT side. Object storage, the Iceberg REST catalog or Hive Metastore, and the Trino cluster are all deployed in this zone, and each needs its own access controls.
- Placement: Object storage, catalog and Trino run on infrastructure the utility operates, with network segmentation separating them from office traffic.
- Cluster layout: A coordinator plus workers is the documented starting shape, and a dedicated coordinator is recommended for larger clusters.
- Catalog choice: Trino needs a table catalog such as an Iceberg REST catalog or Hive Metastore, because it does not hold table metadata itself.
- Identity: Trino supports LDAP, OAuth 2.0, Kerberos, certificates and JWT, and access control can be file-based, Apache Ranger or Open Policy Agent.
- Data provenance: Tag tables that were ingested from OT sources so downstream users know their origin and handling rules.
Confirm which identity provider, catalog service and network segments the utility's corporate standards require before sizing the cluster.
Utilities that run workloads in public cloud accounts can host the lakehouse there, but the choice still depends on the utility's data classification, regulatory position and cloud policy. Self-hosted object storage in the utility's own account, with Iceberg tables and Trino, keeps the table format portable. Some cloud platforms expose Iceberg tables or read Iceberg through their own catalogs, which can help interoperability, but the team should confirm each vendor's current capabilities rather than assume them.
- Account structure: Place the lakehouse in a dedicated analytics account or subscription, separate from any account that connects to OT.
- Storage: Object storage is provisioned by the utility, and Trino's exchange manager for fault-tolerant execution can spool to S3, Azure Blob Storage, Google Cloud Storage or HDFS.
- Kubernetes option: Trino documents a Helm-based deployment on Kubernetes. Confirm against the utility's container policy, image sourcing and admission controls before choosing it.
- Network paths: Restrict egress from the analytics account and avoid private links to OT segments.
- Portability: Keep tables in Iceberg so engines and catalogs can change without rewriting data.
Confirm the utility's cloud governance, data residency rules and container policy before choosing between virtual machines and Kubernetes.
Which open source platforms fit a utility lakehouse and why
A utility lakehouse needs a table format, a query layer, a batch engine for forecasting, orchestration, lineage and operations dashboards. Each platform below covers one of those roles, and the licensing and deployment notes matter as much as the features when the data sits on a regulated grid.
Apache Iceberg as the table format
Iceberg is the Apache-2.0 table format that lets Spark, Trino and Flink work on the same meter and grid tables without copying them. Time travel to exact snapshots and rollback help you reproduce a figure submitted to a regulator long after the original load. Check which catalog you will use (Hive Metastore, AWS Glue, REST or Nessie), because the catalog holds the table state and must be backed up and access controlled like any other critical service.
Trino for interactive grid queries
Trino is an Apache-2.0 distributed SQL engine that reads Iceberg tables through its connector and also reaches operational sources such as PostgreSQL, Oracle and SQL Server. Its connector list includes 37 connectors, which helps when asset registers and outage systems live in different databases. A Trino cluster is a coordinator plus workers and needs Java 25 on 64-bit Linux, so plan the platform team's ownership of upgrades before the first dashboard depends on it.
Fault-tolerant execution for batch queries
Trino fault-tolerant execution is off by default and must be enabled with a QUERY or TASK retry policy. The TASK policy suits large batch queries, such as month-end interval aggregations, but it needs an exchange manager that spools intermediate data to S3, Azure Blob Storage, Google Cloud Storage or HDFS. Confirm which of those your data center or cloud account provides, size that spool storage separately from the lakehouse tables, and test a failed-node recovery before relying on it for settlement reports.
Apache Spark for forecasting and batch
Spark runs the heavier workloads that a SQL engine handles poorly: load forecasting feature builds, demand model training on years of interval data, and rebuilding derived tables after a correction from a meter data management system. It reads and writes Iceberg tables, so forecast outputs land where operations dashboards can query them. Ask the implementation partner how Spark cluster capacity is shared with interactive Trino workloads, since contention between the two is a common cause of missed batch windows.
Apache Airflow 3.3.2 for orchestration
Airflow is an Apache-2.0 platform for authoring, scheduling and monitoring data workflows as code, and version 3.3.2 was published on September 17, 2026. In a utility lakehouse it sequences meter file ingestion, validation, Iceberg table maintenance and forecast runs, and it keeps the run history that auditors ask for. Decide early whether pipelines must run inside the operational-technology-adjacent network zone or only in the corporate zone, because that choice determines where workers and their credentials can sit.
Dagster as an alternative orchestrator
Dagster is an Apache-2.0 data orchestrator built around software-defined assets, which fits a lakehouse where each grid or meter table has a clear owner and a defined freshness target. It can replace Airflow for teams that want asset lineage and checks built into the scheduler. Choose one orchestrator for the estate rather than mixing both, and confirm that the chosen tool supports the Iceberg catalog and the Spark and Trino connections your team will operate.
DataHub or OpenMetadata for lineage
DataHub (v1.7.0.1) and OpenMetadata (2.0.5) are Apache-2.0 catalogs that record lineage, ownership and data classification across meter, grid and asset tables. Classification tags help separate customer-identifying fields from grid topology data, which matters when topology details could be sensitive under the CIP standards. Compare how each handles the connectors you need for your MDM, SCADA historian and GIS sources, and test lineage capture on one real pipeline before choosing.
Superset for operations dashboards
Apache Superset 6.1.0 is an Apache-2.0 visualization platform for operations dashboards on Trino or other SQL sources. It suits outage tracking, transformer loading and forecast accuracy views for operations staff. Check its row-level security and SSO integration against your access rules before exposing feeder-level data beyond the control room. Avoid Metabase where the enterprise directory is needed, since it is AGPL outside that directory and under the Metabase Commercial License inside it.
Integration points and deployment patterns for utilities
A utility lakehouse sits between operational data, metering systems and analysts, so each integration point carries its own security and retention decisions. The cards below cover the sources that feed the platform, the access layer that governs queries, and the metadata and retention controls that keep the data trustworthy over years. Check each boundary against the network zoning that applies to your operational technology before any connection is designed.
Meter data management system ingestion
Interval reads from the meter data management system are usually the largest and most time-sensitive feed. Agree in writing which system is the system of record for validated, estimated and edited reads, and whether the lakehouse receives corrections as new snapshots or as row-level merges. Confirm the export format, the export schedule and how late-arriving reads are identified so forecasting jobs do not train on incomplete intervals.
Historian export into the lakehouse
Process historians hold high-frequency sensor and SCADA values that the lakehouse should receive as a copy, not as a live connection into the control environment. Define the export boundary, the tag list in scope and the batching interval, and confirm that export jobs initiate outbound from the historian side only. Verify how time zones, quality flags and tag renames are carried across so long-term trend analysis stays comparable.
Query access control with Trino
Trino supports file-based access control, Apache Ranger or Open Policy Agent, so you can decide who reads customer, meter and grid tables. Map roles to utility functions such as metering, grid planning and customer operations, and restrict personal data columns for analysts who do not need them. Test the policy engine against real query patterns before go-live, because table-level grants alone rarely meet the need for column and row restrictions.
Authentication for analysts and services
Trino documents password file, LDAP, Kerberos, certificate and JWT authentication, and the right choice depends on how your identity provider and service accounts are run. Separate interactive analyst access from service accounts used by Spark jobs and dashboards, and rotate or certificate-bind the service identities. Confirm that the chosen method works with the segmentation between the corporate network and any environment that touches grid data.
Lineage and metadata catalog
Meter, grid and asset tables are reused across billing, reliability reporting and forecasting, so a change upstream must be traceable to every consumer. DataHub and OpenMetadata both record ownership, lineage and discovery metadata, and both are licensed under Apache-2.0. Confirm which pipeline tools will emit lineage automatically, and assign a data owner for each feed so regulatory extracts can be traced back to their source.
Iceberg snapshot backup and retention
Iceberg keeps historical snapshots for time travel and rollback, and those snapshots also hold data that must eventually expire. Set snapshot retention separately for operational replay, regulatory records and personal data, and schedule expiry and orphan file cleanup so storage does not grow without limit. Back up the table metadata and the catalog database as well as the data files, and test a restore to a prior snapshot before relying on it for an audit.
How a utility lakehouse rollout moves from scoping to handover
The sequence starts with regulatory impact classification and ends with a handed-over platform, documented controls and evidence the compliance team can take into a CIP or NIS2 review. Each phase produces a named artifact that the next phase depends on.
Scope CIP impact and agree boundaries
Working with the compliance team, the operations technology owners and the registered entity's CIP lead, we map which systems and data sources sit near or inside BES Cyber System boundaries. Impact classification under CIP-002-5.1a is a bright-line exercise, so it is settled first and recorded in writing. The output is a signed scope document that states which data flows are in scope, which are excluded, and who approves changes to that boundary.
Define requirements and data zones
We agree the interval data volumes, retention periods, freshness needs and consumers for smart meter, grid telemetry, asset and market data. Zones are defined so that raw meter reads, curated tables and forecasting outputs have separate access rules, and no lakehouse zone is reachable from an operational network without a documented path. The output is a requirements register and a zone diagram that the security team has reviewed.
Build storage, catalog and query layers
The platform team deploys self-hosted object storage, an Iceberg-compatible catalog such as a REST catalog or Hive Metastore, and Trino and Spark clusters inside the utility's own data centers or cloud accounts. Trino reads data through connectors and needs its own storage and table catalog, so these components are installed and tested together. The output is a running environment with tested catalog operations, backup of catalog metadata, and a runbook for restarts and upgrades.
Design ingestion pipelines and table retention
We build ingestion from meter data management, SCADA historians and asset systems into Iceberg tables, using orchestration that records each load. Table design covers partitioning by time and meter or asset, snapshot expiry, and retention rules that match the utility's records schedule and any CIP evidence needs. The output is a tested set of pipelines, table specifications and a retention schedule signed by the data owner.
Implement access control and test security
Authentication and authorization are wired to the utility's identity provider, with role and column-level rules enforced through the query engine. Security testing covers the data platform, its APIs and its connections to any adjacent network, and findings are fixed and retested before go-live. The output is a documented access model, a test report with closed findings, and a list of accepted residual risks approved by the owner.
Deploy forecasting workloads on Spark or Trino
Load forecasting, demand and asset-health models are run on Spark jobs or Trino queries against curated tables, with outputs written back to a separate forecast zone. Model inputs and outputs are versioned so results can be reproduced from an Iceberg snapshot. The output is a working forecast pipeline with validated results and a record of which table versions each run used.
Package evidence and hand over
We assemble the evidence the compliance team needs: scope documents, data flow and zone diagrams, access rules, test reports, retention schedules, backup and restore results, and change logs. Operators are trained on the runbooks and on-call procedures, and ownership of the platform moves to the utility's team. The output is a handover pack and a completed training record. DeployOpen prepares this evidence, but auditors and the regulator decide whether it meets the applicable requirements.
Evidence auditors and regulators ask for, and which component produces it
This table maps the evidence that CIP, NIS2 and EU AI Act reviewers typically request to the lakehouse component that produces it. Read each row as a question to answer with a record, then check which component owns that record. DeployOpen prepares the evidence, but auditors and regulators assess it, and the assessment scope depends on your registration and impact classification.
| Evidence item | Component that produces it | What the assessor will check |
|---|---|---|
| CIP-005-7 Electronic Security Perimeter diagrams | Network architecture documentation and firewall or segmentation configuration maintained by your network team, with the lakehouse hosts placed inside or outside the perimeter as documented | Whether every routable asset tied to high or medium impact BES Cyber Systems sits inside the perimeter, and whether analytics hosts are clearly classified |
| Interactive remote access logs for CIP-005-7 | Intermediate System and jump host logs at the access layer, with session records for any vendor or engineer access to the lakehouse administration plane | That all Interactive Remote Access passes through the Intermediate System with encryption and multi-factor authentication, and that sessions are traceable to named individuals |
| CIP-013-2 vendor records for each open source component | Your supply chain register listing each component, version, source repository, license and the checksums or signatures verified at install, maintained alongside the vendor risk review | Software integrity and authenticity verification, vendor incident notification terms, vulnerability disclosure handling, and a plan review at least every 15 calendar months |
| CIP-011-3 information protection for BES Cyber System Information | Access control records from the catalog and object storage layer (Iceberg REST catalog or Hive Metastore, Ranger or OPA policies), plus encryption configuration and disposal records | That BCSI datasets are labeled, access is limited to authorized roles, and copies in analytics zones or backups follow the same protection rules |
| Iceberg snapshot history and table change records | Iceberg table metadata, which keeps snapshots that support time travel and rollback, combined with expiration and orphan file cleanup jobs | Who changed which table and when, whether historical states can be reproduced for an incident or audit, and that expiration follows the retention schedule |
| Trino query and access logs | Trino coordinator event logs and the access control layer (file-based, Apache Ranger or Open Policy Agent), forwarded to your central log store | Which users queried BCSI or operational tables, what they ran, and whether denied requests are also recorded |
| Data lineage from source to forecast | DataHub or OpenMetadata lineage graph, ownership records and classification tags for meter, grid and asset datasets | Traceability from SCADA or historian exports to the forecast outputs, and identified owners for each dataset |
| EU AI Act Article 12 event records for a high-risk forecasting system | Application and model logs written by the forecasting pipeline (Spark or Trino jobs, model registry entries, inference request logs), stored in your own object storage | Whether automatic event logging exists across the system lifetime, and whether the classification of the system as high-risk has been confirmed |
| EU AI Act Article 19(1) log retention | Log retention policy and object storage lifecycle rules set to keep AI system logs for at least six months unless other law requires longer | That retention is at least six months and that logs are protected against alteration and deletion before expiry |
What to ask an implementation partner before a utility lakehouse build
How do you document CIP scoping decisions and keep lakehouse data feeds outside the Electronic Security Perimeter?
Ask for the written scoping record first: which systems were assessed against CIP-002-5.1a, which BES Cyber Systems were classified as high, medium or low impact, and why each lakehouse feed sits where it does. A capable partner will show how ingestion from operational networks crosses the boundary through controlled, one-way or tightly defined paths, and how that matches CIP-005-7 requirements for routable-connected assets inside an Electronic Security Perimeter and for Interactive Remote Access through an Intermediate System with encryption and multi-factor authentication. Ask who signs off the scoping decisions on your side, how often they are reviewed, and where the diagrams and evidence are stored. Ask the partner to confirm the final CIP interpretation with your compliance team and your Regional Entity, since DeployOpen prepares systems and evidence but does not issue audit reports.
How do you verify software integrity and supply chain risk for each lakehouse component?
Ask for a component inventory that lists every package, container image and connector in the stack, with its version, license and source, plus the method used to verify integrity before installation. CIP-013-2 expects supply chain risk management plans that cover vendor incident notification, vulnerability disclosure, software integrity and authenticity verification, and coordination of vendor-initiated remote access, reviewed at least every 15 calendar months. Ask how the partner will map each of those plan elements to Trino, Iceberg, Spark, the catalog, the object storage and the orchestration tools, how vulnerability feeds are monitored, and how updates are staged in a test environment before production. Confirm that the plan review cadence matches your own program.
How do you design Iceberg retention and snapshot expiry for long-term interval data?
Start with the retention requirements from your regulatory and records schedule, then ask how the partner translates them into table properties, snapshot expiration rules and orphan file cleanup. Iceberg time travel and rollback depend on retained snapshots, so the expiry policy determines how far back you can reproduce a result and how much storage the metadata and data files keep. Ask for a written schedule per table, a test that shows expired snapshots and unreferenced files are removed, and a procedure for legal holds that pause expiry. Ask how the partner will size storage for interval data over the retention period and how they will prove that data was removed when the schedule requires it.
How do you test Trino access control before users and services get production data?
Ask which access control model is used, because Trino documents file-based rules, Apache Ranger and Open Policy Agent, alongside authentication through password file, LDAP, OAuth 2.0, Kerberos, certificates and JWT. Ask for a test plan that covers each catalog, schema and table, including denied queries, service accounts used by Spark jobs and forecasting pipelines, and the operator roles that can see metadata. Request the test results as evidence, and ask how changes to rules are reviewed, version controlled and logged. Confirm how audit records will be retained, since logs that support investigations and regulatory evidence have their own retention needs.
How do you support upgrades of Trino, Iceberg and Spark over time?
Ask for the upgrade method and the cadence the partner recommends, and whether upgrades are covered by a retainer or a separate fixed-scope project. Trino publishes frequent releases, and its deployment requirements include 64-bit Java 25 with a minimum of 25.0.1, so the Java runtime is part of every upgrade. Ask how the partner checks compatibility between Trino connectors, the Iceberg format version in use (versions 1 and 2 are supported, with version 3 experimental), the catalog and the Spark version before each change. Ask for a staging environment that mirrors production, a rollback plan, and written release notes for each change window.
What does the handover look like, and can our own team run the platform afterward?
Ask for a handover list that names the documents, runbooks, configuration repositories, access records and evidence packages you will receive, and the training your operators will complete before the engagement closes. The partner should show how the system is documented for your own staff, how backups and restores are tested, and how alerts and on-call responsibilities move to your team. Ask whether a support retainer is needed after handover, what it covers (upgrades, patching, continuous testing) and how it is priced, since DeployOpen works from a written, fixed scope and does not publish a price list. Ask your team to define the operational skills it already has, so the gaps and the training needs are clear before the work starts.
What drives the cost and timeline of a utility lakehouse build?
Cost and timing depend mainly on the number of data sources, the volume and frequency of interval data, the number of CIP-scoped boundaries that need design and evidence, the retention schedule, and the number of query engines and users. Ask the partner to break the quote into one-time work (architecture, CIP scoping support, integration, hardening, testing, documentation and migration) and recurring work (upgrades, patching, testing and support). Request a written scope that lists assumptions about data sources and environments, plus a change process for scope growth. Your own costs will include infrastructure, staff time from operations, compliance and security, and any outside audit or regulatory review, which the partner will not carry for you.
Which parts of the CIP work can we do ourselves, and which need an outside specialist?
Decide first which responsibilities stay with your compliance and operations teams, because the registered entity owns its compliance obligations. Your team can usually run day-to-day operations, data quality checks and user access requests, while the partner can build the platform, document the configuration, prepare evidence and test controls. Ask the partner which tasks it will hand back to you, such as approving CIP scoping decisions, signing risk acceptances and deciding how to respond to a Regional Entity audit. Ask for a responsibility matrix that shows who does each task, who approves it and who keeps the evidence.
How DeployOpen scopes and supports a utility lakehouse
DeployOpen plans, implements, hardens, documents and supports self-hosted open source platforms inside the client's own infrastructure, whether that is the utility's data center, its cloud accounts or an isolated network segment. For a lakehouse, that means the engagement covers the object storage, the Iceberg or Delta table layer, the catalog, the Trino or Spark query tier, the orchestration and the access model, all deployed where the utility's data governance and OT segmentation rules already apply. Work is agreed under a fixed-scope written agreement that names the systems in scope, the deliverables, the environments and the timeline before work starts, so the boundary between the lakehouse and any operational network is documented from day one.
DeployOpen also prepares evidence for compliance assessments, such as configuration baselines, access reviews, logging and retention settings, backup and restore records and the supply chain documentation a NERC CIP or NIS2 assessment may ask for. It does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, so the utility should confirm early which evidence its auditor expects and in what format. Ask any partner to show how each control will be evidenced, who in the utility owns the sign-off, and where the records will be stored.
Upgrades, security patching and continuous testing run under separate retainers, which matters for a lakehouse that will hold fifteen or more years of interval data and will need Trino, Iceberg and catalog upgrades on a steady cadence. Retainers are the place to agree on the test environment, the change window, the rollback procedure and who approves production changes. Ask for the retainer scope in writing, including which components are covered, how security testing of the query tier and access layer is scheduled, and how knowledge is handed over so the utility's own team can operate the platform if the relationship changes.

