Why data pipelines for energy and utilities start with classifying the data
A utility or grid operator that wants smart meter and grid telemetry in analytics should classify every data flow before choosing components. The regulatory category of each flow decides where it may be processed, who may access it and how long it is kept, so the architecture follows from that classification rather than from tool preference. Three categories usually matter: BES Cyber System information and operational telemetry, which may fall under NERC CIP in the United States; customer and metering data, which carries privacy duties; and corporate or market data, which may sit outside the operational perimeter. In the EU, NIS2 lists energy as a sector of high criticality, so the same flows may carry operator obligations there. The short platform answer is Apache Kafka for ingestion, Apache Airflow or Dagster for orchestration and ClickHouse for time-series storage, but each placement depends on the classification.
Start by confirming your registration and impact rating. Under NERC CIP, the Balancing Authority, Reliability Coordinator and Transmission Operator functions and the high, medium or low impact rating under CIP-002-5.1a determine which assets are in scope. Ask your compliance team whether any telemetry feed originates from or can influence a BES Cyber System, and whether it crosses an Electronic Security Perimeter defined under CIP-005-7. A feed that touches those systems should not land in a shared analytics cluster without a documented boundary, and the Kafka brokers and ClickHouse nodes receiving it may themselves need to be classified. Confirm the rating with your regulator or auditor, not with the vendor.
Write the classification down as a data flow register before any connector is configured. For each flow, record the source, the destination, the classification, the permitted network zone, the retention period and the named owner. Check whether the vendor supply chain for each component is covered by your CIP-013-2 plan, including software integrity verification for Kafka, Airflow or Dagster and ClickHouse releases. Confirm with your architects which ClickHouse and Kafka releases you will run, since both projects publish support windows that affect upgrade planning. The register becomes the input for the pipeline design, the access model and the evidence you show assessors.
Which energy regulations shape pipeline requirements
This table maps each energy regulation or standard in scope to what it means for the pipeline build and the evidence an assessor will expect. Read each row as a design constraint that should be settled before ingestion starts, since moving data later is far more costly. The main takeaway is that high and medium impact operational data needs its own zone, controlled remote access and signed software before analytics workloads begin.
| Requirement | What it means for the pipeline build | Evidence to prepare |
|---|---|---|
| NERC CIP-002-5.1a impact classification | Classify each source system as high, medium or low impact before any feed is designed. Keep high and medium impact data out of the general analytics zone. | Documented impact classification of each source and the data flows that leave it |
| NERC CIP-005-7 Electronic Security Perimeter | Place routable pipeline components for high and medium impact systems inside an Electronic Security Perimeter. Route interactive access through an Intermediate System with encryption and multi-factor authentication. | Network diagrams of the perimeter, access logs and MFA records for interactive sessions |
| NERC CIP-013-2 supply chain risk management | Require signed packages and verified checksums for Kafka, ClickHouse, Airflow and connector images. Document vendor remote access for upgrades and support. | Supply chain plan, vendor notification terms, software verification records, periodic plan reviews |
| NERC CIP-011-3 information protection | Define which telemetry fields count as BES Cyber System Information. Encrypt that data in transit and at rest, and limit who can query it in ClickHouse. | Information classification register, access lists and encryption configuration |
| NERC CIP-015-1 internal network security monitoring | Enforceable from 1 October 2028. Plan for monitoring of east-west traffic between Kafka brokers, pipeline workers and ClickHouse nodes, and retain the logs. | Monitoring design, log retention settings and alert handling records |
| NIS2 Annex I energy sector | Treat electricity, district heating and cooling, oil, gas and hydrogen operators as high criticality. Apply risk management and incident reporting to pipeline operations, not only to the grid. | Risk assessment covering pipelines, incident response runbooks and reporting contacts |
| Delegated Regulation (EU) 2024/1366 cross-border electricity flows | For data exchanged across borders with transmission or market partners, agree cybersecurity rules and data formats in writing. Build interfaces that enforce those agreed rules. | Partner interface agreements, data exchange specifications and access control records |
Which open-source platforms fit smart meter and grid telemetry
Smart meter and grid telemetry pipelines usually combine a streaming backbone, a time-series store, an orchestrator, edge collection and a lineage catalog. Each component below carries its own licensing, version and security decisions, and each needs confirmation against the utility's network zones, impact classification and retention obligations before a design is fixed.
Apache Kafka for streaming ingestion
Kafka is licensed under Apache-2.0 with no paid edition from the project itself, so cost sits in infrastructure and operations. Supported releases are 4.3.1, 4.2.2 and 4.1.2, and Kafka 4.0 and later run only in KRaft mode, so any ZooKeeper-based cluster must migrate first. Confirm which SASL mechanism and TLS listener model fit the utility's identity stack, and confirm how ACLs are set, because in KRaft clusters resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is enabled.
ClickHouse for time-series storage
ClickHouse server is Apache-2.0 licensed, and ClickHouse Inc. sells ClickHouse Cloud separately, so a self-hosted deployment stays within the utility's own data center. Replicated tables coordinated by ClickHouse Keeper need at least three Keeper nodes in production. Backups must be restore-tested, since replication does not protect against human error or corruption. Confirm which of the supported versions (26.9, 26.8, 26.7 and 26.3) the utility's patch policy requires, and whether LDAP, Kerberos or JWT integrates with its directory.
Apache Airflow or Dagster for orchestration
Airflow 3.3.2 and Dagster 1.13.26 are both Apache-2.0 orchestrators that schedule batch jobs such as meter read validation, daily settlement extracts and ClickHouse partition maintenance. The choice depends on whether the team prefers task-centric DAGs or asset-centric definitions. Confirm how failed loads, late-arriving meter reads and backfills are rerun without duplicating interval data, and confirm that DAG and asset changes pass through the utility's change approval process.
Apache NiFi or Node-RED at the edge
Collection from substations, head-end systems and meter data management exports often crosses zones where NiFi or Node-RED (Apache-2.0, release 5.0.8) can run as a controlled collector. Node-RED suits small, low-throughput flows, while NiFi offers provenance and back-pressure for larger transfers. Any component that touches operational technology must be placed by a documented zone and conduit design, and the utility must confirm that collectors do not create an unmanaged path into an Electronic Security Perimeter.
OpenMetadata or DataHub for lineage
Both catalogs are Apache-2.0 licensed and can record ownership, lineage and classification for meter datasets, which helps show where customer usage and grid data travel. This matters for records of data handling and for explaining which pipelines produce regulated reports. Confirm that the catalog can ingest lineage from the specific Airflow or Dagster version in use and that catalog metadata itself is classified and access-controlled, since it can reveal network topology.
Secrets, identity and access controls
Pipeline credentials for meter head ends, Kafka brokers and ClickHouse users are a common path to compromise. Each service account should map to a named role, with secrets held in a vault the utility controls rather than in DAG code or flow files. Confirm how service accounts are reviewed, rotated and removed and which external authenticator ClickHouse will use, so that human and machine access follow the same identity policy.
Deployment patterns for IT, OT-adjacent and cross-border estates
This pattern fits an estate where smart meter billing data, customer usage history and aggregated grid reports stay on the corporate network and no pipeline component reads from or writes to control systems. The pipeline runs in the operator's own data center or cloud account, with Kafka or Airflow ingesting from the head-end and billing systems into a ClickHouse cluster used for analytics. The design question is less about network segmentation and more about classifying each source, since meter identifiers and customer records carry privacy duties even when no operational control is involved.
- Source classification: Tag each feed as customer data, operational data or derived metrics before any connector is built, so retention and access rules attach to the right dataset.
- Placement: Keep Kafka brokers, Keeper nodes and ClickHouse replicas in the corporate zone, with no route to OT networks.
- Credentials: Store connector secrets in the operator's own secrets manager and limit service accounts to the specific topics and tables each job needs.
- Upgrades: Plan Kafka and ClickHouse upgrades against the vendors' published support windows, since unsupported versions receive no security fixes.
- Restore testing: Run restores of ClickHouse backups on a spare cluster on a schedule, because a backup never restored in testing may fail when it is needed.
Before committing to this pattern, confirm with compliance counsel that no feed in scope is a BES Cyber System data flow, because that changes the design entirely.
This pattern applies when grid telemetry, substation or generation data is collected from systems that sit near operational technology and must cross into an analytics environment. NIST SP 800-82 Rev. 3 (*Guide to Operational Technology (OT) Security*) and the ISA/IEC 62443 series of standards describe how to segment these zones and control what passes between them. Where a BES Cyber System is involved, the NERC CIP standards define the Electronic Security Perimeter and the rules for Interactive Remote Access, so the pipeline design must start from the impact classification of the source systems.
- Unidirectional flow: Prefer one-way movement from the OT zone into a demilitarized or analytics zone, with no inbound path that could reach control systems.
- Perimeter placement: Confirm with the compliance team whether any pipeline component falls inside an Electronic Security Perimeter, since CIP-005-7 governs routable connectivity for high and medium impact assets.
- Remote access: Route any administrative access to pipeline hosts through controlled intermediate systems with encryption and multi-factor authentication.
- Vendor access: Document vendor-initiated remote access to connectors and brokers as part of the supply chain plan that CIP-013-2 requires.
- Time synchronization and logging: Record ingestion events with reliable timestamps so the monitoring duties in CIP-015-1 can be met when that standard becomes enforceable.
Confirm the impact rating of each source with the operator's CIP program owner before placing any component, and do not rely on this summary as a classification.
EU energy operators in scope of NIS2 must manage cybersecurity risk across their operations, and energy is listed as a sector of high criticality in Annex I of Directive (EU) 2022/2555, with electricity, district heating and cooling, oil, gas and hydrogen as subsectors. Where electricity flows cross borders, Commission Delegated Regulation (EU) 2024/1366 adds sector-specific cybersecurity rules that supplement Regulation (EU) 2019/943. The pipeline therefore needs to show which data leaves the country, which operator holds it, and how each transfer is controlled.
- Data residency map: Record where each stream is ingested, stored, replicated and backed up, including any offsite ClickHouse backup target such as S3-compatible storage.
- Replication scope: Keep replicas inside the jurisdiction the regulator expects, and confirm that Kafka geo-replication or cross-cluster mirroring to other countries is approved before it is enabled.
- Authentication across operators: Use Kafka SASL or SSL and ClickHouse external authenticators such as LDAP, Kerberos or JWT, so partner access is tied to identities the operator controls.
- Incident notification path: Build alerting so that the operator can meet NIS2 reporting timelines, and confirm those timelines with legal counsel.
- Partner data sharing: Limit cross-border exports to aggregated or pseudonymized values where the partner does not need raw meter data.
Ask the implementation partner to produce the data flow diagram and transfer register before any cross-border feed goes live, and confirm the applicable national transposition with local counsel.
How a utility rolls out telemetry pipelines in phases
The sequence moves a utility from impact classification and source mapping to a tested ClickHouse environment, secured Kafka ingestion, proven backups and a supported production handover. Each phase closes with a written output that the operations, security and compliance owners can review before the next phase begins.
Classify scope and CIP impact
The engagement starts by agreeing the scope in writing and mapping each telemetry source and downstream system against the impact categories in CIP-002-5.1a. Utility security, compliance and operations leads confirm which assets are BES Cyber Systems and at what impact level, because that result sets the boundary rules for every later step. The output is a signed scope document with an impact classification for each system in the pipeline, and an open list of items the utility must confirm with its Regional Entity or internal compliance team.
Inventory sources and map data flows
DeployOpen works with the utility's operations and engineering teams to list every meter data source, grid telemetry feed, historian and SCADA-adjacent export, along with its protocol, message rate, owner and network zone. The team then draws each data flow from source to Kafka to ClickHouse, marking every crossing between OT and IT networks and every point where credentials or intermediate files are stored. The output is a data flow diagram and a source register that the security team reviews against the Electronic Security Perimeter design.
Build a pre-production ClickHouse environment
The team installs a pre-production ClickHouse cluster in the utility's own infrastructure, sized from the volumes agreed in the source register, and loads representative telemetry that reflects real meter counts, timestamp patterns and late-arriving records. Table designs, partitioning and query patterns for operational dashboards and regulatory reports are tested against this data. The output is a benchmark record and a tested table design, with the ClickHouse version chosen from the supported release lines and the changelog reviewed before the upgrade decision is recorded.
Configure Kafka security and ACL design
Kafka is deployed with TLS on all client and broker listeners, and the team selects SASL mechanisms with the utility's identity provider in mind. Topic-level access control lists are designed so each producer, consumer and Kafka Connect job has only the permissions it needs, and the StandardAuthorizer default for resources without ACLs is reviewed so that unlisted resources are not open by accident. The output is a reviewed ACL matrix, a configuration baseline, and a test report showing that unauthorized clients are refused on each listener.
Set retention and prove backups on a spare cluster
The team defines retention periods for raw telemetry, aggregated series and audit records with the compliance owner, then configures TTL rules and partition handling in ClickHouse to enforce them. BACKUP and RESTORE jobs are set up to local disk, S3-compatible storage or Azure Blob Storage as the utility's standard requires. Because replication does not protect against human error or corruption, the team restores a backup onto a spare cluster and compares row counts and sample queries against the source. The output is a retention schedule, a backup configuration and a signed restore test report.
Cut over production ingestion in controlled steps
Production producers are switched to the new Kafka topics in stages, starting with a limited set of meters or substations that the operations team has chosen. Consumer lag, ingestion error rates and ClickHouse merge and insert behavior are watched against the pre-production baseline, and the rollback procedure is rehearsed before the stage is widened. The output is a cutover log recording each stage, the monitoring evidence for it and the decision to proceed, signed by the operations owner.
Hand over operations and runbooks
DeployOpen documents the architecture, configuration baselines, ACL matrix, retention schedule, backup and restore runbooks, and incident procedures, then trains the utility's operations and security staff to run them. Upgrade planning is agreed for Kafka and ClickHouse, including how supported release lines and security updates will be tracked. The output is an operations handover pack and a record of trained staff, and any ongoing support or upgrade work is contracted separately as a retainer if the utility chooses it.
Retention rules and restore tests for operational time-series data
Retention and restore design decide whether an energy utility can answer a CIP audit question months after the telemetry was produced. In a ClickHouse deployment for smart meter and grid data, the retention window is set by the compliance lead, then enforced through table design: partitions (typically by month or day) hold the data, and TTL rules in the table definition move or delete parts once they age past the window. Backups are scheduled separately and must cover the retention horizon you promised, not just the last few weeks. Confirm with your compliance lead which records must be kept for CIP evidence (for example, configuration baselines, access logs and change records under CIP-011-3 information protection), how long each must remain retrievable, and which data may be deleted. Do not let a TTL rule delete evidence that a backup policy was meant to preserve. Write down the mapping between each table, its TTL, and the backup set that covers it.
Replication does not satisfy the backup requirement. The ClickHouse replication guide shows that a recovered replica resynchronizes automatically from a healthy peer, which is valuable for hardware failure, but a bad DELETE, a faulty ALTER or a corrupted part is copied to every replica. The ClickHouse backup documentation makes the same point: replication alone does not protect against human error or corruption. Use BACKUP and RESTORE for full and incremental copies to S3-compatible or local storage held in a separate failure domain, and restrict who can run destructive statements through access controls, such as the roles documented in ClickHouse access control.
The ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly when needed. Treat a restore test as a recurring control, not a one-time check. Restore a sampled partition and a full table into a spare cluster on a schedule your change process approves, then compare row counts, checksums or aggregate values against production, and record the result as evidence. Measure how long the restore takes so the recovery time you commit to is based on an observed number. Ask your partner to show the restore runbook, the spare cluster sizing, the owner of each test and where results are stored. Plan the spare cluster and its storage cost before you commit to the schedule.
Evidence auditors and regulators ask for from the pipeline
This table maps the records an assessor is likely to request to the pipeline component that produces them. Read each row as a question the entity should be able to answer with a dated, retrievable artifact. DeployOpen prepares this evidence, while NERC audits, regulator assessments and any independent review are performed by others.
| Evidence item | What it shows | Where it usually comes from in the pipeline |
|---|---|---|
| Impact classification records | Which systems are BES Cyber Systems and whether they are high, medium or low impact under CIP-002 | Asset inventory and the entity's signed classification worksheet, linked to the Kafka brokers, ClickHouse nodes and ingestion hosts in scope |
| Electronic Security Perimeter diagrams | Where routable-connected high and medium impact assets sit relative to the ESP and Intermediate System under CIP-005 | Current network diagrams with firewall rules for Kafka listeners, ClickHouse ports and collector links, kept under change control |
| Kafka authentication and ACL configuration | Which clients and tools authenticate, over which mechanism, and which principals may read or write each topic | Broker listener settings, SASL and TLS configuration, and exported ACLs from the KRaft metadata log, reviewed against the access list |
| ClickHouse access control and external authenticator settings | Who can query telemetry tables, how users are authenticated and how privileges are granted and revoked | User and role definitions, external authenticator configuration (LDAP, Kerberos, X.509 or JWT), and periodic access reviews |
| Backup and restore test logs | That backups exist, cover the required tables and actually restore to a usable state | BACKUP and RESTORE job logs, checksums and dated restore-test reports from a spare cluster, since an untested backup is unproven |
| Vendor patch and version records | That Kafka and ClickHouse versions are supported and that patches were applied within the entity's own timeline | Upgrade change records compared against the Kafka downloads page and the ClickHouse security policy, including the supported lines at the time of review |
| Supply chain risk records under CIP-013 | Vendor notification terms, vulnerability disclosure handling and software integrity checks for pipeline components | Signed package verification logs, software bills of materials and the entity's vendor review file, reviewed at least every 15 calendar months |
| Data lineage and ownership | Where telemetry originated, how it was transformed and who owns each dataset | Lineage graphs and ownership records exported from OpenMetadata or DataHub, with pipeline run history from the orchestrator |
What do utilities ask before building telemetry pipelines?
Is a telemetry pipeline project a fixed-scope engagement, and what drives its cost?
Yes, most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and the quote follows from that written scope. The main cost drivers for an energy telemetry pipeline are the number of source systems (smart meter head-end systems, SCADA historians, substation devices), the impact classification of the systems involved under CIP-002, the number of environments (development, test, production, disaster recovery), the retention periods, and how much OT integration is required. Ongoing costs include infrastructure in your own accounts or data centers, licensing where a component is not Apache-2.0, and support or upgrade retainers if you choose them. Ask any partner to itemize one-time build work separately from recurring operations, and to state which sources and environments the quote covers.
How long does a utility telemetry pipeline take to build?
The duration depends on the scope agreed in writing, so no fixed timeline applies across utilities. Scope that moves the schedule most includes the number of sources, whether systems in scope are classified as high or medium impact under CIP-002, how many compliance reviews and change approvals must occur before production, and whether a pre-production environment with representative data is required. A narrow first phase, such as ingesting one meter data stream into a test ClickHouse cluster with retention rules, will take far less time than a multi-source rollout with failover and backup testing. Ask the partner for a phased plan with a milestone for each compliance gate, and confirm that your internal change board calendar is included.
What does the utility's staff need to provide before the work starts?
The project needs accurate inputs from your side, and the partner will rely on them for scoping. Typically this includes an asset inventory of the systems that will feed or receive telemetry, the impact ratings already assigned to BES Cyber Systems, network diagrams showing routable connections and any Electronic Security Perimeter boundaries, data classification and retention requirements, and the named approvers for compliance sign-off. You should also provide access rules for the environments, the change management process, and the vendor list for any supply chain obligations under CIP-013-2. If impact ratings or diagrams are incomplete, confirm that the first phase includes validating them, because pipeline design depends on them.
Can our own team run the pipeline after the engagement ends?
Yes, if your team has the skills and the handover is planned from the start. Because the platforms are self-hosted open source, the components run in your infrastructure and your staff can operate them without a vendor license key. Running Apache Kafka, ClickHouse and an orchestrator such as Airflow or Dagster requires on-call coverage, monitoring, patching, backup testing and capacity management. The handover should include runbooks, restore tests, upgrade procedures and documentation of every configuration change. If your team cannot cover these tasks, a support retainer for upgrades and continuous operations is an option, and it should be priced separately in the written scope.
How are Kafka and ClickHouse upgrades planned against supported release lines?
Upgrades are planned against each project's published support policy, and the plan must be tested in a pre-production environment before production changes. For Kafka, the downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and rolling upgrades are supported from each release in the past year to the latest version. Kafka 4.0 supports only KRaft mode, so any ZooKeeper-based cluster must migrate first, and KRaft clusters older than 3.3.x are recommended to move to 3.9.x before 4.x. For ClickHouse, the security policy lists 26.9, 26.8, 26.7 and 26.3 as supported with security updates, and stable releases ship roughly monthly, so the plan should name which line you run and when it will be reviewed. Read the changelog before each upgrade, because minor backward-incompatible changes can occur.
Does DeployOpen provide NERC CIP certification for our pipeline?
No. There is no CIP certificate, and DeployOpen does not issue audit reports, attestations or certifications. Entities registered as a Balancing Authority, Reliability Coordinator or Transmission Operator receive compliance audits from NERC and the Regional Entities, at least once every three years. DeployOpen can help prepare the systems, configuration baselines, access controls, logging and evidence that your auditors will request for the in-scope pipeline components, including the controls in CIP-005-7 and CIP-013-2 where they apply. Confirm the mapping of each control to your environment with your compliance lead and your auditors before the work begins.
Which security duties apply to operational telemetry data?
Telemetry from energy systems may fall under NERC CIP, NIS2 for entities in the energy sector, and sector-specific rules for cross-border electricity flows, depending on your registration and jurisdiction. Engineers should check which systems are BES Cyber Systems, whether any pipeline component would sit in a routable connection to them, how Kafka clients and brokers authenticate and encrypt traffic (using SSL or SASL, with ACLs stored in the KRaft metadata log), and how ClickHouse users are authenticated and backed up. Backups should be restore-tested on a spare cluster, because the ClickHouse backup documentation warns that a backup never test-restored will probably not restore properly. Confirm each duty with your legal and compliance teams, since this page does not provide legal advice.
Questions to ask an implementation partner about energy telemetry
Start with classification. Ask the partner how they map each telemetry flow to the CIP-002 impact categories before any pipeline is built. Ask them to show which assets are BES Cyber Systems, which flows touch high or medium impact systems, and which are outside that scope but still feed a NIS2 risk assessment for the energy sector. A written data flow inventory with source, transport, storage and consumer for every hop is the document to request. If the partner cannot produce one before design starts, the later controls will be sized against guesses.
Then ask how the boundary for remote access is designed. Under CIP-005-7, routable-connected high and medium impact assets must sit inside an Electronic Security Perimeter, and all Interactive Remote Access must pass through an Intermediate System with encryption and multi-factor authentication. Ask where the Kafka brokers, ClickHouse nodes and Keeper hosts sit relative to that perimeter, which SASL mechanism and TLS settings the listeners and sasl.enabled.mechanisms configuration uses, and how vendor engineers reach the cluster. Ask also for the firewall rules and the access records the partner will keep, since those records are what an auditor will sample.
Ask how signed artifacts and vendor access are handled under CIP-013-2. The partner should describe how connector images, Kafka and ClickHouse packages are verified before install, who holds signing keys, and how vendor-initiated remote sessions are logged and approved. Ask how the plan is reviewed at least every 15 calendar months and how a vendor incident notification reaches your team. On restores, ask for a dated restore test using ClickHouse BACKUP and RESTORE on a spare cluster, with the result recorded. Replication alone does not cover human error, so a backup that has never been restored is not evidence. Finally, ask how Kafka and ClickHouse upgrades are staged: the partner should name the pre-production cluster, the changelog review step, and the rollback path. Confirm current supported lines on the vendor pages before accepting any plan.
How DeployOpen scopes and supports energy telemetry pipelines
Energy telemetry pipelines touch operational data, regulated assets and long retention periods, so scope, ownership and evidence need to be settled before any code is written. The cards below describe how DeployOpen works on these engagements, from written scope and hardening through backup, documentation and ongoing support.
Fixed scope agreed in writing
Most engagements are fixed-scope, so the source list, meter and grid telemetry feeds, latency targets, retention rules and deliverables are written down before work starts. A quote follows from that written scope, and DeployOpen does not publish a price list. Ask for the scope document to state which feeds are in, which are excluded, and who owns each interface to the utility's systems.
Self-hosted deployment in your infrastructure
DeployOpen plans and implements Kafka, orchestration, and ClickHouse inside the utility's own data centers or cloud accounts, so telemetry and operational records stay under the operator's control. Confirm where the operational technology network boundary sits, which hosts run each component, and whether any vendor cloud service receives meter data.
Hardening Kafka and ClickHouse access
DeployOpen configures authentication and encryption for each component, such as SSL or SASL listeners on Kafka and external authenticators on ClickHouse. Access control lists and user roles are set so that only named service identities can write telemetry. Ask how authorization defaults are handled, since Kafka denies resources without ACLs to super users unless the setting is changed.
Backup and restore testing
DeployOpen sets up ClickHouse backups to storage the utility controls and schedules restore tests on a spare cluster. Replication protects against hardware failure but not against operator error or corruption, so the restore procedure matters as much as the backup job. Ask for evidence of a completed restore test, not only a backup log.
Compliance evidence, not audit reports
DeployOpen prepares the configuration records, change history, access reviews and monitoring output that a utility will need for a compliance assessment. DeployOpen does not issue audit reports or certifications, and there is no CIP certificate. Confirm with your compliance team which evidence the regional entity or auditor expects, and agree the evidence list in the scope.
Upgrades and continuous testing by retainer
Ongoing support covers Kafka and ClickHouse upgrades, orchestrator patches, and recurring security testing, and these are offered as retainers after the initial build. Upgrades need a staging environment with representative data, because both projects publish version support windows that the operator must track. Ask how the retainer schedules upgrades around grid maintenance windows.
