Apache Kafka for energy and utilities

Apache Kafka for energy and utilities: where it fits in the grid

For a utility or grid operator deciding how telemetry moves from field devices and substations to analytics and control centers, Apache Kafka can serve as the event backbone when placement and access are designed first. The design questions come before the product choice: which networks each producer sits on, which consumers may read which topics, and which data may cross from operational technology (OT) networks into corporate IT. Kafka is Apache-2.0 software and the project offers no paid edition, so the cost sits in the work around it: architecture, build, hosting, monitoring, patching and support. Those costs are yours to plan whether you run the platform with your own staff or with a partner.

Two things change once Kafka sits between substations and corporate systems. First, traffic crosses OT and IT boundaries, so each topic, producer and consumer needs an owner, a network zone and a documented reason to exist there. Second, compliance scope follows asset impact classification. Under CIP-002-5.1a, BES Cyber Systems are sorted into high, medium and low impact, and that sorting determines which CIP requirements reach the systems that store, route or process their data. A Kafka cluster that receives data from a medium impact asset may therefore fall within scope, even though it is not itself a control system. Confirm your classification with your Compliance Enforcement Authority and your own asset inventory before you draw the cluster boundary.

Which energy requirements shape the Kafka design

This table maps each energy and utility requirement in the supplied scope to what it means for a self-hosted Kafka build, such as cluster placement, authentication, ACLs, monitoring and replication. Read each row as a design input to settle before topics and connectors are created. Requirements that apply depend on your registration, impact classification and jurisdiction, so confirm them with your compliance lead.

Requirement	What it asks for	Kafka build impact
NERC CIP-002-5.1a impact classification	Sorts BES Cyber Systems into high, medium or low impact using bright-line criteria in Attachment 1	Classify the systems that produce or consume Kafka data before choosing where brokers run, since the impact level sets which controls apply to the cluster
NERC CIP-005-7 Electronic Security Perimeter	Places routable-connected high and medium impact assets inside an Electronic Security Perimeter and routes Interactive Remote Access through an Intermediate System	Keep brokers and controllers inside the perimeter, expose listeners only to approved producers and consumers, and route administrative access through the controlled path
NERC CIP-013-2 supply chain risk management	Requires plans covering vendor incident notification, vulnerability disclosure, software integrity and authenticity verification, and vendor remote access, reviewed at least every 15 calendar months	Verify Kafka release artifacts and container images before deployment, record the versions in use, and document how upstream vulnerability notices reach the operations team
NERC CIP-012-2 Real-time data availability	Adds protection of the availability of Real-time data between Control Centers, enforceable 1 July 2026	Design replication between control centers with MirrorMaker 2 or equivalent, monitor replication lag, and test failover of consumer offsets and topic configuration
NERC CIP-015-1 Internal network security monitoring	Approved 26 June 2025 and enforceable from 1 October 2028, with CIP-015-2 extending monitoring to associated EACMS and PACS	Log authentication failures, ACL denials and administrative changes on brokers and send them to a monitoring pipeline, and confirm the retention your monitoring program requires
NIS2 Annex I energy sector and Article 21 risk management	Lists electricity, district heating and cooling, oil, gas and hydrogen as high-criticality subsectors, with risk-management measures under Article 21	Document the Kafka platform in the risk assessment, cover access control, backups, encryption in transit and incident handling, and confirm which entities in your group are essential or important
EU network code on cross-border electricity cybersecurity (Delegated Regulation 2024/1366)	Sets sector-specific cybersecurity rules for cross-border electricity flows, supplementing Regulation (EU) 2019/943	Where Kafka carries cross-border flow data, confirm with your regulator and transmission operator which network code obligations apply to the data path and its operators

Where the Kafka cluster sits across OT and IT networks

Placing a Kafka cluster inside a medium or high impact Electronic Security Perimeter keeps control-adjacent telemetry close to the BES Cyber Systems that produce it. Producers and consumers that need short paths to field data stay on the OT side, and the cluster itself becomes an asset that the compliance program must classify and protect. The trade-off is that every broker, controller, and administrative host you add falls within the scope of the controls that apply to that perimeter.

Latency: Local producers and consumers avoid crossing a boundary, which keeps the path short for data that operators or automation read.
CIP scope: Brokers, their storage, and their management hosts may be assessed as Protected Cyber Assets or as parts of the systems the standards govern, so confirm that classification before design.
Access paths: Any remote administration or vendor access must pass through the Intermediate System and multi-factor controls the standards require, which shapes how the cluster is maintained.
Change control: Patching, upgrades, and configuration changes follow the same change and vulnerability processes as the rest of the perimeter, which adds lead time.
Operators: The OT engineering team usually runs the cluster, so it needs Kafka skills and a documented on-call path that fits the operating environment.

Before you place anything inside a perimeter, ask your compliance team to confirm the impact categorization of each source system and the applicable systems for each control.

Kafka settings and modules that matter for utility telemetry

Utility telemetry runs on a small set of Kafka controls: cluster layout, transport security, authorization, cross-site replication and runtime versions. Each one decides whether meter and SCADA data stays protected, available and attributable when it crosses from OT into IT and between control centers.

KRaft cluster layout and retention

Kafka 4.x runs only in KRaft mode, so the controller quorum is part of the cluster design rather than an external dependency. For meter and SCADA topics, plan partition counts and retention per topic from the telemetry rate and the replay window your operators and analysts need, not from defaults. Verify that controller and broker roles, replication factors and disk capacity were sized for the busiest topic, and that retention settings are written down per topic.

TLS and SASL authentication

Kafka supports SSL/TLS for encryption and SASL for client and broker authentication, and security is optional, so a cluster can accept unsecured listeners without anyone deciding that. Choose SCRAM-SHA-512, OAUTHBEARER or GSSAPI (where Kerberos already exists) for every listener that carries telemetry. Verify that no plaintext listener is exposed on a network segment that touches OT, and that each meter gateway, historian and SCADA bridge has its own credential.

StandardAuthorizer ACLs in the metadata log

In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log, so access rules replicate with the cluster itself. Keep allow.everyone.if.no.acl.found set to false, which restricts resources without ACLs to super users. Verify that each producer can write only to its own telemetry topics, that control-room consumers have read-only grants, and that super user membership is short and reviewed.

MirrorMaker 2 for cross-site replication

MirrorMaker 2 runs on Kafka Connect and replicates topics with their configuration, consumer groups with offsets and ACLs between clusters, while preserving partitioning. This supports a standby control-center cluster or a regional aggregation site. Verify replication lag against the operational tolerance agreed with the control room, test failover and failback with real consumer group offsets, and confirm that the replicated ACLs match the source policy.

Pluggable authorization integration

Kafka authorization is pluggable and can integrate with external authorization services, which matters when utility identity lives in a central directory and access must be reviewed on the same cycle as other OT and IT access. Decide whether ACLs are managed in Kafka or generated from an external policy source, and document the single source of truth. Verify that removing a person or vendor account in the directory also removes Kafka access within a defined time.

Java 17 runtime for brokers and Connect

Since Kafka 4.0, brokers, Kafka Connect and tools require Java 17, while clients and Kafka Streams applications need at least Java 11. Utilities often run older application stacks, so client libraries and Streams jobs may lag the broker runtime. Verify the JVM on every broker, Connect worker and admin host, and confirm that older consumers and producers on the OT side are tested against the chosen release before cutover.

How a utility rollout moves from asset inventory to live replication

The sequence starts with classifying which assets and data flows fall under CIP obligations and ends with a tested, monitored replication link between control centers. Each phase produces a document or check that the next phase depends on.

Classify assets and impact with compliance
Your compliance team and DeployOpen review the asset inventory against the CIP-002 impact criteria and identify which systems would be BES Cyber Systems, which are associated EACMS, PACS or PCA, and which Kafka brokers, Connect workers or telemetry sources would sit in those scopes. Confirm the applicable CIP versions for your registration with your Regional Entity and compliance lead, since several revised standards carry later effective dates. The step closes with a signed impact register that names each Kafka component and its classification. The register sets the boundary for every later step.
Design network and perimeter layout
The team maps where brokers, controllers, Connect workers, MirrorMaker 2 nodes and telemetry gateways sit relative to the Electronic Security Perimeter, and how data crosses from OT to IT zones. Decisions cover which flows are one-way, which need an intermediate system for any remote access, and where Kafka clients from the IT side can read data without reaching control networks. The output is a data flow diagram with firewall rules and listener assignments that your network and security teams have reviewed. Verify the diagram against the actual routing before any hardware is racked.
Build the cluster on the chosen release line
DeployOpen builds a KRaft cluster on the Kafka release line you approve, using the controller and broker layout from the design step and infrastructure you control. Confirm the supported release line and Java runtime requirements against the Kafka downloads page and upgrade notes before installation, because the project lists supported releases and the Java versions differ by component. The step ends with a healthy quorum, replicated test topics and documented startup, shutdown and config files. Record the exact version and build artifacts so later upgrades start from a known baseline.
Configure security and test ACLs
The team enables TLS on every listener that carries telemetry or control-related data, selects an authentication mechanism that your identity team can operate, and sets the StandardAuthorizer rules so each producer and consumer principal has only the topics it needs. It also confirms that resources without an ACL are blocked for non-super users, since the default behavior depends on the allow.everyone.if.no.acl.found setting. Test each principal with positive and negative cases, including unauthorized reads and writes, and capture the results. The step closes when the test log shows every denied path failing as designed.
Run a pilot telemetry topic
A limited set of substation or meter telemetry feeds into a pilot topic with agreed partition count, retention and key design. The team measures ingestion, consumer behavior and failover on broker restart, and checks that the data reaching the pilot consumers matches the source records. Operators review whether the timestamps, naming and access model fit how control room staff will use the stream. The output is a signed pilot report listing defects, configuration changes and the production topic settings approved for the next phase.
Replicate to the second control center
DeployOpen configures MirrorMaker 2 on Kafka Connect to replicate approved topics, their configuration, consumer group offsets and ACLs to the second control center cluster, with the replication path reviewed against the Real-time data availability expectations in CIP-012. The team tests planned and unplanned failover, including how consumers resume from replicated offsets and how replicated ACLs behave in the target. Confirm the replication link is protected in transit and that the secondary cluster is subject to the same security controls. The step closes with a recorded failover test and a list of topics that do not yet replicate.
Monitor and collect compliance evidence
The team deploys metrics, alerting and log collection for brokers, controllers, Connect workers and replication lag, and defines who responds to each alert. It also sets up a recurring evidence set: configuration exports, ACL snapshots, patch and upgrade records, access reviews and test results, mapped to the CIP requirements in scope. Your compliance team confirms which evidence the Compliance Enforcement Authority is likely to request, since CIP has no certificate and compliance is judged through audits and spot checks. The step closes when a sample evidence package has been produced and reviewed by your compliance lead.

How Kafka carries smart meter and SCADA data without crossing the wrong boundary

Every hop between a meter or remote terminal unit and a control center is a trust boundary, and each one needs its own explicit identity and authorization decision. In a typical layout, meters and RTUs report to collection gateways or protocol converters inside the operational technology (OT) network. Those gateways publish to Kafka topics on a cluster that sits in a controlled zone, and downstream consumers read from those topics: historians, analytics platforms, and control center applications. Each arrow in that path crosses a different set of owners, networks and risks, so the design should show where every producer, consumer and broker identity sits before any topic is created. Which hops are permitted to write to OT-facing topics, and which may only read, are questions for the architecture review, not for the first deployment.

Kafka gives you the mechanisms, but it does not decide the boundaries for you. Clients and brokers can authenticate with SSL or one of the documented SASL mechanisms (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or OAUTHBEARER), and traffic can be encrypted with SSL/TLS. Security is optional in Kafka, so a cluster can run with unsecured or mixed listeners unless you require otherwise. Authorization in KRaft clusters is stored as ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm that this setting is false or unset in your environment, and write an ACL for every producer and consumer group that the design names. Review the Kafka authorization documentation before finalizing principal naming.

The practical test is whether a compromised gateway or historian account could publish a command or change a topic that a control application trusts. Separate the topics that carry field telemetry from the topics that feed control center decisions, give each gateway a principal limited to its own topics, and keep administrative principals out of routine clients. Confirm which listeners are reachable from the OT side and which only from IT, and record that mapping as evidence. Cross-site replication for control centers should be planned with the same discipline, because MirrorMaker 2 copies topic configuration, consumer offsets and ACLs between clusters. A mirror that replicates permissions carries the same boundary risk to the second site.

What NERC CIP and NIS2 require from the platform around Kafka

A Kafka cluster rarely falls under NERC CIP or NIS2 on its own. Its scope follows from the assets it touches, the access paths into it and the suppliers who build and patch it. The four decisions below set the boundary that auditors and national authorities will examine. Neither regime issues a certificate, so the evidence you keep matters as much as the design.

CIP scope under CIP-002-5.1a

Start by deciding whether the systems that produce or consume Kafka data are BES Cyber Systems. CIP-002-5.1a sorts them into high, medium or low impact using bright-line criteria in its Attachment 1. High impact mainly covers qualifying control centers, and medium impact includes, for example, generation of 1500 MW or more at one plant. A telemetry broker feeding a control center may inherit that classification, so document the impact rating before choosing a Kafka topology. Ask whether each cluster, proxy and connector is in scope or only data flowing to it.

Perimeter and remote access rules

CIP-005-7 requires routable-connected assets of high and medium impact BES Cyber Systems to sit inside an Electronic Security Perimeter. All Interactive Remote Access must pass through an Intermediate System with encryption and multi-factor authentication. For Kafka, this means brokers, Schema Registry, Connect workers and admin consoles must be placed deliberately, and vendor or engineer access for upgrades must use the same path. Confirm the exact applicability tables with your Regional Entity before you draw the boundary.

Vendor plans for the Kafka distribution

CIP-013-2 requires supply chain risk management plans covering vendor incident notification, vulnerability disclosure, software integrity verification and coordination of vendor-initiated remote access. The plan is reviewed at least every 15 calendar months. Apache Kafka is Apache-2.0 software with no paid edition from the project, so the supplier chain is your build pipeline, your container images and any partner who patches the cluster. Record where binaries come from, how checksums are verified and who receives patch notices.

NIS2 risk management and reporting

Under NIS2, essential and important entities face the same risk management measures under Article 21 and incident reporting under Article 23. Essential entities face ex ante supervision, while important entities face ex post supervision. Energy is a high-criticality sector in Annex I, and Commission Implementing Regulation (EU) 2024/2690 sets technical requirements for some digital providers. Decide which Kafka-related systems fall under your national transposition law, and map reporting timelines to your incident response runbooks.

Evidence instead of certificates

Neither NERC CIP nor NIS2 issues a certificate. Compliance is monitored through CIP Compliance Audits, Spot Checks and Self-Reports, and under NIS2 by national authorities through inspections, security audits and information requests. DeployOpen prepares systems, controls and evidence for these assessments but does not issue audit reports or attestations. Ask each reviewer which evidence they expect, such as configuration baselines, access logs, change records and vendor review minutes.

Questions utilities ask before hiring help for Kafka

What drives the cost and effort of a utility Kafka build?

The largest drivers are the number of impact-classified systems in scope, how many data sources feed the cluster (smart meters, SCADA historians, substation gateways), and how many sites need replication. Each source adds connector work, schema agreement and testing. Cost splits into the services you pay for (scoping, design, build, testing, documentation, handover) and your internal time (asset lists, network diagrams, change approvals, operator sign-off). Ask any quote to separate one-time build work from recurring support, upgrades and testing retainers, and to state which systems are excluded. Kafka itself is licensed under Apache-2.0 with no paid edition from the project, so license cost is not the main driver. Hardware, storage for retention and staff time are the other components to request in writing.

Can our utility run a self-hosted Kafka cluster internally?

Many utilities can, provided they have engineers who understand Kafka's operating model and the OT boundary they must respect. The Kafka upgrade notes state that Apache Kafka 4.0 supports only KRaft mode, so the cluster must be on KRaft from the start, and Kafka 4.x requires Java 17 for brokers and Kafka Connect. Rolling upgrades are supported from each of the last three releases to the latest version, which means a standing upgrade routine is needed. Assess whether your team can own patching, certificate rotation, topic and ACL administration, monitoring and recovery testing. If any of these would sit with a small team, consider a retainer for upgrades and continuous testing alongside internal ownership.

How does impact classification change the timeline and approvals?

Impact classification under CIP-002-5.1a decides which CIP requirements apply to the systems that a Kafka cluster touches. High impact BES Cyber Systems mainly cover qualifying control centers, and medium impact includes, for example, generation of 1500 MW or more at one plant or transmission facilities at 500 kV or higher. Systems at higher impact carry more requirements, such as Electronic Security Perimeter placement under CIP-005-7, so design reviews and change approvals take longer. The classification is your entity's determination, so confirm it with your compliance team and regional entity before design starts. We cannot give a duration without the scope, and any schedule should follow the written approval steps your organization requires.

What does the work need from our utility before it starts?

The build depends on accurate inputs from your side. Ask for an inventory of BES Cyber Systems and associated Electronic Access Control or Monitoring Systems, Physical Access Control Systems and Protected Cyber Assets, with their impact categories. Network diagrams should show where the OT and IT networks meet, which routable connections exist, and where any Interactive Remote Access enters. You also need data source documentation for each meter or SCADA feed, including message formats, expected volumes, retention needs and owners. Confirm who approves firewall and boundary changes and who signs off on cutover. Missing inventories are the most common reason design timelines slip, so agree on them in the written scope.

How do we verify that replication to a second control center works?

Verification has to be tested, not assumed. MirrorMaker 2, built on Kafka Connect, replicates topics with their configuration, consumer groups with offsets and ACLs between clusters while preserving partitioning, according to the Kafka geo-replication documentation. Test plans should cover planned failover and failback, a simulated loss of the primary site, replication lag under expected load, and whether consumers resume at the correct offsets. Confirm that ACLs and encryption settings replicate as intended, and that the secondary site meets the same access and monitoring controls. CIP-012-2, enforceable 1 July 2026, adds protection of the availability of Real-time data between Control Centers, so record the test results as evidence for that requirement.

Does DeployOpen issue a NERC CIP or NIS2 certificate for our Kafka platform?

No. There is no CIP certificate, and compliance is monitored by the Compliance Enforcement Authority through audits, spot checks, self-certifications, self-reports and investigations. NIS2 also creates no certificate of compliance, and national competent authorities oversee compliance through inspections, security audits and administrative fines. DeployOpen prepares systems, controls and evidence for these assessments, but independent auditors and certification bodies issue any audit reports or attestations. Ask us to map your evidence to the applicable CIP requirements, and confirm with your regional entity or national authority which evidence they accept.

Which parts of the design are fixed by the standards, and which are our choice?

The standards set outcomes and evidence expectations, not product settings. Your choices include the KRaft cluster layout, the authentication mechanisms (the documented SASL options include SCRAM-SHA-512 and OAUTHBEARER, alongside GSSAPI and SSL), the authorization model, network segmentation and the replication topology. Kafka authorization in KRaft clusters uses the StandardAuthorizer, which stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Ask us to document each choice against the requirement it supports, and to confirm the vendor or standard references for each control before relying on them.

What evidence a grid auditor will ask Kafka to produce

A grid auditor will not accept a statement that Kafka is secure. The Compliance Enforcement Authority, a Regional Entity or NERC, monitors compliance through Compliance Audits, Spot Checks, Self-Certifications, Self-Reports, periodic data submittals and investigations, and each of these asks for records that show what the system was configured to do, who could reach it and how it changed over time. For a Kafka cluster that carries BES Cyber System data, the useful evidence sits in four places: the broker and controller configuration, the TLS and SASL listener settings, the ACL state held in the KRaft metadata log, and the monitoring and audit logs that show access and change events. Before building any evidence pack, confirm the exact artifacts and retention period with your Regional Entity or compliance advisor, because the applicable requirements depend on the impact rating of each system and the CIP version in force for it.

Kafka can produce much of this on request if it is set up for it. An ACL export from the StandardAuthorizer, taken on a schedule, shows which principals held which operations on which topics and consumer groups, and a dated copy lets you show state at a past point, not only today. The allow.everyone.if.no.acl.found setting should be recorded with its value, since the default behavior affects what an auditor can read from the ACL export. Listener configuration in files such as server.properties, with the ssl.* and sasl.* keys, documents encryption and authentication for each listener. Store these exports and configuration files under version control with change history, so the record of who altered a setting and when is separate from the cluster itself. Audit logs from the authorizer and monitoring alerts on failed authentication should be forwarded to a system outside the cluster, so the evidence survives a broker failure or a compromised host.

Plan these records as part of the design, not as a document written before an audit. Decide which Kafka clusters fall inside the scope of a CIP standard, then define who owns each evidence artifact, how often it is generated and where it is stored, and test that you can reproduce a configuration as it stood on a given date. Vendor responsibilities under CIP-013 also belong in this pack, such as how you verify the integrity of the Kafka release you deploy and how vendor-initiated remote access to the cluster is logged. Kafka itself does not provide a compliance certificate, and DeployOpen does not issue audit reports. The aim is a clear, reproducible set of records that the entity can present, which the entity's own auditors and the Compliance Enforcement Authority can then assess.

How DeployOpen works on this

DeployOpen plans, implements, hardens, documents and supports self-hosted Apache Kafka inside the utility's own infrastructure, including its cloud accounts, data centers or isolated networks. Work starts with a written scope that separates OT telemetry flows from IT consumers and records which systems fall under the CIP or NIS2 obligations that apply.

How we scope

Inventory the telemetry sources, SCADA and smart meter feeds, consumers and retention needs for each site.
Map which systems and data flows fall under the applicable NERC CIP or NIS2 obligations, confirmed with your compliance lead.
Agree topic plans, partition counts, KRaft topology and cross-site replication scope in writing before work starts.

How we build

Deploy KRaft clusters on your infrastructure with TLS and SASL authentication, ACLs and network segmentation between OT and IT zones.
Configure MirrorMaker 2 replication for control center sites, with topic configuration, consumer offsets and ACLs preserved.
Test failover, broker loss, certificate rotation and recovery steps, and record the results as evidence for your assessments.

How we hand over

Deliver architecture diagrams, runbooks, access control matrices and configuration baselines your team can maintain.
Walk operators and security staff through monitoring, alerting, backup and restore procedures on your own systems.
Document the change control and vendor patch process so your supply chain risk procedures can reference it.

How we support

Plan upgrades along the supported Kafka 4.x release path, tested in a non-production environment first.
Offer ongoing support, upgrades and continuous testing as a retainer, scoped separately from the initial build.
Review each release and advisory against your deployed configuration and agree remediation timelines with your team.

Apache Kafka for energy and utilities