Apache Kafka for manufacturing: why the factory floor needs its own streaming layer
A manufacturer with PLCs, historians, MES and quality systems usually has the same problem: plant events need to reach enterprise systems, but the OT network cannot be opened to them directly. Kafka fits as the buffered event layer between plant and enterprise. Machine data arrives at a plant-side cluster, is held there for as long as the plant needs, and is replicated outward only through a controlled path. Apache Kafka is released under the Apache License 2.0 and the project offers no paid edition, so the software cost is zero. What you pay for is the design around it: the plant cluster, the mirroring topology, the network segmentation, the access model and the operations to keep it running.
Three changes follow from that design. First, each plant or line gets a local cluster so production keeps flowing if the link to headquarters drops. Second, a central cluster receives selected topics through MirrorMaker 2, which replicates topic configuration, consumer groups and ACLs along with the data. Third, the boundary between OT and IT becomes an explicit component you can name, monitor and audit. Teams should confirm early which zones and conduits the plant's segmentation model defines, since the Kafka listeners, MirrorMaker 2 connections and any MQTT bridge all have to sit on a documented side of that boundary. The Kafka geo-replication documentation describes the mirroring behavior you will be designing around.
Plan the decisions before the first broker is installed. Decide which machine signals become topics, who owns each topic, how long plant data is retained for quality analysis, and which side of the boundary each bridge runs on. Confirm with your OT and security teams how the plant network is zoned, and agree which direction traffic may flow. These choices shape the sizing, the authentication model and the audit evidence later, so changing them after go-live is costly.
Which manufacturing requirements shape the Kafka build
This table compares the frameworks and regulations that touch factory data flows, the source behind each one, and what it means for topic design, network placement, patching and retention. Read each row as a question to answer before the cluster layout is fixed, not as a checklist that Kafka satisfies on its own.
| Requirement | What it asks for | Effect on the Kafka build |
|---|---|---|
| ISA/IEC 62443 series | Groups standards into general, policies and procedures, system and component requirements for industrial automation and control systems (ISA series page) | Defines zones and conduits: place the Kafka plant cluster in a zone with documented boundaries and control traffic crossing into the central cluster |
| ANSI/ISA-62443-2-1-2024 | Security program requirements for asset owners running IACS | Assign owners for each topic and bridge, and document who patches brokers and MQTT bridges on the plant side |
| ANSI/ISA-62443-3-2-2020 | Security risk assessment during system design | Assess data flows before topic design: which signals cross zones, and which topics may hold safety or production-critical values |
| ISA-62443-3-3-2013 | System security requirements and security levels | Set the target security level per zone, then choose TLS, SASL and ACL settings on each Kafka listener to match it |
| NIST SP 800-82 Rev. 3 (September 2023) | OT security guidance covering SCADA, DCS, PLCs and building systems, with OT performance, reliability and safety needs | Design for OT availability: plant clusters must keep collecting when the central link is down, and patching windows must respect line uptime |
| NIS2 Annex II, manufacturing | Lists manufacturing as a critical sector, covering NACE Rev. 2 divisions 26 to 30 and medical devices (Directive (EU) 2022/2555) | Confirm whether your entity is in scope, then map Kafka logs and alerts to incident handling so significant incidents can be reported in time |
| NIS2 reporting clocks | Significant incidents: early warning within 24 hours, notification within 72 hours, final report within one month (Article 23(4)) | Keep broker, bridge and ACL audit logs for at least the investigation window, and automate alerts from the Kafka cluster to your incident team |
| Cyber Resilience Act (Regulation (EU) 2024/2847) | Article 14 reporting of actively exploited vulnerabilities and severe incidents has applied since 11 September 2026 (Article 71(2)); broader obligations from 11 December 2027 | If you ship machines with embedded Kafka clients or gateways, track component versions and vulnerability disclosures so you can report on time |
| Machinery Regulation (EU) 2023/1230 | Applies from 14 January 2027; Annex III, 1.1.9 requires machinery to resist hazardous outcomes when another or remote device connects | Treat any Kafka connection to machine controllers as a connection to safety-relevant equipment: one-way data paths and no write-back topics into control logic |
| CMMC levels (32 CFR Part 170) | Level 1 uses 15 FAR 52.204-21 requirements, Level 2 uses 110 NIST SP 800-171 Rev. 2 requirements for CUI, Level 3 adds 24 NIST SP 800-172 requirements (32 CFR 170.14) | Where defense work data flows through Kafka, separate those topics and cluster access, and confirm the applicable level and phase status with your contracting officer |
How plant clusters mirror to a central Kafka cluster
The edge or plant cluster pattern keeps production data moving inside the site when the wide area network fails. Each plant runs its own Kafka cluster close to the lines, so machine telemetry, alarms and quality records land in local topics even if the link to headquarters or a cloud region is down. A central cluster then receives copies of selected topics once connectivity returns. The design question is which workloads must keep running during an outage. Local control, alerting and buffering usually do, while enterprise reporting can tolerate delay. Ask the plant engineering team to list those workloads before sizing the local cluster and the link.
MirrorMaker 2 is the replication tool built on Kafka Connect that moves data between clusters. According to the Kafka geo-replication documentation, it replicates topics along with their configuration, consumer groups with their offsets and ACLs, and it preserves partitioning, so records keep their partition assignment at the target. Plan the mirroring direction, the topic filters and the naming of replicated topics before deployment. Decide which side holds authority for each dataset, because offsets and ACLs replicate in one direction and a site that writes to both clusters creates conflicts that the design must resolve. Confirm these replication behaviors and any configuration keys against the documentation for the release you deploy, using the Kafka geo-replication documentation as the reference for the 4.3 line.
Several items need confirmation during design. Check the ACL and credential handling for the mirror connectors, since the replication identity needs read access on the source and write access on the target, and that access should be scoped to the mirrored topics only. Test failover and failback with a plant cut-off in a non-production environment, and measure how far the central copy trails the plant during normal operation. Record the decisions in the design document so auditors and operators can see which site is authoritative for each topic.
Which Kafka modules and settings matter on the plant side
A plant-side Kafka deployment is a set of components that each carry a different risk: how machine data gets in, where controllers sit, how clients prove identity, who may touch which topics, what crosses to the central cluster and how long quality data is kept. Review each one with the people who own the machines, the network and the historian before the design is fixed.
MQTT bridge for machine telemetry
PLCs and machine gateways usually publish over MQTT, not the Kafka protocol, so a bridge or connector translates topics into Kafka records. Decide where it runs (inside the OT zone or in a DMZ), how it reconnects after a network drop, and whether it preserves the machine timestamp or stamps arrival time. Confirm message keys, batching behavior and error handling with the connector vendor and the plant's historian team before testing against live lines.
KRaft controller placement per site
In KRaft mode the controllers hold cluster metadata, so each plant cluster needs its own controller quorum that survives the loss of a single site link. Place controllers on separate hosts, keep the quorum size odd, and avoid putting all of them in one cabinet or one network segment. Confirm the number of controllers and their failure behavior against the Kafka deployment guide for your version before sizing hardware.
TLS and SASL for client access
Plant clients include historians, edge analytics and the bridge itself, and each should authenticate with its own identity. Use TLS for traffic between clients, brokers and tools, and choose SASL/SCRAM-SHA-512 or SASL/OAUTHBEARER for credentials, depending on whether your identity provider can issue tokens to devices and services. Avoid shared accounts across lines, because one leaked secret would expose every topic.
StandardAuthorizer ACLs in the metadata log
In KRaft clusters the StandardAuthorizer stores ACLs in the metadata log, so access rules replicate with the cluster. Keep allow.everyone.if.no.acl.found left off, so that any topic or group without an explicit ACL is denied to everyone except super users. Write ACLs per client identity and per topic prefix for each line, and review them when a machine or bridge is retired.
MirrorMaker 2 topic filters
MirrorMaker 2 replicates topics, their configuration, consumer offsets and ACLs to the central cluster, so a broad filter can send raw sensor streams and maintenance-only topics off site. Define an allow list of topic patterns for quality and summary data, exclude raw OT topics unless the data owner approves, and confirm with the network and security teams that the replication path is permitted by the segmentation policy. Test filters with a sample line before enabling them.
Retention for quality data
Quality analysis often needs months of machine history to trace a defect back to a batch, while raw sensor topics can grow fast and fill plant disks. Set retention per topic rather than cluster-wide, and separate the long-retention quality topics from high-volume telemetry. Confirm the required lookback period with the quality department and any record-keeping rule that applies, and size storage against the actual message rate from a pilot line.
Rollout phases for a plant-first Kafka deployment
The sequence moves from knowing what sits on the factory floor to a pilot line that streams into a central cluster, then hardens the segmentation and hands the platform to the people who will run it. Each phase ends with a documented output that the next phase depends on.
Inventory OT assets and zones
Plant engineering, controls engineers and the DeployOpen team list the machines, PLCs, historians, MES servers and gateways that will produce or consume data, then group them into zones by function and network location. The client's OT team owns the asset data because only they can confirm what runs on each line and which vendors need remote access. The step closes when the inventory is signed off by OT engineering and each asset has a zone and an owner.
Assess risk and design zones and conduits
Using the zone list, the client's security and OT teams run a risk assessment aligned with the ISA/IEC 62443 approach, then decide which traffic may cross each zone boundary, through which conduit, and in which direction. DeployOpen drafts the data flow diagrams, the Kafka listener and authentication plan per zone, and the ACL model. The output is a zone and conduit design that the plant manager, OT security lead and IT architecture lead approve before any cluster is built.
Build the pilot plant cluster
DeployOpen installs a KRaft-mode Kafka cluster inside the pilot plant's segment, using the agreed listener, TLS and SASL settings and the topic plan for one production line. The client's infrastructure team provides the hosts, storage, time synchronization and monitoring, and the plant IT lead approves the access model. The step is complete when the cluster passes a documented connectivity and failover test and the topic configuration matches the signed design.
Connect pilot line producers and gateways
The pilot line's MQTT bridge or gateway is connected to the plant cluster, and consumers are limited to the systems named in the conduit design. Controls engineers verify that the data path does not allow writes back into control networks, and that gateway credentials are scoped to the pilot topics. The output is a logged test run showing that the expected messages arrive, retention behaves as designed and no unapproved flows exist.
Mirror the plant cluster to central
The plant cluster is replicated to the central cluster with MirrorMaker 2, which carries topic configuration, consumer offsets and ACLs along with the data. The IT platform team owns the central cluster and the replication links, and the OT team confirms that only the approved topics leave the plant. Acceptance means the mirrored topics match source partitioning and lag stays within a level the teams agree on in advance.
Integrate historian and MES consumers
Historian and MES integrations are built as consumers on the plant or central cluster, with service accounts, quotas and topic access scoped to each system. The plant's MES and historian owners validate data formats, timestamps and batch identifiers against their source records. The quality and operations leads sign off the first production data set, and the step closes when reconciliation between Kafka and the historian shows no unexplained gaps.
Harden segmentation and run tests
DeployOpen reviews the firewall rules, conduit enforcement, authentication settings and logging against the signed design, then runs documented tests for unauthorized access, broker exposure and recovery from a lost link. The OT security lead decides which findings must be fixed before cutover. Sign-offs needed before cutover include the zone and conduit design, the risk assessment, the access model, the retention plan and the rollback procedure.
Hand over runbooks and ownership
DeployOpen delivers runbooks for broker operations, certificate and credential rotation, patching, backup and restore, and change requests across OT and IT. Named client staff from both teams take on day-two responsibility, and the handover is recorded with a clear split between plant-side and central-side duties. The step closes when those staff have run the runbooks in a supervised session and the ownership matrix has been approved.
Keeping quality data long enough for root-cause analysis
A quality investigation usually starts weeks after a defect is found, when a customer complaint or a field return reveals a pattern. To trace it back, engineers need the process data that was recorded when the affected batch was made: spindle loads, oven temperatures, torque values, recipe versions and operator actions. Retention is therefore a quality decision before it is an infrastructure decision. Ask your quality and records owners how long each category of process data must remain available, and which topics fall under that requirement. The retention period is set by your quality system and any records obligations you carry, not by Kafka defaults.
Plant and central clusters usually hold data for different purposes. A plant cluster buffers machine telemetry close to the line, so it can keep data for the period needed to recover from a WAN outage and to serve local analysis. The central cluster, fed by mirroring, is where cross-site investigations run, so its retention often needs to be longer and its storage sized for the combined output of every plant. Confirm which topics are mirrored, whether compacted topics are needed for reference data such as recipes, and whether archival to object storage is part of the design rather than an afterthought.
Storage sizing depends on four inputs: message rate, average record size, replication factor and retention period. Multiply the sustained rate by record size, then by replication factor and by the retention window, and add headroom for growth, compaction overhead and rebalancing. Do not assume these values. Measure message rates and record sizes on a representative line during design, and confirm the replication factor your availability requirements call for. The Apache Kafka documentation on geo-replication explains how mirrored data adds storage at the destination, which your sizing must include.
Compliance and regulatory obligations to map before the build
Six obligations shape the Kafka design before the build starts. Some apply to the products you make, some to plant security and safety, and some only if you sell to defense or regulated medical customers. Map each one to the topics, networks and records it touches, and have legal and compliance teams confirm applicability. DeployOpen prepares systems, controls and evidence for assessments; it does not issue audit reports or certifications.
ISA/IEC 62443 zones and conduits
ISA/IEC 62443 sets security program and system requirements for industrial automation and control systems. For a plant Kafka deployment, the design question is zones and conduits: which systems sit in each zone and which data flows cross the boundary. A mirror link from a plant cluster to a central cluster is a conduit, so its TLS, SASL, ACLs and monitoring need documented evidence. Ask the asset owner which security levels apply to each zone before choosing listeners and network paths.
NIS2 incident reporting timelines
NIS2 Annex II lists manufacturing as a critical sector, covering NACE Rev. 2 divisions 26 to 30 and medical devices. In-scope entities must send an early warning within 24 hours, an incident notification within 72 hours and a final report within one month. Important entities face fines of at least EUR 7 million or 1.4% of worldwide turnover. A Kafka outage or unauthorized topic write can start that clock, so the detection, log export and on-call paths must be designed before go-live. Confirm your entity classification with counsel.
CRA reporting of exploited vulnerabilities
The Cyber Resilience Act applies to products with digital elements from 11 December 2027, but Article 14 reporting of actively exploited vulnerabilities and severe incidents has applied since 11 September 2026. If your factory ships software or connected equipment that embeds Kafka or Kafka client code, you may carry these reporting duties as a manufacturer. Map which products include Kafka components, track upstream releases on the Kafka downloads page, and document who triages vulnerabilities in your supply chain and how reports reach the authorities.
Machinery Regulation essential requirement
Regulation (EU) 2023/1230 applies from 14 January 2027. Its Annex III essential requirement 1.1.9 expects machinery to be protected so that connecting another device or a remote device does not lead to a hazardous situation. A broker or MQTT bridge that accepts commands from a plant network creates that connection path. Document which topics can influence machine control, keep command paths separate from telemetry, and confirm with your conformity assessment route how the requirement applies to each machine.
CMMC Level 2 scope for defense suppliers
CMMC under 32 CFR Part 170 sets three levels for defense contractors. Level 2 applies the 110 requirements of NIST SP 800-171 Rev. 2 to controlled unclassified information, so a Kafka cluster that carries CUI falls inside the assessed boundary. The Department of War's 13 July 2026 release reports that CMMC Phase II requirements were suspended. Check the current status with your contracting officer before scoping, because the timing of your obligation depends on it, and keep CUI topics out of shared clusters where possible.
21 CFR Part 11 electronic records
21 CFR Part 11 sets requirements for electronic records and electronic signatures where FDA rules apply, such as batch or quality records in a regulated plant. Kafka topics can hold those records in transit, so they need controlled access, change history for configuration, and retention that matches the record schedule. ACLs control who can read and write, but the audit trail, validation and signature controls usually sit in the systems around the cluster. Ask quality assurance which records pass through topics before setting retention.
Questions manufacturers ask before hiring a Kafka partner
What drives the cost and effort of a plant Kafka build?
The main drivers are the number of sites, the number and type of machine data sources, how many OT networks must be crossed, and how much of the work is new versus adapting existing systems. Software is free: Apache Kafka is released under the Apache License 2.0 and the project itself offers no paid edition, so the spend goes to people, infrastructure and support. Ask any partner to split a quote into one-time work (architecture, bridge and connector build, network segmentation design, testing, runbooks) and recurring work (upgrades, patching, monitoring, retainers). Each line should name the sites, sources and retention targets it covers.
How long does a plant Kafka project take, and what sets the timeline?
Duration depends mainly on how many plants are in scope, how well the OT network and firewall changes are understood before work starts, and how many machine protocols need a bridge. Release cadence also matters: the Kafka project aims for three releases a year, and the time-based release plan supports rolling upgrades from the last three releases to the latest version, so a build should start on a supported release that will still be supported at go-live. Ask for a written schedule with dated gates for network approval, a pilot line and cutover, and confirm which items depend on your change board.
What must our OT, IT and quality teams provide before work starts?
Each team owns a different input. OT engineering provides PLC and SCADA data points, protocols, sampling rates and maintenance windows. IT provides identity sources, certificate authority access, DNS, storage and monitoring standards. Quality provides the data they need for analysis, retention periods tied to batch records and any record-keeping rules that apply. Security should define the zone and conduit model, which the ISA/IEC 62443 series describes at a high level. Ask the partner for a written input list with an owner and due date for each item.
Can an internal team run a Kafka plant platform on its own?
Yes, if the team has the skills and time to operate it. Running Kafka means monitoring brokers and controllers, handling KRaft metadata, managing topics and ACLs, rolling upgrades and recovery testing, and keeping bridges and mirror links healthy across sites. Many plants have strong OT skills but limited streaming experience, so a common split is an internal team owning daily operations while a partner handles the initial build, runbooks and scheduled upgrades. Decide this before the build, because the design choices (cluster layout, access model, backup approach) determine how hard day-two work will be.
How are air-gapped or restricted plant networks handled?
Plants that cannot reach the internet need every artifact moved in on approved media: the Kafka release, operating system packages, container images, connector plugins, and JVM. Each package should be checked for integrity and authenticity before it crosses the boundary, which is the control NERC CIP-013-2 describes for vendor software and patches, even though it is an energy standard. Some vendors reserve air-gapped support for paid tiers, so check the license of every component you bring in. Ask the partner how it will produce an offline bundle and how updates will be carried across the boundary.
What does a fixed-scope quote for a Kafka build cover?
A fixed-scope quote should list the sites, data sources, protocols, topics or data products, cluster layout, security controls, integrations, testing and handover in writing before work starts. It should also name what is excluded, such as new PLC programming, MES changes or hardware purchases, and state the acceptance tests and the deliverables for each phase. DeployOpen plans, implements, hardens, documents and supports self-hosted platforms in your own infrastructure and most engagements are fixed-scope, so quotes follow from the written scope rather than a price list. Ask what changes require a new quote and how ongoing support is priced if you want a retainer.
Which compliance rules affect a manufacturer's Kafka design?
For EU manufacturers, NIS2 lists manufacturing as a critical sector and sets incident reporting timelines: an early warning within 24 hours, a notification within 72 hours and a final report within one month. The Cyber Resilience Act applies to products with digital elements from 11 December 2027, and manufacturer reporting duties under its Article 14 have applied since 11 September 2026. The Machinery Regulation adds a requirement from 14 January 2027 that machinery is protected against connection of other or remote devices. Kafka does not make a plant compliant on its own, so ask a partner which evidence the platform produces and which obligations remain with your team.
How DeployOpen scopes a Kafka build for a plant
A written scope is agreed before any work starts, and for a plant that scope is shaped by discovery. In discovery, DeployOpen inventories the OT zones that will feed or consume Kafka, the data sources in each zone (PLCs, SCADA historians, MES and quality systems, and any MQTT brokers already in use), and the existing historians whose data must keep flowing. The output is a zone and flow map that shows which network boundaries each topic crosses, which systems are allowed to publish, and where the edge and plant clusters will sit relative to the central cluster. Confirm early whether your plant networks follow an ISA/IEC 62443 zone and conduit model, because that decides where MirrorMaker 2 links may be opened and who approves them.
Discovery also settles the decisions that change cost and effort: topic and partition plans per line, retention for quality analysis, which ACLs the StandardAuthorizer will enforce in each cluster, and how credentials for MQTT bridges and Connect workers are issued and rotated. Deliverables typically include the architecture and zone diagrams, configuration and build records, runbooks for broker restarts, partition moves and MirrorMaker 2 failover, test and verification evidence, and a handover session with your operations team. Ask for each item to be listed by name in the written scope, along with which party owns the evidence after handover.
Ongoing support, upgrades and continuous testing are available as retainers, which you can add to the scope or agree separately. Upgrades follow the supported 4.x releases: according to the Kafka time-based release plan, rolling upgrades are supported from each release in the past year to the latest version, so the upgrade path for your plant is set by the release you deploy and the schedule you agree. Confirm with DeployOpen which supported release the build targets and how plant change windows will be handled.
How DeployOpen delivers Kafka for manufacturing
DeployOpen plans, implements, hardens, documents and supports self-hosted Apache Kafka inside the infrastructure you control, including plant, edge and central sites. For manufacturing, the work starts with the factory network design and the data each line produces, then builds the cluster around those constraints.
How we scope
- Written scope covers plant sites, edge clusters, central cluster, and the machine data sources to be ingested.
- Scope records MQTT bridge choices, topic and partition plans, retention periods per data class, and OT zone boundaries.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before any build work starts.
How we build
- Deploy Kafka in KRaft mode on the chosen platform, with TLS and SASL authentication and ACLs defined from the start.
- Configure MirrorMaker 2 between plant and central clusters, with topic, offset and ACL replication tested end to end.
- Test failure cases such as WAN loss, broker restarts and consumer lag, and record the results against the agreed scope.
How we hand over
- Deliver architecture and network diagrams showing where Kafka sits relative to each OT and IT zone boundary.
- Provide runbooks for topic creation, ACL changes, certificate rotation, upgrades and mirror recovery, written for your team.
- Walk your engineers through operations, monitoring and evidence collection so day-two work stays with your staff.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
- Upgrade planning follows the Kafka release plan, with the supported version path confirmed before each change window.
- Ask in any quote which patching, vulnerability review and on-call responsibilities sit with DeployOpen and which stay with your team.
