Why event streaming in plants needs OT-specific design
Plant data is usually trapped in PLCs, historians and SCADA systems, while IT teams want it in analytics platforms and maintenance systems. The short answer is that a plant streaming design needs to be built around the operational technology (OT) network first and the analytics demand second. A streaming stack that works well in a corporate data center can break production networks if it is connected without zones, buffering and clear ownership of each hop. The frame to design against is the ISA/IEC 62443 series, which organizes requirements for industrial automation and control systems, and NIST SP 800-82 Rev. 3, which covers OT performance, reliability and safety alongside security. Both treat availability of the process as a first-class requirement, and a streaming pipeline has to respect that. The same design choices used in IT do not carry over unchanged. Plant networks are segmented into zones and conduits, and data crosses from the control zone to the enterprise only through controlled points. That means the edge collector, the MQTT or OPC UA gateway and the plant-to-enterprise broker each need a defined role, a defined direction of traffic and an owner. A single cluster stretched across both sides is hard to justify under that model. Buffering is the other gap: when a WAN link or the enterprise side drops, edge sites must store telemetry locally and replay it in order without starving the control network or overwhelming the link on recovery. Confirm how much local storage each site has, how long it must hold data and what happens to timestamps during replay. Platform fit follows from those constraints. Lightweight brokers at the edge such as NATS, which runs as a small single server process with MQTT support as configuration, or RabbitMQ with its MQTT plugin, are often evaluated for site-level collection. At the plant-to-enterprise boundary, Apache Kafka is commonly the backbone for analytics and maintenance integration, with the Kafka security model (TLS, SASL and ACLs) enforced on each listener. Confirm protocol support, version lifecycles and resource needs with each vendor and against your own plant pilot before committing to a design.
Which requirements shape the streaming design in manufacturing
This table maps each regulatory and technical requirement that bears on plant telemetry to the design decision it drives. Read each row as a question to answer before choosing broker placement, zones, authentication and logging. The main takeaway is that the OT security standards set the architecture, while NIS2 and the CRA set reporting clocks and evidence duties.
| Requirement | What it asks for | What it means for the build |
|---|---|---|
| ISA/IEC 62443-2-1-2024 (asset owner programs) | A documented security program for the asset owner, covering roles, policies, and how plant systems are maintained over time | Assign owners for each zone, define who may change broker ACLs and certificates, and document how streaming components are added to the plant asset inventory |
| ISA/IEC 62443-3-2-2020 (risk assessment for system design) | A risk assessment that defines zones and conduits before the system design is fixed | Draw zones around PLCs, historians and the broker tier, then define each conduit (MQTT, Kafka, or NATS flows) with its allowed direction, ports and data types |
| ISA/IEC 62443-3-3-2013 (system security requirements and levels) | Target security levels per zone and the technical controls needed to reach them | Place brokers in a DMZ or demilitarized conduit, require TLS and authenticated clients on every listener, and avoid unauthenticated plant-floor listeners |
| NIST SP 800-82 Rev. 3 (OT security guide) | OT security guidance that also weights performance, reliability and safety over pure confidentiality | Size brokers for buffering during link loss, avoid security controls that block control traffic, and test failover without disturbing control loops |
| NIS2 Annex II (manufacturing sector scope) | Lists manufacturing divisions NACE 26 to 30 as a critical sector, which may bring in-scope entities under the directive | Confirm with counsel whether the entity and its plants fall in scope, then record which systems and data flows support the regulated functions |
| NIS2 Article 23(4) reporting timelines | Early warning within 24 hours, incident notification within 72 hours and a final report within one month for significant incidents | Log broker authentication failures, ACL denials, certificate events and link outages with timestamps that can be exported within those windows |
| CRA Article 14 reporting (applies since 11 September 2026) | Manufacturers report actively exploited vulnerabilities and severe incidents for products with digital elements | Track versions of brokers, bridges and edge gateways you build or supply, and keep a process to report exploited flaws in products you place on the market |
| Machinery Regulation Annex III 1.1.9 (applies from 14 January 2027) | Machinery must be protected so that connecting another or remote device does not lead to a hazardous situation | Make sure new telemetry gateways and remote connections cannot write to control functions, and confirm the design with the machine manufacturer and notified body where needed |
What data moves from machines to analytics and maintenance
The most common flow is OT to IT: data leaves the plant floor and lands in historians, analytics platforms, maintenance systems and quality records. The sources differ in shape and urgency. PLC tags (values such as temperatures, cycle counts, valve states and motor currents) are read at fixed intervals and are the highest-volume source. Sensor telemetry from add-on vibration, energy or environmental sensors often arrives through MQTT gateways. Historian exports are batch or near-real-time extracts of time series already stored on site. Alarms, work orders and quality records are event-shaped: they are sparse, but each one usually matters to a downstream process such as a shift handover, a maintenance ticket or a batch release. Before choosing topics, list every source with its owner, its native protocol and its tolerance for delay and loss.
Message size and sampling rate drive most of the sizing work, and they are the numbers you most need to confirm yourself. Ask the controls team for the tag count per line, the configured scan or publish interval for each tag, whether values are sent on change or on a fixed schedule, and the typical payload with timestamp, quality code and units. Vendor defaults often publish far more often than the analytics use case needs, so a plant can generate a large share of its traffic from a few fast loops. Estimate peak bursts, such as a start-up after a stoppage that replays buffered data, rather than averages alone.
The design must keep telemetry separate from control. Telemetry flows upward and can be lost, delayed or sampled without stopping a machine. Control commands, such as setpoint changes, recipe downloads or start and stop instructions, flow back into the plant and can create hazardous states if they are delayed, duplicated or spoofed. Treat any path that writes to a PLC as a separate design decision with its own review under the zone and conduit model of ISA/IEC 62443, not as a convenient extension of the telemetry pipeline. In practice, confirm with operations which signals are read-only, which are acknowledgements, and which ever need a return path, and document that boundary before the first topic is created.
Which open-source brokers fit each layer of a plant network
Sensors, PLCs and gateways usually speak MQTT, and the edge layer is where protocol support matters most. RabbitMQ accepts MQTT clients through plugins and supports MQTT 3.1, 3.1.1 and 5.0, which covers most device firmware and gateway software in service today. NATS takes a different approach: MQTT is a configuration option on the same server process that handles its core messaging, so a small footprint can serve both device connections and internal subjects.
- Protocol versions: Confirm which MQTT version each device or gateway speaks, since RabbitMQ and NATS may differ in feature coverage for retained messages, session handling and MQTT 5.0 properties.
- Footprint: NATS documents a memory footprint of typically about 15 MB, which suits small edge nodes and industrial PCs with limited resources.
- Plugin management: RabbitMQ MQTT support depends on enabled plugins, so change control on edge hosts needs to cover plugin versions.
- Device authentication: Decide per device or per site credentials before rollout, and check how each broker enforces them.
- Node-RED for edge flows: Node-RED, licensed under Apache-2.0 with version 5.0.8 published on October 8, 2026, can normalize device payloads and publish them to either broker before the data leaves the plant.
In a pilot, test MQTT behavior with the real device fleet, not a simulator, and record which features each device relies on.
The plant bridge sits between the OT network and the enterprise network. Its job is to keep local flows running, buffer data when the link to the enterprise drops, and forward it once connectivity returns. NATS fits this role well because leaf nodes connect a local server to a larger cluster, and JetStream provides persistence so messages can be stored locally and replayed. These capabilities are configuration on the same server, which keeps the number of moving parts low at each site.
- Leaf nodes: A plant-level NATS server can run leaf node connections to a central cluster, so local subjects keep working when the upstream link is unavailable.
- Store and forward: JetStream streams bound to subjects store messages on disk, and consumers can resume from a sequence number or point in time after an outage.
- Delivery semantics: JetStream provides at-least-once delivery, so downstream consumers need to handle duplicates.
- Buffer sizing: Confirm disk capacity and retention per site against the length of outages the plant must tolerate, which is a site-specific figure you should set with operations staff.
- Ownership of the zone boundary: The bridge host is a conduit between zones, so its administrative access, logging and patching should follow the segmentation design under ISA/IEC 62443.
In a pilot, simulate a link outage and measure whether buffered data arrives complete and in order for each consumer.
The enterprise backbone carries plant data to analytics, maintenance and quality systems at scale, and Apache Kafka is a common choice here. The 4.x line runs only in KRaft mode, so ZooKeeper-based clusters must migrate to KRaft before upgrading. Brokers, Kafka Connect and tools require Java 17 from Kafka 4.0 onward. Kafka supports SSL/TLS for traffic encryption and SASL mechanisms including SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER, and the StandardAuthorizer stores ACLs in the cluster metadata log.
- Supported releases: The downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported, so confirm the target release against the time-based release plan before standardizing.
- Upgrade path: KRaft clusters older than 3.3.x should move to 3.9.x before 4.x, and rolling upgrades are supported from each of the last three releases.
- Access control: Set ACLs explicitly, because resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true.
- Authentication and encryption: Plan SASL mechanisms and TLS listeners per zone, and avoid unsecured listeners reaching the enterprise network.
- Downstream pipelines: Apache Airflow (Apache-2.0, 3.3.2 published on September 17, 2026) or Dagster (Apache-2.0, 1.13.26 published on October 9, 2026) can orchestrate batch jobs that read from the backbone into warehouses and maintenance tools.
In a pilot, confirm the Java runtime, the KRaft migration status and the ACL model with your platform team before any production topic is created.
Deployment patterns that keep OT and IT separated
Plant data has to reach analytics and maintenance systems without giving IT networks a direct path into control networks. The patterns below separate the two zones at different points, and each one trades protection against operational effort. Pick them together per site, based on the zone model and the segmentation rules agreed under ISA/IEC 62443.
Site-local broker with store-and-forward
Run a broker inside each plant, close to the MQTT sources, so machines keep publishing when the link to head office drops. Messages queue locally and drain when connectivity returns. Check how much local disk the queue can use, how long a plant can run offline, and whether the broker keeps message order and ownership of each topic during replay. The operating cost is a second place to patch, back up and monitor, usually on plant-adjacent hardware.
DMZ relay between OT and IT
Place a relay broker in a demilitarized zone between the plant network and the corporate network. Plant-side components push only to the relay, and IT consumers read from it, so no IT system opens a session into the control zone. Confirm which direction each flow may take, which accounts can publish or subscribe, and how TLS and authentication are enforced on each side. Expect firewall rule reviews and a team that owns the DMZ host as an additional system to patch.
One-way data diode or unidirectional gateway
Where policy forbids any return path from IT into the plant, a data diode or unidirectional gateway allows telemetry to leave but nothing to enter. Commands and configuration changes then need a separate, controlled route. Confirm with the asset owner and the plant safety team which flows qualify, how lost packets are handled, and how the receiving side detects gaps. Operations cost includes the hardware, change control for the one-way path, and reduced remote troubleshooting.
Per-site clusters versus central cluster
A cluster per plant keeps each site working during a wide-area outage and limits blast radius, but multiplies upgrades, certificates and monitoring. A central cluster fed by the DMZ relay is simpler to run and gives one view of the estate, but depends on WAN links and concentrates risk. Ask how many sites run what, what happens to plant dashboards when the central cluster is down, and who approves a version change that must roll across every site.
Kafka Connect or NATS bridges to historians
Connectors move data from the broker into historians, maintenance platforms and data lakes, and carry the schema and offset handling those systems need. For Kafka, Kafka Connect runs the connectors. For NATS, bridges and JetStream consumers do the same job. Confirm which historian and CMMS connectors are maintained, who writes the custom ones, and how a failed sink is retried without duplicating work orders. Each connector is another component with its own credentials and upgrade cycle.
Operating the platform on plant-adjacent hardware
Brokers at the plant need power, cooling and physical access controls that match the site, and they often run on small servers outside a data center. Maintenance windows are set by production schedules, not IT calendars. Ask how a failed node is replaced on site, whether the plant has staff who can do that, how time is synchronized, and how logs leave the plant for security monitoring. Plan for spares and tested restore steps as recurring work.
How a plant rollout moves from one line to the enterprise backbone
The sequence starts with zones and conduits derived from a risk assessment, proves the design on one production line, and ends with a handover the plant team can run without the integrator on site. Each phase closes with a named output so the plant, security and IT owners can sign off before the next phase begins.
Scope zones and conduits from the risk assessment
Before any broker is installed, the team reviews the plant's asset inventory and network diagrams and runs a security risk assessment in the style of ANSI/ISA-62443-3-2-2020, which covers risk assessment for system design. The output is a zone and conduit model that lists each zone, the assets it contains, the data flows that cross each conduit, and the target security level for each. Plant engineering, OT security and IT architecture sign the model. The step is complete when every flow from a machine to an enterprise consumer crosses a documented conduit and no flow is undocumented.
Pilot one production line with a local broker
The team deploys a single broker and MQTT ingestion on one production line, using a local broker at the edge so the line keeps producing data even if the enterprise network is unavailable. Tags from PLCs or gateways are mapped to topics with an agreed naming scheme and payload format. The output is a working pilot with documented topics, message rates measured against the line's actual signals, and a sign-off from the plant operator that the data matches what the line shows on its HMI. Only this line is connected upstream during the pilot.
Test buffering and reconnect with a dropped link
The team deliberately breaks the link between the edge broker and the upstream cluster to prove that telemetry is buffered and replayed without gaps or duplicate handling surprises. The test covers a short outage, a longer outage, and a reconnect under load, and checks that consumers resume from the right offset or sequence. The output is a written test report with observed loss, ordering and replay behavior against the requirements agreed in the first step. The step closes only when the plant and analytics owners accept the results or document the gaps that remain.
Harden the DMZ and access control lists
The team places the upstream relay in a demilitarized zone between the OT and IT networks, enables TLS on every listener, and replaces anonymous or shared credentials with per-client identities. Access control lists limit each producer to its own topics and each consumer to the topics it needs. The output is a configuration baseline, a firewall rule set mapped to the conduits from the first step, and evidence that unauthorized publish and subscribe attempts are refused. Security tests confirm the configuration before any further line is connected.
Integrate analytics and maintenance consumers
Analytics platforms and the computerized maintenance management system subscribe to the approved topics through their own consumer identities. The team validates message schemas, timestamps and asset identifiers against the maintenance system's records so alerts map to the correct equipment. The output is a set of tested integrations with a documented schema for each topic, and a confirmation from maintenance planners that the alerts they receive can be acted on. Consumers that were not in scope for the pilot are added only after this check passes.
Extend the rollout line by line
Once the pilot and hardening steps have closed, the same pattern is repeated for each additional line or cell, reusing the zone model, topic conventions and configuration baseline. Each new line gets its own buffering test and access review before it joins the enterprise backbone. The output for each line is a dated change record and an updated asset list. Lines that cannot pass the buffering or access checks stay on local operation until the gap is closed.
Hand over runbooks and operating procedures
The team writes runbooks for broker restarts, certificate renewal, access changes, topic creation, backup and restore, and the buffering test, and trains the plant and IT operators who will run the platform. Monitoring alerts are tied to named owners. The output is a handover package with runbooks, the test reports, the configuration baseline and a list of open items with owners. Handover is complete when the operators run one scheduled test themselves while the integrator observes.
Evidence auditors and regulators ask for from a plant streaming platform
This table maps each evidence item an assessor or regulator is likely to request to the platform component that produces it. Read it as a checklist to build before an assessment, not as a list of findings. DeployOpen prepares this evidence, but auditors, certification bodies and regulators reach their own conclusions.
| Evidence item | Produced by | What to confirm or retain |
|---|---|---|
| Zone and conduit diagram | Network design and broker placement records | Diagram showing which IACS zones the brokers, MQTT edge gateways and IT consumers sit in, and every conduit between them, kept under change control |
| Risk assessment record | Security risk assessment for the system design (ANSI/ISA-62443-3-2-2020) | Per-zone and per-conduit risk results, the target security levels chosen, and the owner's sign-off dated before go-live |
| Kafka ACL and authentication configuration | Kafka StandardAuthorizer in KRaft mode and SASL or SSL listener settings | Exported ACL metadata, confirmation that allow.everyone.if.no.acl.found is not set to true, and the SASL mechanisms enabled per listener |
| NATS account configuration | NATS accounts and user permissions for edge and plant clients | Account definitions showing which plant sites and applications can publish or subscribe to which subjects, reviewed on a set schedule |
| TLS settings | Broker, MQTT bridge and client TLS configuration | Certificate inventory with issuers and expiry dates, protocol and cipher settings, and proof that unsecured listeners are disabled or documented as exceptions |
| Broker and Java version records | Version inventory and change log for each broker, Kafka Connect and client runtime | Installed Kafka or NATS versions, Java 17 for brokers and Connect, and the release notes reviewed for each change |
| Incident logs supporting NIS2 timelines | Broker audit logs, authorization denials and monitoring alerts forwarded to the SOC | Timestamps that show when an incident was detected, when the early warning was sent within 24 hours, and when notification and final report followed |
| Patch and upgrade records | Change tickets, test results and upgrade runbooks for brokers and edge gateways | Dated records of each patch, rolling upgrade path followed and vulnerability advisories tracked against the installed versions |
How much work is it to stream plant data, and what does it cost
What drives the cost of streaming plant data?
The cost follows the scope you can write down: the number of sites, the number and type of sensors or machines, the message rate, how long data is retained, and how many systems consume it. Each added site adds edge gateways, network paths and sign-offs, and each added consumer adds integration and testing. Retention multiplies storage, and replication across sites or a disaster recovery copy multiplies it again. Ask for a quote that lists each of these as a separate line item so you can see which assumptions move the price.
What is one-time work and what recurs after go-live?
One-time work usually covers architecture and segmentation design, edge and broker installation, MQTT bridge configuration, integration with analytics and maintenance systems, testing, hardening and documentation. Recurring cost covers infrastructure, licenses for any commercial tooling you add, patching and upgrades, monitoring, backups, on-call coverage and any retainer for ongoing support. Self-hosted Apache Kafka is Apache-2.0 licensed with no paid edition from the project, so the recurring line is mainly your own infrastructure and people, not a software subscription. Confirm the license terms of any connector, gateway or dashboard tool you add.
How long does it take to get a first pilot line streaming?
The time to a first pilot depends on a few drivers rather than a fixed duration: how many OT networks must be crossed, whether firewall and gateway changes are needed, how quickly OT access approvals are granted, whether a single line or a whole site is in scope, and how many sensors speak a protocol your edge stack already supports. A pilot on one line with an existing MQTT or OPC gateway moves faster than one that needs new hardware, new segmentation zones and new change control. Ask the partner to name the approval steps and their owners before agreeing a schedule.
What must the plant team provide before streaming can start?
The plant team should provide an inventory of machines, PLCs and sensors with their protocols and data rates, network diagrams showing existing zones and conduits, and the names of the OT engineers who can approve changes. They also need to state which systems will consume the data, the retention they require, and who owns each segment of the network. Without these inputs, scope cannot be fixed and work is at risk of being repeated. Engineering time from the plant side is often the largest dependency, so agree who is released for workshops and testing before the engagement starts.
Can our team run a plant streaming platform itself?
Yes, if you have people who can operate Linux hosts, Java-based brokers or a lightweight server such as NATS, manage certificates and ACLs, and keep up with patches on a schedule that OT change control allows. Many teams split the work: an internal team owns operations and on-call, while an implementation partner handles the initial build, hardening, documentation and handover. Apache Kafka requires Java 17 for brokers and Kafka Connect since version 4.0, so the platform team must be ready for that upgrade path. Ask a partner to define what handover includes and what a retainer would cover afterward.
How do the Machinery Regulation and CRA affect the scope?
Regulation (EU) 2023/1230 applies from 14 January 2027 and includes an essential requirement that machinery be protected so connecting another device or remote device does not lead to a hazardous situation. If your plant builds or modifies machinery sold into the EU, that requirement can shape how telemetry links are designed and tested. The Cyber Resilience Act applies to products with digital elements from 11 December 2027, but manufacturers' Article 14 reporting obligations have applied since 11 September 2026. Confirm with your legal and compliance team which of your products and internal systems fall under each regulation, because that decides which controls and evidence belong in the scope.
Which OT access approvals take the longest to secure?
Approvals for connections that cross from OT to IT networks usually take the longest because they touch safety, production uptime and security at once. Expect reviews from plant operations, OT security, network engineering and the asset owner, and possibly a risk assessment aligned with ISA/IEC 62443-3-2. Each new conduit between zones needs a documented purpose, allowed protocols and a rollback plan. Ask the partner which approvals they have run before, what evidence they prepare for each review, and whether they can work inside your existing change calendar.
What to ask an implementation partner before signing
Start by asking for specifics on OT experience. Ask the partner to walk you through a past or illustrative zone and conduit model under ISA/IEC 62443, including how they assigned security levels with the ANSI/ISA-62443-3-2-2020 risk assessment and which system requirements from ISA-62443-3-3-2013 they mapped to each zone. Ask who on their team holds plant-floor experience, and whether they have worked with the PLC and historian vendors on your site. A partner who answers with general statements about segmentation, rather than naming zones, conduits, firewall rules and the owner of each boundary, has not done this work in a plant.
Next, ask how they handle network changes in production. Request their change procedure for a broker, a bridge or an MQTT gateway: how they schedule maintenance windows with plant operations, how they test a configuration in a staging zone that mirrors the line, and how they roll back a failed change. Ask for the buffering and recovery test plan in writing: what happens to telemetry when the WAN link to the IT side drops, how long the edge buffer holds data at your sample rate, and how the consumer resumes without duplicate or missing records. For Kafka, ask for the upgrade path from your current release to the latest supported one, including whether they will migrate any ZooKeeper-based clusters to KRaft first, as the Kafka upgrade notes require. For NATS, ask how they will plan JetStream replicas, and whether they will test a rolling upgrade on a copy of production before touching the plant cluster.
Then settle who holds on-call for plant sites and in what hours. Ask for named roles, escalation times and what the partner will and will not touch without your operations staff present. Ask for the evidence delivered per zone: asset inventories, zone and conduit diagrams, configuration baselines, ACL and authorization exports, test results and the record of who approved each change. Finally, ask how they track vendor advisories for edge gateways, PLC-adjacent hardware and the broker software itself. Expect a named monitoring source, a patch decision timeline agreed with you, and a method for applying updates to a segment with no direct internet path. Confirm each answer in the statement of work, because vague answers on any of these points are a warning sign.
