Why plant data pipelines need OT-aware design before any platform choice
A plant or multi-site manufacturer should design the data path around the operational network before it chooses any pipeline platform. Sensor, PLC and MES data are useful in analytics only if collection never gives the IT side a route into control networks, and the design has to assume that plant links will sometimes drop. Generic ELT assumes stable connectivity, a flat network and data sources that can be queried freely. Plant environments break all three assumptions. Control systems were built for availability and deterministic behavior, so a poll, a scan or a retry storm that is harmless in a warehouse can disturb a controller or a historian. Safety and reliability constraints also mean that data collection must never influence the process it observes. The framing that most security and operations teams will expect is the ISA/IEC 62443 series, which sets requirements for IACS asset owners, system design, system security levels and component suppliers, and NIST SP 800-82 Rev. 3, which covers PLCs, SCADA and DCS together with the performance, reliability and safety requirements of OT. Before any platform is selected, confirm with the plant security lead which zones and conduits exist, which assets sit in each zone, and which data flows cross the boundary and in which direction. Those answers decide where collection components may run and what they are allowed to reach. The short answer for most plants is a layered, self-hosted design: Apache NiFi with MiNiFi at the edge to collect and buffer data from the plant floor, Apache Kafka to carry events between zones and to multiple consumers, and Apache Airflow or Dagster for batch loads and orchestration into the analytics store. Each layer runs in infrastructure the organization controls. The rest of this page explains how to place these components, how to handle intermittent links, and how to keep OT and IT separated.
Which regulations and constraints shape a manufacturing pipeline
This table maps the standards and EU rules that touch plant data collection to what each one means for the pipeline design. Read each row as a constraint to settle before choosing collectors, brokers or network paths, and confirm applicability with your legal and OT security teams. The main takeaway is that segmentation, read-only collection and change control on edge flows are shared needs across most of these obligations.
| Requirement | What it asks for | What it means for the build |
|---|---|---|
| ISA/IEC 62443 zone and conduit model (ANSI/ISA-62443-2-1-2024, ANSI/ISA-62443-3-2-2020, ISA-62443-3-3-2013) | Asset owners define security zones, conduits between them, a security program, risk assessment for system design and target security levels | Place collectors in a defined zone, route data through documented conduits such as a demilitarized zone or one-way path, and record the risk assessment for each flow |
| NIST SP 800-82 Rev. 3 (PLCs, SCADA, DCS) | Guidance for OT security that also weighs performance, reliability and safety over confidentiality alone | Collect with passive or read-only methods, avoid polling loads that slow controllers, and test every flow against plant performance limits |
| NIS2 Annex II manufacturing scope and Article 23(4) reporting | In-scope entities report significant incidents with an early warning within 24 hours, a notification within 72 hours and a final report within one month | Log pipeline failures and unauthorized access to edge and broker components so incident timelines can be reconstructed, and keep a runbook mapped to these three deadlines |
| Machinery Regulation (EU) 2023/1230, Annex III 1.1.9, applying from 14 January 2027 | Machinery must be protected so that connecting another or remote device does not lead to a hazardous situation | Treat new gateways, remote access and edge devices added for data collection as connections to machinery, and review them with the machine safety team before go-live |
| Cyber Resilience Act Article 14 for manufacturers, applying from 11 September 2026 | Manufacturers of products with digital elements report actively exploited vulnerabilities and severe incidents to the designated authorities | Maintain a vulnerability and patch inventory for edge collectors and gateways, and define who triages a vulnerability report and within what internal process |
What data moves from the plant floor and where it lands
The first decision is to list every source and assign it a tier before choosing any tool. PLC tags are usually read through OPC UA where the controller supports it, or through vendor protocols where it does not, and the data arrives as high-rate signals such as temperatures, pressures, cycle counts and drive currents. Historians and SCADA systems hold a second copy of much of this data, often exported in batches, so the pipeline owner must decide whether the historian or the controller is the system of record for each tag. Machine sensors added for condition monitoring usually sit beside the PLCs and need the same treatment. MES work orders and quality records are business-relevant but arrive on a slower cadence, typically as scheduled extracts from the MES database or its API. Maintenance records from the CMMS usually follow the same batch pattern. Confirm with the plant engineering team which sources exist on each line, who owns them, and whether any source can be read without touching the control network.
Volume and latency should split the sources into two paths. High-rate telemetry, sampled at sub-second or second intervals across many machines, belongs on a streaming path such as Apache Kafka, where it can be buffered, replayed and fanned out to several consumers. MES extracts, quality results and work order changes can run in batch through an orchestrator such as Apache Airflow or Dagster, with each run recording its own completeness checks. Operational data (setpoints, alarms, machine state, tags used by control logic) must be kept distinct from business data (orders, materials, quality outcomes, labor), because the two groups have different owners, retention rules and access needs. Mixing them in one table makes it hard to show which copy controls production and which feeds reporting. Ask each plant for its expected message rate per line and its acceptable delay for dashboards and for quality holds, since these numbers determine partitions, retention and the cost of the storage layer.
Tag names and event schemas change during line changeovers, when a product recipe is swapped, a sensor is replaced or a PLC program is revised. A pipeline that expects a fixed set of fields will silently drop data or fail the whole batch at that point. Plan for this from the start: keep a versioned schema registry for each message type, decide which changes are additive (a new tag is added) and which are breaking (a tag is renamed or retyped), and route breaking changes to a review step instead of letting them reach the warehouse. Maintain a mapping table from raw tag names to engineering names, owned by the plant, so a changeover updates one mapping instead of every downstream query. Add a check that flags tags that stop reporting, since a silent sensor is often the first sign of a changeover that nobody told the data team about.
Which open source platforms fit each layer of a plant pipeline
A plant pipeline has distinct layers: edge collection on the plant floor, streaming across the OT boundary, batch orchestration, and a landing store for analytics. Each layer has a different owner, latency profile and security boundary, so the platform choice should follow the layer rather than a single product for everything. The cards below cover the components we would evaluate first and the checks to run against each vendor's own documentation.
Apache NiFi for flow management
NiFi is Apache-2.0 licensed with no paid edition from the project itself, and version 2.12.0 was released on September 13, 2026. It requires Java 21 as a minimum runtime, and a 1.x installation must be upgraded to 1.27.0 before moving to 2.0.0, with the flow file changing from flow.xml.gz to flow.json.gz. For plant data, NiFi gives visual provenance on every hop, which helps when a sensor feed drops records. Confirm that your identity provider fits its LDAP, OpenID Connect, SAML 2.0 or X.509 options, since Kerberos is deprecated.
MiNiFi for edge collection
MiNiFi runs close to PLCs, historians and gateways, and the project publishes two lines relevant here: MiNiFi Java 2.12.0 and MiNiFi C++ 1.0.0, released August 19, 2026. The C++ agent suits constrained edge hardware, while the Java agent fits hosts with a JVM already in place. Store buffered data on local disk so that a plant link outage does not drop readings. Confirm which MiNiFi flavor supports the processors and protocols your PLCs and historians need, because the set differs between builds.
Apache Kafka for the streaming backbone
Kafka is Apache-2.0 licensed, and the supported releases on the downloads page are 4.3.1, 4.2.2 and 4.1.2. Since Kafka 4.0 it runs only in KRaft mode, so any ZooKeeper-based cluster must migrate before upgrading. Place brokers in a DMZ-style zone between OT and IT, and enable SSL/TLS plus SASL (SCRAM-SHA-512 or OAUTHBEARER are common choices to evaluate). The StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set, so confirm that default in your design review.
Airflow or Dagster for batch orchestration
Batch work such as daily quality reports, historian backfills and MES reconciliation runs fits a scheduled orchestrator better than a streaming tool. Apache Airflow 3.3.2 and Dagster 1.13.26 are both open source and suit this layer, and the choice usually depends on whether your team prefers task-based DAGs or asset-based definitions. Keep orchestrator workers inside the IT zone and pull data through controlled connections rather than letting the scheduler reach into OT. Confirm the retry, backfill and alerting behavior in each project's current documentation before committing.
Node-RED for lightweight edge logic
Node-RED is Apache-2.0 licensed, and version 5.0.8 was published on October 8, 2026. It is useful for simple protocol translation, threshold alerts or a single machine cell where running NiFi would add more operational weight than the task needs. Keep its flows under version control and restrict the editor to named engineers, because a browser-based editor on a plant network is an exposed surface. Confirm which industrial protocol nodes you require and whether they are maintained by the core project or by third parties.
ClickHouse as the landing store
ClickHouse suits high-volume, time-stamped plant data that analysts query by line, asset and shift. Land raw readings in an append-oriented table with retention rules agreed with operations, and build curated tables for OEE and quality reporting from there. Keep the landing store on the IT side of the boundary, with dashboards and downstream consumers reading from it rather than reaching back into OT. Confirm the ingestion path from Kafka, the replication and backup approach, and the retention periods with your plant and quality teams, since none of these are set by default.
Deployment patterns for connected and intermittently connected plants
A single plant with a dependable WAN link can usually run one NiFi cluster in a manufacturing IT zone or the plant DMZ, with the historian, MES and quality systems feeding it through defined interfaces. Kafka sits between the flows and the analytics platform, and buffering mainly covers consumer outages and short upstream interruptions.
- Integration points: Historian exports, MES production orders and quality results are read through database or API processors in NiFi, while ERP master data flows in the opposite direction from the central side.
- NiFi placement: Run a NiFi 2.12.0 cluster in the DMZ, and deploy MiNiFi Java 2.12.0 or MiNiFi C++ 1.0.0 agents only on lines where local collection is needed.
- Kafka retention: Set topic retention from the recovery time your consumers need after an outage, and confirm the retention and replication settings with your Kafka team by testing a consumer failure.
- Credential separation: Give OT-facing flows their own service accounts with read-only historian access, and keep ERP and analytics credentials in separate process groups with their own authentication provider. NiFi supports LDAP, OpenID Connect, SAML 2.0 and X.509 client certificates, while its Kerberos provider is deprecated.
Confirm that the NiFi UI cannot be reached from the plant floor network, because without a security configuration NiFi binds to 127.0.0.1 and any later change to that binding needs a firewall review.
Sites with unreliable or metered links need the edge to keep working when the connection drops. The usual approach is store and forward at each plant, with MiNiFi agents holding data in local repositories during an outage and sending it when the link returns. Replay into Kafka then happens through a regional or central NiFi cluster, so ordering and deduplication are designed before the first outage rather than handled afterward.
- Store and forward at the edge: Agents hold records in local queues during outages, so confirm local disk sizing and the behavior when that disk fills, including whether the agent drops data or stops reading from the source.
- Replay into Kafka: On reconnect, agents resend in the order they were produced. Use event timestamps and stable message keys so downstream consumers can deduplicate, and confirm the producer idempotence settings with the platform team.
- Kafka placement and version: Kafka 4.x supports only KRaft mode, so any ZooKeeper-based cluster must migrate before upgrading. Current supported lines include 4.1.2, 4.2.2 and 4.3.1, and geo-replication can be planned for cross-site copies.
- Integration points: MES and historian events are grouped by site, quality results go to a central quality system, and ERP orders and master data are pushed down during connectivity windows.
- Credentials: Issue each agent its own certificate or service identity so access can be revoked centrally, and enable SASL or SSL on Kafka listeners with per-topic ACLs, since resources without ACLs are restricted to super users by default.
Confirm backlog alerts for each site and check what a large replay does to downstream aggregates and dashboards before the first production rollout.
Some cells, such as safety-critical lines or sites with no route to the corporate network, can only exchange data through a controlled boundary. The pattern is local collection inside the cell, a documented transfer point with a small set of approved flows, and no direct network path from the cell to IT systems. Requirements for zones and conduits come from the ISA/IEC 62443 series, and the risk assessment in ISA-62443-3-2 should define them before any flow is designed.
- Cell-level collection: Sensors, PLCs and historians publish to a local broker or a standalone NiFi instance with no outbound route, so a failure inside the cell does not depend on the enterprise network.
- Controlled transfer: Place a NiFi instance on each side of the boundary and move only the agreed datasets. The transfer device and its configuration should be reviewed by OT security and the network team, and the failure of the transfer path should be tested.
- Offline updates: Connectors, images and NiFi and Kafka packages reach the cell through a reviewed media process. Confirm which release lines are supported at the time of each update, since the Kafka downloads page lists 4.1.2, 4.2.2 and 4.3.1 as supported.
- Credentials: Cell accounts are separate from IT identities, are never reused across the boundary, and are managed in a local identity store. Shared administrator accounts are avoided, and each transfer flow runs under its own identity.
- Standards mapping: Use ISA-62443-3-3 to set the security level targets for each zone and map each conduit to the flows that cross it.
Confirm the zone and conduit documentation with your OT security team before deploying any flow, and record how the cell will be audited against the agreed requirements.
How a manufacturing pipeline rollout moves from one line to the plant network
The rollout starts with an inventory of assets and network zones, proves collection on a single line, then hardens the streaming and orchestration layers before extending to the plant network. Each phase closes with a documented output that the next phase depends on.
Inventory assets and map them to zones
The team, usually plant engineering and OT security working with DeployOpen, lists PLCs, historians, MES servers, edge gateways and the network paths between them, then groups them into zones and conduits that follow the ISA/IEC 62443 model. Each zone gets an owner and a stated purpose, and the inventory records which systems are allowed to cross each boundary. The step is done when the zone map is approved by plant and security leads and every planned data flow has a named source, destination and direction.
Define use cases and data contracts per line
For each line, the team writes down the business question the data answers, such as downtime root cause or scrap tracking, and the exact tags, units, sampling rates and timestamps needed to answer it. Each agreed set becomes a data contract that names the producer, the schema version, the acceptable latency and who approves schema changes. Plant and analytics owners sign the contracts. The output is a contract register that limits what leaves the OT zone and gives later pipeline tests something to check against.
Pilot edge collection on one line with MiNiFi
On a single line, DeployOpen deploys MiNiFi agents on the edge gateway to collect the agreed tags, buffer them when the link to the plant network drops, and forward them through a one-way or tightly restricted path toward NiFi. The pilot checks that buffered data arrives in order after an outage and that no inbound control traffic is opened. The step closes when the team can show, from the agent logs and the receiving side, that every tag in the contract arrived with the expected timestamps.
Harden NiFi and Kafka before wider use
Before more lines are connected, DeployOpen configures NiFi and Kafka for production use. NiFi starts bound to the loopback interface until its security configuration is in place, then is set up with an approved authentication provider, such as LDAP, OpenID Connect or SAML 2.0, and TLS. Kafka gets TLS on every listener, SASL authentication for clients and brokers, and ACLs that grant each producer and consumer access only to its own topics, with no open default for resources that lack ACLs. Credentials are stored in the approved secrets tool. The output is a hardening record with the configuration reviewed against the zone map.
Integrate batch and MES data with Airflow or Dagster
Batch sources such as MES work orders, quality records and historian exports are scheduled through Airflow or Dagster, which run the transformations, enforce dependencies and retry or alert on failure. Pipelines read from the MES through the interface the plant already exposes, and results land in the analytics store that the contracts name. The team tests each DAG or asset against sample plant data, runs a backfill for one shift, and confirms that failed runs alert the owners. The step ends when a scheduled run completes end to end and the failure path has been exercised.
Cut over with runbooks and an upgrade plan
The team moves the pilot line to production use, then adds lines in agreed groups. Each addition follows the same contract, collection and hardening checks. DeployOpen writes runbooks for agent restart, buffer recovery, Kafka topic and ACL changes, NiFi flow deployment, and rollback. The upgrade plan records which NiFi and Kafka versions the plant runs, the supported upgrade paths described in each project's documentation, and the test environment used before any change. The output is a signed handover with runbooks, the upgrade plan and a list of open risks, which the plant and IT owners accept in writing.
Evidence auditors and regulators ask for from plant data flows
This table maps each evidence item that assessors commonly request for plant-to-analytics data flows to the component that produces it and what a reviewer will look for. Use it to plan evidence collection before an assessment starts, not after. DeployOpen prepares this evidence, but auditors, certification bodies and regulators assess it.
| Evidence item | Produced by | What the reviewer checks |
|---|---|---|
| Network zone diagrams and conduit inventory | Architecture design documents, reconciled against NiFi and Kafka configuration exports | Every OT-to-IT crossing is named, the zone boundary is documented, and no undocumented connection exists |
| Access records and ACL exports | Kafka StandardAuthorizer ACLs stored in the cluster metadata log, plus NiFi user and policy settings | Each principal has a documented purpose, no resource falls back to open access, and reviews are dated |
| NiFi data provenance and flow change history | NiFi provenance repository and flow version control through the registry | Data lineage from edge collector to destination can be traced, and every flow version maps to an approved change |
| Incident timelines for NIS2 reporting windows | Monitoring and logging stack, combined with the incident response record kept by the organization | Detection, early warning, notification and final report timestamps fit the 24-hour, 72-hour and one-month windows |
| Vulnerability and patch records for edge components | Version inventory of MiNiFi, NiFi, Kafka and host operating systems, with the organization's patch tracking | Installed versions are supported, known vulnerabilities are tracked to closure, and end-of-support dates are planned |
| Change approval records for flow and DAG changes | Ticketing system, Git history for DAG and flow definitions, and deployment logs | Each production change links to an approver, a tested version and a rollback path |
What manufacturers ask before building plant data pipelines
What drives the cost and effort of a plant data pipeline?
The main drivers are the number of lines and machines, the protocols they speak, the number of sites, how many network zones data must cross, and who operates the platform afterward. Each additional PLC family, MES instance or plant usually adds connector work, testing and validation. One-time costs cover design, edge collection setup, connector build, integration testing and documentation. Recurring costs cover compute and storage, patching, upgrades, monitoring and on-call coverage. Ask any partner for a quote that lists each line, protocol and site in the written scope, and for the operating model after go-live.
How long does it take to get data from one production line?
Time to first data from one line depends on the agreed scope, the state of the asset list, how quickly network access is approved and how many plant downtime windows are available for connecting equipment. A single line with well-documented signals and an existing MES interface is a much smaller job than a line with undocumented PLC tags. Treat any duration as a figure to be agreed in writing before work starts, not as a default. Ask for a plan that lists the milestones, the approvals each one depends on and the downtime the plant must grant.
What do OT and IT teams need to provide before the work starts?
The project needs three things from the client. First, an asset list that identifies PLCs, sensors, historians and MES components, with their models, firmware and the signals each exposes. Second, approved network access between the plant zone and the collection point, with firewall rules reviewed by both OT and IT owners. Third, planned plant downtime windows for installing edge agents and testing connectivity on live equipment. Security owners should also confirm where data may flow and who signs off on changes to the OT side.
Can an internal team run a plant data pipeline without an outside partner?
Yes, an internal team can run it, but it needs ongoing capacity for upgrades, monitoring and on-call response, not only the initial build. Apache Kafka supports rolling upgrades from each release in the past year to the latest version, so a team must plan regular upgrade cycles. Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate first. Apache NiFi 2.x requires Java 21, and the 1.x line can no longer be maintained. Confirm that your team has the skills, the on-call rota and the test environment to carry these upgrades.
How does the NiFi 1.x end of life affect existing installations?
NiFi 1.28 is the last minor release of the 1.x series, and the project lists an end-of-support date of December 8, 2024 for 1.28.1. The project states that 1.x cannot be maintained further because it depends on Jetty 9.4, Spring Framework 5.3 and AngularJS 1.8, which cannot be upgraded in that line. Migration to NiFi 2 requires first upgrading to NiFi 1.27.0, then moving to 2.0.0, and changing the flow file from flow.xml.gz to flow.json.gz. The variable registry expression language scope was also replaced by an ENVIRONMENT scope in 2.0. Plan the migration, test every flow, and confirm the path with your NiFi support provider.
Will MES vendors support an integration with an open-source pipeline?
Whether a MES vendor supports an integration depends on that vendor's license terms, its supported interfaces and whether the vendor accepts third-party connections to its database or APIs. Some vendors restrict direct database access or require their own certified connectors, and that can change the design. Confirm in writing with each MES vendor which interfaces are supported, what is permitted for data extraction and whether the integration could affect support status. Include the answer in the scope before any connector is built.
How do pipelines keep plant networks separated from IT networks?
Pipelines keep the two networks separate by placing edge collection inside the plant zone, using a one-way or tightly controlled path to the IT zone, and allowing only the protocols and endpoints listed in an approved rule set. Apache NiFi can run collection at the edge with MiNiFi agents, and Kafka can carry the data with TLS and SASL authentication and ACLs that limit which clients may write or read. Ask for a zone and conduit diagram, a list of permitted flows and a record of who approved each rule. The ISA/IEC 62443 series is a useful reference for these zone and conduit decisions.
Questions to put to a pipeline partner before signing
Start with OT segmentation, because every other answer depends on it. Ask the partner to show, in writing, which zones they would place the PLC network, the plant historian, the MiNiFi agents and the Kafka brokers in, and which flows cross each boundary. Ask who owns the conduits between zones, whether data moves outbound only from the OT side, and how they would prove that no IT-side component can write back to a controller. A credible answer names the firewall rules, the direction of each flow and the account that operates each component, and it treats the MES and ERP links as separate boundaries rather than one flat path. Next, ask how edge flows are tested against plant outages. Request the test plan for a store-and-forward scenario: what happens when the WAN link drops for a shift, when the central Kafka cluster is unreachable, and when a MiNiFi agent restarts with unsent data. Ask how they confirm that buffered records arrive in order and without duplicates, and how backfills are checked against the historian before anyone trusts the numbers. Also ask how they handle a NiFi flow that was edited on site during an outage, because that is where configuration drift usually starts.
Ask how Kafka and NiFi upgrades are planned against supported release lines. The partner should state which NiFi line they would deploy and why, given that the NiFi downloads page describes the 1.x series as end of support and recommends NiFi 2, and whether the migration guidance (which requires an intermediate step before 2.0.0 and a change from flow.xml.gz to flow.json.gz) is in their plan. For Kafka, ask which release lines they support, how they would reach KRaft if any cluster still runs ZooKeeper, and how rolling upgrades are sequenced across plants. Ask who signs off each change and how plant maintenance windows are respected. Finally, ask how they map flows to ISA/IEC 62443 and NIST SP 800-82 Rev. 3 without claiming certification. A good partner will describe a zone and conduit model, a security risk assessment that informs the target security levels, and a control matrix that links each flow to the requirements it supports. Ask how change control covers PLC-adjacent data sources, such as who approves a new tag, a new OPC endpoint or a changed sampling rate, and whether those approvals are tied to the plant's own management of change. Confirm that they will state plainly that DeployOpen prepares evidence and does not issue certifications, and that the assessment of conformity sits with the asset owner and its independent assessors.
Build, partner or managed service for plant data
Building in-house means your own engineers select, deploy and operate the collection and streaming stack, typically MiNiFi at the plant edge, a NiFi or Kafka core, and the monitoring around them. You pay for staff time, infrastructure in your own accounts or data centers, and the learning curve for every component. The one-time cost is mostly engineering effort to design the flows, wire the OT and IT boundary and write runbooks. The recurring cost is staff time for upgrades, certificate and credential rotation, and troubleshooting at each site.
- Who pays: your payroll covers design, build, testing and the on-call rota, and your hardware and hosting bills cover every site.
- What drives cost up: a growing site count, many protocol types such as OPC UA, Modbus and vendor-specific PLC interfaces, and a 24-hour on-call requirement for plant operations.
- Upgrade load: NiFi 1.x can no longer be maintained, and the 2.x line changed its flow file format and expression language scope, so migrations need planned effort.
- Kafka version planning: supported Kafka releases change over time, so budget for periodic broker upgrades.
- Skills risk: few teams hold both OT protocol knowledge and streaming operations skills.
Before committing, ask your team to estimate the hours for one plant pilot and confirm who owns the OT network change approvals.
In this model an implementation partner plans the architecture, builds the edge collection and streaming flows, hardens them against your OT security requirements and hands over documentation and runbooks. Your team then operates the platform. Most of the cost is front-loaded into a fixed-scope implementation, followed by a smaller recurring cost for your own staff. A partner can also be engaged on a retainer for upgrades and testing if your team lacks capacity.
- Who pays: you pay the partner for scoped design and implementation, and your staff carry day-to-day operation and any retained support.
- One-time versus recurring: the design, integration with MES and historians, and initial security testing are one-time. Patching, Kafka and NiFi upgrades, and on-call coverage recur.
- What drives cost up: the number of plants with unique network zones, the count of PLC and historian interfaces, and the depth of OT security validation required by your program.
- Knowledge transfer: ask whether the handover includes recorded training, documented flows and tested recovery procedures.
- Retainer scope: clarify whether upgrades, vulnerability response and after-hours support are included or quoted separately.
When requesting a written quote, ask for the scope per site, the list of protocols covered, the handover deliverables and the support hours offered, and confirm that the partner does not issue audit reports or certifications.
A managed cloud IoT service takes on the collection, device connectivity and much of the operation on the provider's platform. You pay a subscription or usage fee that typically scales with devices, messages or data volume, and you take on less infrastructure work. The trade-off is that plant data often leaves your network boundary, and your options for protocol support, retention and sovereignty depend on the provider's terms. Pricing models vary by provider, so the quote must be read against your actual message volumes and device counts.
- Who pays: the provider charges recurring fees, and you still pay for integration work, device onboarding and any gateway hardware at plants.
- What drives cost up: device count, message frequency, data retention periods, and any additional features for analytics or regional hosting.
- Connectivity limits: confirm how the service buffers data when a plant link drops and how long it holds data offline.
- OT boundary: check whether an outbound connection from plant networks to the cloud is acceptable under your zone model.
- Exit cost: ask how data and device definitions can be exported if you change provider.
Request a written quote that states the pricing unit, the included retention, the regions where data is stored and the cost of exporting your data at the end of the contract.
How the scope of a manufacturing pipeline is decided
Should we connect every plant at once or start with one site?
Start with one site that is representative of the others, then extend the pattern. A single pilot site lets you test edge collection, buffering during outages and the OT and IT boundary under real conditions before those decisions are copied across the estate. Choose the pilot by asking which site has the most typical equipment mix, a stable network path and a plant team willing to validate data against what operators see on the shop floor. Plants differ in PLC vendors, historian products and MES versions, so the first site should not be the most unusual one. Ask your implementation partner to write the per-site onboarding steps so each later plant is a repeat of a documented procedure rather than a new project.
Should the first use case be analytics or control-adjacent data?
Start with analytics that read from data already exported out of the control layer, such as historian copies, MES production records and quality results, and leave anything that feeds back toward control for later. Read-only flows carry far less operational risk because a failure in the analytics pipeline does not change how a machine runs. Control-adjacent data, such as alarm streams or setpoint history used for supervisory decisions, needs a formal review with plant engineering and the OT security owner before it leaves the cell or zone. Ask whether any proposed flow could ever write, acknowledge or change a value in a controller, because the answer determines which security and change controls apply.
How do we decide which data must stay on premises?
Classify each data class by what it reveals and what it could affect, then set the on-premises boundary from that classification. Process recipes, machine parameters, yield and quality data, and maintenance records often carry trade secret and operational value, so they are usually kept in the plant or in the organization's own data center. Data that goes to a central analytics platform should be the minimum needed for the use case. Also list any personal data about operators, since it may fall under data protection rules. Confirm the boundary with legal, the plant manager and the OT security owner, and record the decision so auditors can trace why each flow exists.
What do we need to know about the network zones before scoping?
You need an inventory of the zones and conduits between OT and IT before deciding which systems can talk to each other. The ISA/IEC 62443 series organizes industrial security into zones and conduits, and the risk assessment defined in ANSI/ISA-62443-3-2-2020 is the natural place to record which flows cross each boundary. Ask the plant team for network diagrams, a list of historians, gateways and remote access paths, and any existing firewalls or data diodes. Scope the pipeline so each flow crosses a single, documented conduit, and confirm with the OT owner which direction traffic may travel. Treat any undocumented path as out of scope until it is reviewed.
How does intermittent plant connectivity change the scope?
Intermittent connectivity changes the scope from a simple stream into a store-and-forward design with defined behavior during outages. Decide how long each site must operate without reaching the central platform, how much data to buffer at the edge, and what happens when buffers fill. Those answers determine the edge collector sizing, the replay order and whether the central side must deduplicate. Ask your partner to define the outage behavior per site type, because a site that runs two shifts needs different buffering than a remote site on a satellite link. Also confirm how timestamps are assigned at the edge so late-arriving data lands in the right reporting period.
Can our own team run the pipeline, or do we need a partner?
Your team can run the pipeline if it has people who understand both the OT data sources and the open-source components, and if the scope is limited to a few flows. The work splits into one-time build tasks, such as network design, connector and flow development, security hardening and documentation, and recurring tasks, such as patching, upgrades, on-call and onboarding new sites. Scope decides the split: a single pilot site may be manageable in-house, while a multi-plant rollout with NiFi, MiNiFi and Kafka usually needs a partner to deliver the architecture and runbooks. DeployOpen plans, implements and supports self-hosted open-source platforms in the client's own infrastructure, and most engagements are fixed-scope with the scope agreed in writing before work starts. Ask for a written split of responsibilities in any proposal.
Which upgrade and version constraints should shape the scope?
Version constraints should shape scope early, because they decide which components can be deployed and how long they stay supported. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and Kafka 4.0 supports only KRaft mode, so ZooKeeper-based clusters must migrate first. For NiFi, the project states that 1.x cannot be maintained further and that 2.x is the path forward, with NiFi 2.12.0 released on September 13, 2026. Confirm which versions each site's equipment and integration partners support, and include the upgrade path in the scope rather than treating it as later work.
Which regulations should we check before fixing the scope?
Check the regulations that apply to your products, sites and customers before fixing the scope, since they can require specific evidence and reporting. Manufacturing is a critical sector under NIS2 Annex II, and in-scope entities must report significant incidents with an early warning within 24 hours. The Machinery Regulation, which applies from 14 January 2027, adds an essential requirement about protecting machinery against connection of other or remote devices, which may affect how pipelines connect to machines. Defense suppliers should also check CMMC requirements, noting the Department of War has suspended CMMC Phase II requirements as of 13 July 2026. Confirm applicability with counsel, because the scope of a pipeline often shifts once these obligations are mapped.
How DeployOpen scopes, builds and supports plant pipelines
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open source platforms inside the client's own plants, data centers and cloud accounts. For plant pipelines, that means the collection layer at the site, the Kafka or NiFi tier that carries data out of the OT zone, the warehouse or time-series store that receives it, and the monitoring that tells operations and IT when a link has stopped. Work starts with a written scope that names each plant, each data source (PLCs, historians, MES tables, sensor gateways), each network zone the data crosses, and the owner on the plant side who can approve a change to a site. Most engagements are fixed-scope, so the scope, deliverables and timelines are agreed in writing before any build begins. A quote follows from that written scope, not from a price list, and the reader should expect the quote to itemize the number of sites, source types, zones and integration points. The reader should confirm with any partner, DeployOpen included, which plants and interfaces are in scope for the first phase and which are deferred.
Once the first pipelines are live, the work shifts to running them. Ongoing support, version upgrades and continuous testing are available as retainers, which matter here because Kafka and NiFi both publish version lines with defined support windows, and a plant pipeline that falls out of support becomes a standing exposure. A retainer should cover who applies the upgrade at each site, how a change is tested against a staging copy of the flow before it reaches a plant, and how a rollback is run when a site's connectivity is limited. DeployOpen can also run security testing against the pipelines, including the edge agents, broker listeners and the boundary between OT and IT networks. The reader should ask any partner to name the testing method, the systems in scope and the form in which findings are delivered and retested.
DeployOpen prepares systems, configuration records, network diagrams, access reviews and test results that a client can give to its own auditors, certification bodies or regulators. It does not issue audit reports, attestations or certifications, and no DeployOpen deliverable should be presented as one. Plant operators often need evidence against standards such as the ISA/IEC 62443 series, which an independent assessor, not DeployOpen, must judge. The practical step for a plant leader is to agree in the scope which evidence artifacts will be delivered at the end of each phase, who in the client organization owns them, and which external assessment will consume them, so that the implementation is built to produce that evidence from the start.
