Why self-hosted LLM inference for manufacturing starts in the OT network
A plant can run an LLM maintenance assistant or quality report generator on its own servers, but the network design has to come before the model choice. Maintenance and quality teams want help reading fault histories, work orders and inspection records, and that help is most useful when the model sits near PLCs, historians and MES. The answer is a site-local inference service with no required cloud connection, placed in a segment that the OT team already controls. Design data, recipes and process parameters are the assets most likely to be exposed by a careless deployment, so they should never leave the site as prompts or as retrieved context sent to a hosted API.
The regulatory frame shapes this build more than the choice of model does. IEC 62443 gives the zone and conduit approach that asset owners use to decide what the inference servers may reach, and ANSI/ISA-62443-3-2-2020 covers the risk assessment that should precede any connection to a control zone. NIS2 lists manufacturing as a critical sector, and the Machinery Regulation adds a requirement that connecting another device does not lead to a hazardous situation from 14 January 2027. The Cyber Resilience Act's reporting duties for manufacturers already apply from 11 September 2026, which matters if the platform is part of a product you ship. Ask your OT security lead to classify the inference servers before anyone wires them to a historian.
The short platform answer is vLLM on GPU servers in a server room or data center zone, serving the shared assistant to plant users, and Ollama or llama.cpp on smaller edge nodes at individual plants or lines where a GPU is not practical. Confirm each engine's authentication and binding behavior in your own configuration before deployment, since Ollama listens on localhost by default and llama-server needs an API key and TLS set explicitly. Plan for the edge nodes as a fleet from the start.
Which manufacturing requirements shape the build
This table maps each manufacturing requirement to what it means for placing and operating an inference service. Read it as a design checklist: the right-hand column states the consequence for network placement, outbound traffic, logging and patching. The main takeaway is that the model server belongs inside a defined OT zone with no outbound dependencies, and every update path must be planned with plant change control.
| Requirement | What it asks for | What it means for the inference build |
|---|---|---|
| IEC 62443 zones and conduits, security levels | Group assets into zones by risk, control traffic between zones through conduits, and set a target security level per zone under the ISA/IEC 62443 series. | Place the model server in a zone that is separated from the process control network, with the only conduit being a narrow, documented path for maintenance requests and responses. Confirm the target security level with your OT security team before selecting hardware. |
| ANSI/ISA-62443-2-1-2024, asset owner security program | Requires the asset owner to run a security program covering policies, roles, training, incident handling and ongoing maintenance of the industrial environment. | Assign a named owner for the inference service, add it to the plant asset inventory, and include it in incident response and change procedures. Ask the partner how access reviews and maintenance windows will be documented. |
| ANSI/ISA-62443-3-2-2020, risk assessment for system design | Calls for a risk assessment during system design that identifies zones, conduits and the consequences of compromise before the system is built. | Run the assessment before deployment and record what the assistant can read. Maintenance logs and quality records are different data classes, so the scope of retrieval and the data each user can reach should be decided here. |
| ISA-62443-3-3-2013, system security requirements | Defines system-level requirements and security levels, including access control, use control, data integrity and resource availability. | Require authentication in front of the inference engine, since local Ollama and llama-server APIs can run without it. Confirm how TLS, API keys and audit records are enforced at the zone boundary. |
| NIST SP 800-82 Rev. 3, OT security with performance and reliability | Guides OT security for industrial control systems, PLCs, SCADA and DCS while treating performance, reliability and safety as primary requirements. | Keep inference load off controller and historian networks and set resource limits so the service cannot starve plant systems. Model requests should degrade gracefully when the assistant is unavailable, since maintenance and quality work must continue without it. |
| NIS2 Annex II, manufacturing scope and incident reporting | Lists manufacturing divisions in Annex II as a critical sector. In-scope entities report significant incidents with an early warning within 24 hours, a notification within 72 hours and a final report within one month (Article 23(4)). | Confirm whether your legal entity or group falls in scope. Log enough inference activity (who queried what source, when, from which node) to support a timely early warning, and agree on who starts the reporting clock for an AI service incident. |
| Cyber Resilience Act reporting obligations from 11 September 2026 | Manufacturers must report actively exploited vulnerabilities and severe incidents under Article 14 from 11 September 2026, while most product requirements apply from 11 December 2027 (Article 71(2)). | If you ship products with digital elements or build machines with embedded inference, track engine and component vulnerabilities as reportable events. For an in-house service, set a process to monitor advisories for the inference engine and its dependencies, and confirm with counsel how CRA duties apply to your role. |
| Machinery Regulation 1.1.9, remote and connected devices | From 14 January 2027, Annex III 1.1.9 requires machinery to be protected so that connecting another device or remote device does not lead to a hazardous situation (Regulation (EU) 2023/1230). | Do not connect the inference node to machine control networks or let assistant outputs write to machine parameters. Keep recommendations advisory, and document that the model has no path to actuation. |
Which plant data the assistant touches and why it needs its own zone
The assistant should be given a short, named list of plant data classes before anyone designs a network path, because each class carries a different sensitivity and a different owner. Maintenance logs and work orders usually sit in the CMMS, historian extracts come from the process historian, batch records and quality non-conformances come from the MES or quality system, and CAD files, design notes and supplier drawings live in engineering document stores. Design data and process parameters are the classes most likely to be protected as intellectual property, and batch records may fall under 21 CFR Part 11 if the plant makes regulated pharmaceutical or medical device products. Quality and regulatory teams should confirm whether Part 11 applies to each record type before any of them are indexed. Retrieval is where the risk appears. A maintenance assistant that answers from work orders must query the CMMC or MES, pull documents from a store, and return text to a user on a corporate network, so each retrieval crosses at least one zone boundary. The inference server, the vector index and the document connectors should sit in a demilitarized or dedicated zone with no direct route from the office network or the control network. Data should move through a brokered path, such as a staging database or message queue that copies only approved fields, and where feasible in one direction only from the OT side. Ask the partner to document every flow, its source, destination, protocol, direction and owner, and to test that a control network host cannot initiate a connection into the inference zone.
The ISA/IEC 62443 series is the usual reference for this design, and the zone and conduit model it describes maps directly onto the question of which systems may talk to the assistant. ANSI/ISA-62443-3-2-2020 covers security risk assessment for system design, so the plant's architecture team should use it to assign each data class to a zone and a security level before connectors are built. Plant engineers should also confirm which OT systems are in the scope of the risk assessment, since historian and MES servers often straddle the boundary.
For the inference platform itself, the practical controls are local access only and explicit authentication. Ollama binds to 127.0.0.1 on port 11434 by default and its local API does not require authentication, so any exposure beyond the host needs a reverse proxy or gateway in front of it, and the same applies to llama-server, which supports --api-key and TLS. Set OLLAMA_NO_CLOUD=1 on servers that must never reach cloud features. Record which data classes each connector is allowed to read, and review that list with quality and OT owners before go-live. Confirm the regulatory scope for each plant with its own quality function rather than assuming it.
Which open-source platforms fit a plant deployment
A plant deployment usually needs an inference engine, a way to reach it from the plant network, a chat front end, a document layer for manuals and procedures, and a gateway that controls access. The options below cover one role each, so you can pick the combination that matches your hardware, network zones and user count.
vLLM for shared GPU servers
vLLM is an inference and serving engine under Apache-2.0, and it exposes an OpenAI-compatible API that several plant applications can call at once. It fits sites with GPU servers in a central data room serving maintenance and quality teams across lines. Confirm which accelerators and driver stack your hardware needs, how the server is placed in the OT segmentation model, and who patches the engine, since it ships new releases often. The most recent tagged release we list is v0.31.0, published October 5, 2026.
Ollama for single-site and edge nodes
Ollama is an MIT-licensed runtime for running models on one machine, and it suits a single line, cell or gateway box with modest hardware. By default it binds to 127.0.0.1 on port 11434, so reaching it from other plant systems requires a deliberate bind change and a reverse proxy with authentication in front. Confirm that OLLAMA_NO_CLOUD is set so no cloud features are reachable, and check the concurrency settings against the number of technicians who will query it at once.
llama.cpp llama-server for CPU hardware
llama-server, part of the MIT-licensed llama.cpp project, runs quantized GGUF models on CPUs and modest accelerators, which matters when a plant has no GPU or no approval to install one. It supports API key authentication, TLS certificate files and an optional Prometheus-compatible metrics endpoint, so it can meet basic access and monitoring expectations without extra middleware. Confirm which model sizes give acceptable response times on your floor hardware before committing to it.
Open WebUI or LibreChat for chat
Open WebUI and LibreChat give technicians and engineers a browser chat front end over the inference endpoint. LibreChat is MIT licensed, while Open WebUI is under a BSD-style license with a branding clause that restricts removing or altering its branding for larger deployments unless written permission is granted. Confirm which license terms apply to your user count and whether your plant's identity rules require custom branding before you pick one.
RAGFlow or AnythingLLM for manuals and SOPs
Equipment manuals, work instructions and quality procedures are the source material for most plant assistants, and both RAGFlow (Apache-2.0) and AnythingLLM (MIT) can index them so answers come with citations to the source page. The choice depends on how many document types you have, whether scanned drawings need parsing, and who maintains the knowledge base. Confirm how document revisions are retired so a technician never gets an answer from a superseded procedure.
LiteLLM as the gateway
LiteLLM sits between plant applications and the inference engines, presenting one OpenAI-format interface while handling keys, cost tracking and logging. It is MIT licensed except its enterprise directory, so confirm which features your setup needs fall outside that directory. It also gives a single point to enforce which systems may call which models, which helps keep process data from flowing to any endpoint outside the plant zone. Confirm how its logs are stored, since they may contain prompts drawn from design or process data.
How a plant rollout moves from one use case to a supervised site service
The sequence moves a single plant use case from zone assessment to a supervised internal service, with each phase closing on a documented output that plant, OT and quality owners can review. It keeps design data and process information inside the site from the first pilot onward.
Scope the use case and assess the zones
DeployOpen works with plant engineering, OT owners and the quality team to define the first use case, such as a maintenance assistant or a draft quality report, and the data it may read. The team then inventories assets and maps zones and conduits against the IEC 62443 series, starting from the security level and program requirements the asset owner has committed to. The step closes with a zone and conduit diagram, an asset inventory and a written list of which systems the model may and may not reach.
Pilot on one line without outbound access
The pilot runs on one production line on a server placed in a defined zone, with the model weights, engine packages and container images transferred in through an approved route and verified before installation. The server has no outbound internet path, and the engine is bound to an internal address so inference works without any cloud connection. Plant users test the assistant on historical maintenance records and past quality data. The step closes when the team confirms in writing that no outbound connections occur during normal operation and that the pilot answers from site data alone.
Harden the endpoint, TLS and logging
DeployOpen configures authentication in front of the engine, since the local Ollama API does not require authentication by default, and uses API keys or an authenticating proxy for llama-server or vLLM. TLS is enabled for every client path, metrics are enabled for capacity monitoring, and request logs are written to site storage with a retention period agreed with the plant. The team also defines a patch process for the engine and the operating system, including how releases are tested on a staging server before they reach the line. The step closes with a hardening record that the OT security owner signs off.
Connect to MES or CMMS through brokered reads
Integration with the manufacturing execution system or maintenance system uses a broker or read-only service that pulls approved records, such as work orders, equipment history and batch summaries, and passes only the fields the use case needs to the model. The model never holds credentials for plant systems and cannot write back to control or production systems. Each read is logged with the requesting user and the record identifiers. The step closes when the integration owner confirms that every data path is read-only and that the broker's permissions match the agreed scope.
Validate quality report drafts with human approval
Quality engineers test generated report drafts against records they have already reviewed, measuring accuracy, missing fields and invented values. Each draft remains a draft until a named reviewer approves or edits it, and the approval, the source records and the model version are stored together. Where a report feeds a regulated record, the quality owner confirms which electronic record and signature controls apply to the workflow. The step closes with a validation summary listing the test set, the observed failure cases and the approval steps the process must keep.
Hand over runbooks and operate the service
DeployOpen delivers runbooks for starting and stopping the service, adding or replacing a model, rotating API keys, reviewing logs, handling a failed node and rolling back an engine release. The plant's own team or a retained support provider takes over monitoring of queue depth, loaded models and resource use, and the team agrees the escalation path with OT operations. The step closes when the operators run the runbooks in a supervised session and the handover checklist is signed by the plant and IT owners.
What auditors and plant security teams will ask to see
This table maps the evidence that auditors, plant security teams and NIS2 or CRA reviewers typically request to the component that produces each record. Use it to decide which logs, diagrams and inventories must exist before the first inference request reaches a production line. Confirm the retention and classification rules for each record with your own compliance lead, since the required scope depends on how your plants and products are classified.
| Evidence | Produced by | Why it is asked for |
|---|---|---|
| Zone and conduit diagrams | Network design documents and firewall configuration for the OT segmentation around the inference host | Shows that the model server sits in a defined zone with controlled conduits to the plant historian and MES, as ISA/IEC 62443 zone and conduit thinking expects |
| Security risk assessment records | Asset owner risk assessment covering the inference servers, gateway and maintenance assistant data flows | Documents threats, consequences and target security levels for the system, aligned with ANSI/ISA-62443-3-2-2020 for system design |
| Gateway access logs | AI gateway or reverse proxy logging each authenticated request, caller identity and target model | Shows who asked the assistant what, and that only approved maintenance or quality roles reached the endpoint |
| Inference server access logs | Engine logs plus the API key or TLS-protected access record on the llama-server or vLLM endpoint | Confirms that unauthenticated or out-of-zone clients did not reach the model, and supports incident investigation |
| EU AI Act Article 12 and 19 event logs | Automatic event logging in the application and gateway, retained for at least six months where the system is high-risk | Required only if the system falls within high-risk use under the EU AI Act. Confirm classification before designing retention |
| Model and engine version inventory | Configuration management record listing model file hashes, engine builds such as llama.cpp or vLLM releases, and deployment dates | Lets you show which model and engine answered a given question, and supports change control on plant-facing outputs |
| Patch and vulnerability records | Vulnerability scanning output, vendor advisory tracking and change tickets for each engine and dependency | Supports the CRA Article 14 duty to report actively exploited vulnerabilities once it applies to your products, and general patch evidence for IEC 62443 programs |
| Incident response records | Incident log with detection time, classification, early warning, notification and final report timestamps | Shows the NIS2 Article 23 timeline (24-hour early warning, 72-hour notification, one-month final report) if the plant falls in scope |
What manufacturers ask before commissioning an in-plant model
Can a maintenance or quality assistant run with no internet connection?
Yes, an in-plant model can run with no internet connection once the model weights, the inference engine and its container images are on the plant network. Ollama states that when it runs locally it does not see your prompts or data, and cloud features can be turned off with OLLAMA_NO_CLOUD=1 or the server configuration file. llama.cpp runs GGUF models locally on a wide range of hardware, including CPUs. The work that matters is getting artifacts across the boundary: they must be downloaded outside the plant, checked, and then moved in by an approved route. Confirm with your security team which transfer method your OT zone permits.
What drives the cost of running a plant model?
The cost has four main parts: hardware, OT engineering time, validation and ongoing support. Hardware is either GPU servers or edge nodes, and its size depends on how many users query the assistant at once and which model sizes they need. Engineering time covers network placement, access rules and integration with historians or maintenance systems. Validation covers testing answers against approved documents before staff rely on them. Support covers patching engines that release often and updating models. Hardware and software licenses are one-time or periodic purchases, while support and model updates recur. Ask any partner for a quote that splits these four lines and states what each one assumes about user count and plant zones.
How long does a plant deployment take to commission?
Timing depends mainly on zone work and integration scope, so no duration can be given before those are defined. A single server in an existing IT network with one document source is a much smaller job than a multi-site rollout with edge nodes, segmented OT zones and links to a maintenance system. Plant change control and shutdown windows can also set the pace. Most engagements are fixed-scope, so the timeline is agreed in writing before work starts. Ask for a schedule that lists each zone, each integration and each validation step, and state which of those depend on plant approvals outside the project.
What does the plant team need to supply before the project starts?
The plant team supplies the knowledge that only it holds. Typically this includes:
- A list of the source documents the assistant may use, such as work instructions, maintenance manuals and quality procedures, with owners who can approve them
- A network map of the plant, including which zones and conduits exist and who approves changes to them
- Access to the systems to be connected, such as historians or maintenance tools, with named technical contacts
- Decisions on who may use the assistant, what data it may see and how long logs are kept
- Representatives for validation who can judge whether answers are correct
Without these inputs, scope and cost estimates are weak, so ask a partner to list them in the statement of work.
Can the IT team run an in-plant model without outside help?
An IT team with server and container experience can often operate the inference service after handover, but it usually needs help with OT-specific work. Engines such as vLLM, Ollama and llama.cpp expose standard interfaces, so routine operation is familiar to most platform teams. The harder parts are segmentation decisions, approved change paths into plant zones and validating outputs for quality records. Access controls on the endpoint also need deliberate setup, because the local Ollama API does not require authentication by default. Decide early which tasks stay with your team and which go to a retainer, and check that the plan names the people responsible for each.
How does the design data and process know-how stay protected?
Design data stays protected when the model, its documents and its logs all remain inside the plant's own infrastructure, and access is limited by role. Running models on your own hardware means prompts and files are not sent to a hosted API. Protection also depends on controls around the engine: authentication in front of it (llama-server supports API key authentication and TLS), restricted network paths between the assistant and plant systems, and log storage under your control. Industrial security work such as the ISA/IEC 62443 series helps structure zone and conduit decisions. Confirm with your security team how retrieved documents are classified before they are indexed.
Which standards and regulations should the design cover?
The main reference for plant networks is the ISA/IEC 62443 series, which groups requirements into general, policy and procedure, system and component areas. Its ANSI/ISA-62443-3-2-2020 part covers risk assessment for system design, which is the natural starting point for placing an inference server. If your plants are in the EU, NIS2 lists manufacturing as a critical sector, and in-scope entities must report significant incidents with an early warning within 24 hours. The Machinery Regulation applies from 14 January 2027. Ask your compliance lead which of these apply to each site before design begins.
Will the assistant keep working when the engine or model needs an update?
It will keep working only if updates are planned as a controlled change. Ollama and llama.cpp publish releases frequently, and each update needs regression testing against approved documents before it reaches the plant. Keep the previous model files and container images so a rollback is possible, and store them in a controlled repository inside the plant network. Agree in writing who approves an update, how long a test cycle takes and how an emergency patch is handled. Ask any partner to show its upgrade procedure and its rollback test.
Questions to ask an implementation partner before signing
Start with OT experience, because a partner who has only built IT-side model platforms will treat the plant network as an ordinary subnet. Ask for named examples of inference or analytics services placed in an IACS zone, and ask how they apply the zone and conduit model of ISA/IEC 62443, including which security level target they would set for the inference zone and who signs off on it. Ask whether they have worked with the risk assessment method in ANSI/ISA-62443-3-2-2020 and how the output changes the architecture. An answer that treats 62443 as a checklist to complete at the end should worry you. Also ask who on their team is accountable for the plant-side change approvals, since a model server that touches a historian or MES needs the same change control as any other OT asset.
Next, ask how the inference server stays off the process network. The right answer names a one-way or tightly filtered path from the plant data source, a dedicated server in a separate zone, and a deny-by-default firewall rule set that the partner can show you. Ask how they bind the engine: Ollama listens on 127.0.0.1 port 11434 by default, and the bind address changes through OLLAMA_HOST, so ask who reviews that setting and where it is recorded. For llama-server, ask whether they will enable --api-key and TLS from the first day. Ask how they transfer and verify model weights and container images across the boundary, including checksums, signing, and a scan record. A partner that says models will be pulled from the internet at install time has not thought through the air gap.
Finally, ask how they patch an engine that releases often, what they log, and who runs the service after handover. Ask for their patch test process, how they re-verify model and engine versions in a qualified state, and whether they can keep a pinned version for a validated quality process. Ask what they log (requests, model identifiers, prompts or only metadata) and where those logs live, bearing in mind the EU AI Act's record-keeping duties for high-risk systems in Article 12 record-keeping. Ask how they validate outputs for quality reports under a regime such as 21 CFR Part 11 if it applies to you, and confirm the answer in writing. Finally, get a named operator for the handover, a written runbook and a support boundary. If the answer is that the vendor will operate it indefinitely with no transfer plan, confirm the exit terms before you sign.
How DeployOpen scopes and supports a plant inference service
This sequence turns a plant's maintenance and quality use cases into a written, fixed-scope plan, a hardened on-site inference service, tested endpoints, compliance evidence and an agreed support model. Each step ends with an output the plant and its IT and OT teams can check.
Confirm use cases and plant boundaries
DeployOpen works with the plant's operations, quality, maintenance and OT teams to define the first use cases, such as a maintenance assistant or quality report drafting, and which systems, documents and data each one may touch. The team records which sites, networks and zones the service will run in, and who owns each boundary. The step closes with a written list of use cases, data classes and site locations that the client signs off before any design work starts.
Assess OT zones and data exposure
DeployOpen reviews the existing network segmentation, the zones that hold control systems and historians, and where design files and process recipes are stored. The review maps which data may enter prompts, which must stay out, and how the inference servers would sit relative to the plant network. The output is a zone and conduit diagram with a data-classification table, reviewed by the client's OT security owner. Where the plant follows the ISA/IEC 62443 series, the assessment uses its terminology so the results can be mapped to the plant's own program.
Agree a written scope and deliverables
DeployOpen turns the findings into a fixed-scope proposal that lists hardware or edge nodes, the engine and interface components, authentication, logging, backup, integrations with plant systems, documentation and handover items. Each deliverable has an acceptance test that the client can run. The proposal states what is excluded, such as model training or changes to control systems, and which items the client must provide, such as racks, power or network changes. The step closes when both parties sign the scope document.
Plan and harden the inference nodes
DeployOpen builds the server or edge images, configures the inference engine to listen only on internal interfaces, and places an authenticated proxy or gateway in front of it. The hardening covers access control, TLS, metrics exposure, model storage location, disabled outbound features and host configuration. The team checks the configuration against the agreed design and records the settings in a configuration baseline. The output is a hardened node set that passes the acceptance tests in a staging network before anything reaches the plant floor.
Test endpoints and AI interactions
DeployOpen runs security testing against the inference endpoints, the gateway, the chat or assistant interface and the document inputs, including prompt injection and data leakage scenarios that matter for plant use. Findings are ranked, fixed or accepted in writing, and retested. The closing output is a test report with reproduction steps and a retest record that the client keeps with the system documentation.
Prepare compliance evidence
DeployOpen assembles the evidence the plant's auditors or regulators are likely to request for the chosen scope, such as system descriptions, network diagrams, access lists, configuration baselines, logging settings and test results. DeployOpen does not issue audit reports or certifications. Independent auditors and certification bodies make those determinations. The client confirms with its auditor which evidence formats and retention periods apply, and the output is an evidence index mapped to the agreed controls.
Deploy, hand over and train operators
DeployOpen installs the approved configuration on the plant's nodes, connects the agreed integrations, and runs the acceptance tests on site. Operators and administrators receive runbooks for restarts, model updates, backups, log review and incident response, followed by a training session. The step ends when the acceptance tests pass, the handover package is accepted in writing, and the client's named owner confirms operational responsibility.
Set the support and upgrade retainer
After handover, the client chooses between ongoing support, periodic upgrades, continuous testing or a combination, each defined in its own written agreement. The retainer defines how engine releases and model changes are tested in staging before reaching the plant, how backups are verified and who responds to incidents. Costs recur under the retainer, while the initial build is one-time under the fixed-scope agreement. The output is a documented support model with named contacts, review dates and an agreed change process.
