Self-hosted LLM inference in air-gapped networks

Self-hosted LLM inference in air-gapped networks starts with the transfer process

Organizations in regulated or classified environments often need a shared model endpoint with no runtime internet access at all. The hard part is not starting the server. It is getting model weights, container images, documentation, chat templates and engine patches across the boundary, and verifying each one before it runs. Once that transfer process is defined, the engine choice follows from it: vLLM for shared GPU serving, Ollama for small local nodes, and Open WebUI or LibreChat as the front end that staff use.

In a connected deployment, the server pulls models and dependencies on demand and updates come from upstream. In an air-gapped network, every artifact is a deliberate import, so each one needs a named source, a checksum or signature check, a record of who approved it and a path into a private registry or model store. Engine upgrades matter here as well. vLLM ships new minor releases about every two weeks, so a transfer process that takes a month per cycle will fall behind quickly. Confirm the cadence you will actually sustain with your security team before you pick the engine version you deploy.

For shared GPU serving, vLLM is the stronger fit: it exposes OpenAI-compatible endpoints, supports multi-GPU and multi-node serving, and has no paid edition, so cost comes from hardware, infrastructure and the people who run it. For small nodes without a GPU cluster, Ollama is simpler to operate, and its local API does not require authentication, so it must sit behind a controlled proxy. Open WebUI can run offline against Ollama or OpenAI-compatible APIs, and LibreChat is an alternative front end. Check the license terms for your chosen front end, particularly branding clauses, before you distribute it internally.

Which requirements shape an air-gapped inference build

This table maps the requirements that most often shape an air-gapped LLM inference build to their source and to the concrete build work they create. Read it as a checklist to agree with your assessor, security team and implementation partner before hardware and software are chosen. The main takeaway is that package transfer, vendor access and endpoint exposure are design decisions, not operational afterthoughts.

Requirement	Source	Build impact
Verify integrity and authenticity of every package carried across the boundary	NERC CIP-013-2, supply chain risk management (for BES Cyber System operators)	Define a transfer process that checks vendor signatures and checksums for model weights, container images, engine wheels and patches before they enter the enclave, and keep a record of each import.
Vendor remote access passes through an intermediate system with encryption and MFA	NERC CIP-005-7, electronic security perimeters (for BES Cyber System operators)	If vendors need interactive access for support, route it through a hardened jump host with MFA and encryption, and add a way to detect and end active vendor sessions.
Inference endpoints must be limited to the routes users need	vLLM security documentation	The --api-key option does not cover every route, so place a reverse proxy in front that allowlists only the user-facing endpoints and blocks operational routes such as metrics and pause controls from user networks.
Inter-node traffic is unauthenticated and unencrypted by default in multi-node runs	vLLM security documentation	Place multi-node GPU hosts on a dedicated isolated network segment and confirm with the vendor what additional protection is needed before running tensor or pipeline parallelism across machines.
Automatic event logs for high-risk AI systems, kept at least six months	EU AI Act Article 12(1) and Article 19(1)	Only applies if the use case is high-risk. Design request logging, retention and export into the enclave's log store from the start, and confirm with counsel whether the use case is in scope.
Metrics and health endpoints must be reachable by operators without exposing them to users	vLLM metrics documentation and vLLM security documentation	Expose /metrics and /health only to the monitoring network, and confirm that the monitoring stack itself is available offline inside the enclave.
Engine and model updates arrive as a verified offline package set	Vendor release cadence (vLLM publishes minor releases about every two weeks), confirmed with your change process	Plan a recurring offline update cycle with test environments, rollback images and change approvals, since each release must pass the same transfer and verification steps.

Which open-source platforms fit an air-gapped model service

An air-gapped model service needs an inference engine, a chat front end and often a gateway, and each piece must be installed, patched and verified without a live connection. The cards below cover the components most often considered, with the license, latest supplied version and the offline checks each one needs before it enters the enclave.

vLLM for multi-GPU shared serving

vLLM is the usual choice when one model service must serve many staff across several GPUs, using tensor parallelism within a node and pipeline parallelism across nodes. It is Apache-2.0 licensed, has no paid edition, and its latest supplied version is v0.31.0 (October 5, 2026). Confirm offline that the --enable-offline-docs option is set, that the model weights and tokenizer are loaded from local paths, and that --api-key is never the only control, since the security docs recommend a reverse proxy that allowlists user-facing endpoints. Multi-node traffic is insecure by default and must sit on an isolated network.

Ollama for smaller nodes and developers

Ollama suits single servers, workstations and developer test benches inside the enclave, where a simple runtime and a local REST API are enough. It is MIT licensed, and its latest supplied version is v0.40.5 (October 14, 2026). By default it binds to 127.0.0.1 on port 11434 and its local API needs no authentication, so an offline deployment must set OLLAMA_HOST deliberately and place a proxy in front of it. Set OLLAMA_NO_CLOUD=1 so that cloud features cannot attempt outbound calls, and confirm that the model directory set by OLLAMA_MODELS lives on approved storage.

llama.cpp as a CPU option

llama.cpp runs quantized GGUF models on CPUs, which matters where GPUs cannot be imported into a restricted zone or where a branch or plant server is the only hardware available. Ollama itself uses a llama.cpp backend, so teams often evaluate both. Confirm the release you intend to carry across the gap, because it is not covered by the vLLM or Ollama version facts in this page. Plan for the model sizes and response times that CPU hardware can deliver, test them against your own prompts, and treat the GGUF files as artifacts that need the same hash verification as any other model weight.

Open WebUI for an offline chat front end

Open WebUI gives staff a browser chat interface that can run entirely offline against Ollama or any OpenAI-compatible API, which makes it a common front end for an enclave. Its license is BSD-style but clause 4 restricts altering or removing Open WebUI branding, except for deployments with no more than 50 end users in any rolling 30-day period, deployments with written permission, or other listed exceptions. Its latest supplied version is v0.12.3 (October 16, 2026). Have legal review the branding terms before rollout, and confirm that no telemetry, update checks or external model lists are reachable from the enclave.

LibreChat under MIT

LibreChat is an MIT licensed chat interface, which avoids the branding conditions attached to Open WebUI and can be a fit where the organization wants to modify the interface freely. Its latest tagged version is v0.8.8. Confirm offline that its model connections point only at local endpoints, that its file upload and search features do not depend on external services, and that the authentication provider it uses can run inside the enclave. Because its feature set differs from Open WebUI, pilot it with the same document and prompt set before choosing one front end for all staff.

LiteLLM as an OpenAI-format gateway

LiteLLM sits between applications and the model servers, exposing one OpenAI-format interface that can route to vLLM and Ollama backends with key management, cost tracking, load balancing and logging. It is MIT licensed except its enterprise directory, and its latest supplied version is v1.105.1 (October 14, 2026). For an air-gapped service, confirm that the provider definitions point only at internal endpoints, that any callback or telemetry integrations are disabled, and that the enterprise directory features you rely on are covered by an agreed license. Its request logs also help evidence the record-keeping that high-risk AI obligations may require.

How the rollout moves from a staging network to a patched production cluster

The sequence turns an open-weight model and its serving stack into a verified, hardened internal service inside an enclave, with a repeatable path for taking each new engine release across the gap. Each phase produces a written artifact that the next phase depends on.

Agree scope, models and acceptance criteria
The DeployOpen team and the client's security, platform and data owners agree in writing which models, context lengths, concurrency targets, engines and front ends are in scope, along with the enclave boundary and who approves each transfer. The output is a signed scope document and a list of acceptance tests that will decide whether the service goes live. Nothing is downloaded or installed until this document exists.
Build the model and image bill of materials
Every artifact the service needs is listed: model weight files with their source and checksums, the vLLM or Ollama release, container images and their digests, Helm charts or manifests, operating system packages, GPU drivers and the documentation and chat templates. Each item gets a version, a license note and an owner. The output is a bill of materials that the client's security team reviews before any transfer is approved.
Stage and verify on the outside network
On the connected staging network, the team pulls the approved items, checks each checksum against the published value, and scans images for known vulnerabilities before packaging them. The image set can be prepared in advance using the Kubernetes approach of listing and pulling control plane images, then mirrored into a private registry. The output is a signed transfer package with a manifest that records hashes and scan results.
Transfer and import inside the enclave
The package crosses the gap through the client's approved media or cross-domain process. Inside the enclave, the team re-verifies every hash against the manifest, imports images into the internal registry, and loads model files into the storage path used by the engine. The output is an import log showing that every item matched its manifest, and a record of who approved the transfer.
Deploy the engine and front end with endpoint hardening
The engine, such as vLLM, and an optional front end such as Open WebUI run from the internal registry, with no outbound route required at runtime. Because the engine's own API key does not cover every route, a reverse proxy exposes only the endpoints meant for users and blocks operational paths such as metrics and health checks from user networks. Inter-node traffic in multi-node deployments is kept on an isolated network. The output is a configuration baseline and a test report showing blocked routes.
Package offline documentation and chat templates
Model cards, engine documentation, admin runbooks and the chat templates each model needs are copied into the package and tested without internet access, so users and operators have what they need offline. Any feature that calls an external service is switched off and confirmed in configuration. The output is an offline documentation set with a version record and a list of disabled external calls.
Run acceptance tests inside the enclave
The team runs the agreed tests inside the enclave: OpenAI-compatible calls from a sample application, load tests against the concurrency targets, confirmation that unauthorized routes are refused, metrics capture, and a check that no outbound connection is attempted. The client signs off against the criteria set in the first step. The output is a test report and a go-live decision recorded in the change log.
Adopt a patch runbook for frequent engine releases
Because vLLM and Ollama publish releases every few weeks, the team writes a runbook that repeats the full cycle for each update: review release notes, rebuild the bill of materials, stage and verify, transfer, import, rerun acceptance tests and roll back if they fail. The client confirms its own patch approval window and who signs each release. The output is a tested runbook and a dated record of the last applied version.

Choosing a kubeadm, single-host or edge pattern for an enclave

A kubeadm cluster suits organizations that already operate Kubernetes and need several inference services, multiple GPU nodes and a shared platform for other workloads. Before the gap closes, the team pulls control plane images with kubeadm config images list and kubeadm config images pull, then points kubeadm init at an internal registry through the --image-repository flag. Container images for the inference engine, CNI plugin, GPU operator and monitoring stack also move across the boundary, usually as signed archives.

Pre-staged: control plane images, network plugin, GPU device plugin, the vLLM image and its model-serving manifests, mirrored into the private registry.
Operator maintains: registry content, certificate rotation, Kubernetes version skew, and a repeatable process for importing each new image set.
Engine upgrades: vLLM ships new minor releases about every two weeks, so the team needs a standing import, test and promotion cycle rather than occasional upgrades.
Multi-node networking: vLLM documents that inter-node traffic is unauthenticated and unencrypted by default, so isolate those nodes on a dedicated network segment.
Staffing: the cluster adds an operating burden that a single host does not, so confirm who carries on-call duty.

Confirm with the Kubernetes and GPU vendors which image sets their current releases require, because the list of pulled images changes with the version.

A single GPU host running vLLM in Docker suits one model service for a defined group of users, with a smaller set of moving parts and a simpler evidence trail. The transfer is a bundle of model weights, the engine container image and its dependencies, plus the offline documentation and chat templates the interface needs. The host has no need for a cluster control plane, so there is less software to verify at each import.

Pre-staged: the vLLM container image, model weights with checksums, tokenizer and chat template files, and any offline documentation assets.
Operator maintains: the host operating system, GPU drivers, container runtime and the reverse proxy that exposes only the /v1 endpoints users need.
Access control: vLLM's --api-key option does not protect /metrics, /health or /invocations, so the proxy must allowlist routes.
Metrics: the Prometheus endpoint at /metrics should be scraped from an internal collector and kept off user-facing routes.
Recovery: a single host is a single point of failure, so decide early whether a warm spare with the same staged artifacts is needed.

Confirm the exact GPU driver and container runtime versions your hardware vendor supports before building the transfer bundle.

CPU-only and edge servers suit branch offices, plants or ships where GPUs are not present or not permitted. Ollama or llama.cpp runs quantized GGUF models on these machines. Ollama defaults to binding 127.0.0.1 on port 11434, and its local API does not require authentication, so any network exposure must go through a proxy the team configures. Models are stored in a directory the operator can relocate with OLLAMA_MODELS, which makes them easier to place on removable media.

Pre-staged: model files in GGUF or Ollama format with checksums, the runtime binary or container image, and a copy of the configuration.
Configuration: set OLLAMA_NO_CLOUD=1 so cloud features stay disabled, and confirm that no outbound calls remain.
Concurrency: the defaults of one parallel request per model and a queue of 512 requests need tuning for shared use.
Fleet management: many small nodes each need their own import, patch and verification record, so standardize a single image.
Patching: Ollama publishes frequent releases, so set a schedule for importing and testing each one, including pre-releases you choose to evaluate.

Confirm that your Ollama build is the version you verified, because pre-releases appear on the same release page as stable versions.

What auditors and assessors ask for from an offline model service

This table maps each evidence item an assessor is likely to request for an air-gapped model service to what it shows and the component that produces it. Read it as a starting inventory to agree with your assessor, since DeployOpen prepares this evidence but independent assessors decide whether it is sufficient.

Evidence item	What it shows	Component that produces it
Image and model hash records	Each container image and model weight file that crossed the gap matches the digest recorded at the source side	Transfer log and checksum manifest kept with the import media
SBOM-style inventory of engine and dependencies	Exact versions of vLLM, Python packages, CUDA or ROCm drivers and system libraries in each image	Build pipeline output in the internal registry
Change record for each engine release	Which release was approved, when it was tested offline, who signed off and what changed	Ticketing system linked to the import of each new vLLM or Ollama build
vLLM Prometheus metrics at /metrics	Request counts, token throughput, latency histograms and KV cache use over time, supporting capacity and availability review	vLLM API server, scraped by an internal Prometheus instance
Access and gateway logs	Who called which endpoint and model, when, and from which client; supports EU AI Act Article 12 and 19 where those articles apply	Reverse proxy or AI gateway in front of the engine, with logs retained on internal storage
Offline documentation and chat template versions	Which local documentation set and chat template each model used, so behavior can be reproduced later	Configuration repository and the Open WebUI or model server deployment record
Network boundary and endpoint allowlist	Only the endpoints meant for users are reachable; admin and metrics paths are restricted	Reverse proxy configuration and firewall rules reviewed at each change

What do buyers ask before hiring help for an air-gapped model service?

How much effort does the transfer and verification pipeline take?

The effort depends on how many artifacts cross the gap and how often they change. Each transfer covers model weights, container images for the inference engine, the gateway and the interface, Kubernetes or host packages, and any offline documentation or chat templates. Each item needs a hash or signature check, a record of its source and an approved path into the enclave. Ask a partner to show the manifest format, the checksum process and who signs off each import before quoting. Because vLLM ships new minor releases about every two weeks, the pipeline is a recurring workload, not a one-time setup. The effort is set by the number of artifacts and the approval steps in your own change process, so ask for both in the written scope.

Can our internal team run the transfer and patching pipeline alone?

Many teams can, provided they already operate Kubernetes or container hosts and have people who understand GPU drivers and model artifacts. The skills that are hardest to hire for are GPU capacity planning, inference engine tuning and verifying software from an isolated boundary. Check whether your staff can run the full cycle: import, scan, verify, deploy, roll back and document. Ask whether the work fits your existing on-call rotation. If the answer is no for any step, that step belongs in the scope of an outside partner or a retainer. DeployOpen plans, implements, hardens, backs up, upgrades, documents and supports self-hosted platforms inside the client's own infrastructure, including air-gapped networks, under written scopes.

How are engine releases patched when they arrive every few weeks?

Patching needs a defined intake, test and release path that runs on a fixed cadence. Track upstream releases, review changes to flags and API behavior, rebuild or re-verify images, test against a staging copy of the enclave and then promote the release. vLLM's releases in the supplied facts were v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, which shows the pace. Decide in advance which releases are security-relevant and which can wait. Ask a partner how they handle rollback, how they record the version in use and how the change reaches your auditors. Skipping releases is a risk decision you should make on purpose.

What drives the cost of an air-gapped model service?

Costs fall into four groups. GPU or accelerator capacity is the largest recurring item and is bought or leased for your hardware. Enclave staff, including the people who run the transfer cycle and on-call coverage, recur every month. Partner fees cover design, build, hardening and handover, and are one-time unless you add a retainer. Paid editions are the fourth group: several vendors gate air-gap deployment behind enterprise tiers, including Mattermost (Enterprise Advanced), Rocket.Chat (excluded from its free Starter plan) and Metabase (Enterprise plan). vLLM itself has no paid edition. Ask any partner to separate one-time from recurring costs and to list every license that depends on the air-gap feature.

What does the work need from our side before it starts?

The partner needs decisions and access that only your organization can give. Expect to provide the target GPU or server inventory, the classification of the enclave, the approved import path and the people who will approve each transfer. You also need an agreed list of models, context lengths and concurrency targets, plus the identity and access design for the endpoint. The vLLM security documentation says the --api-key option alone is not enough and recommends a reverse proxy that allowlists user-facing endpoints, so network and proxy ownership must be settled early. Ask for a written list of client inputs in the scope, with a named owner for each.

Can the build be done with no internet access at all?

Yes, the core stack can be built offline, but the preparation happens on a connected staging side that is separate from the enclave. Kubernetes clusters can be prepared without internet access by pulling control plane images in advance with kubeadm config images list and pull, then pointing kubeadm init at a private registry with --image-repository. GitLab documents offline, air-gapped installs, and Open WebUI can run offline with Ollama or OpenAI-compatible APIs. Some items, such as updated rule sets or model weights, still have to be carried in from outside. Ask the partner which artifacts must cross the boundary, how each one is verified and which features are confirmed to work offline in your version.

Which compliance evidence does an air-gapped build support?

The build can produce evidence for controls about software integrity, access and change records, but it does not make you compliant on its own. For NERC CIP-013-2, the supply chain procurement process must address verification of software integrity and authenticity of vendor software and patches, which maps directly to the checks run on every carried package. For NERC CIP-005-7, remote access into high and medium impact systems must pass through an Intermediate System with encryption and multi-factor authentication. If you are subject to these or other frameworks, ask which controls the partner's deliverables support and which remain yours. DeployOpen prepares systems, controls and evidence for assessments, but independent auditors issue the findings.

What to ask an implementation partner before signing

Start with the import path, because that is where an air-gapped deployment is most often compromised. Ask the partner how every model weight file and container image is verified before it enters the enclave: which hash or signature is checked, where the reference values come from, who holds the signing keys, and whether the check is repeated on the inside after the media is copied. Ask for the written transfer runbook and whether you keep a copy and the right to run it yourself. Under NERC CIP-013-2, the procurement controls must address verification of software integrity and authenticity for vendor software and patches, so ask which of your own supply chain obligations the runbook is meant to evidence, and confirm that with your compliance lead rather than assuming the partner's answer covers it.

Next, ask how the partner closes the gaps the engine leaves open. The vLLM --api-key option protects only the /v1, /v2, /inference and /cohere paths, while /invocations, /pause, /resume, /abort_requests, /metrics and /health remain open, and the vLLM security docs advise against relying on the key alone. Ask for the reverse proxy configuration that allowlists only the paths meant for users, who can call /pause or /abort_requests, and how /metrics (which exposes request and KV cache data) is reachable only from the monitoring network. Ask the same about multi-node runs: vLLM's documentation says inter-node traffic is insecure by default, so ask where the nodes sit, how that network is isolated, and whether any encryption or authorization is added on top. Ask them to show these checks running in a test enclave, not just describe them.

Finally, ask how a new engine release reaches production. vLLM shipped minor releases roughly every two weeks in 2026 (v0.30.0 on September 22 and v0.31.0 on October 5, for example), so a fixed quarterly patch window will leave you behind. Ask for the offline test procedure: where the candidate build is staged, which regression and load tests run against your own models, how output differences are compared with the current version, and who approves promotion. Ask how the partner will bring the new release, its dependencies and its container image across the gap with the same verification as the first install, and how rollback works if the release fails. Confirm the schedule and fees in writing, because the partner's answer on cadence determines your effort and risk for the life of the service.

How DeployOpen scopes, builds and supports air-gapped inference

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, including air-gapped networks. For model serving, that means agreeing in writing what the inference service must do, building it on the client's hardware, and leaving the client's team able to transfer, verify and patch every component without an internet connection.

How we scope

Inventory the models, engines, container images and documentation that must cross the air gap, with named owners for each.
Define the written scope, deliverables and timeline before work starts, since most engagements are fixed-scope.
Confirm with the client's security team which endpoints users reach and which the engine exposes unauthenticated.

How we build

Stand up the inference engine, model store and private registry on the client's own GPUs or servers with no outbound access.
Place a reverse proxy in front of the engine that allowlists only the endpoints meant for users, then test the blocked paths.
Set up Prometheus scraping of the engine's metrics and document how each artifact was verified before it was loaded.

How we hand over

Deliver runbooks for transferring, hash-checking and importing model weights and images across the gap.
Document the offline chat templates, configuration files and access controls so the client's staff can operate the service.
Walk the client's operators through a full engine upgrade from a transfer package, with the steps recorded for their evidence file.

How we support

Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing alongside the initial work.
Plan a repeatable procedure for each engine release, since vLLM has shipped new minor versions roughly every two weeks in 2026.
Test the deployment against the written scope after changes, and prepare the evidence the client's auditors or assessors will request.

Self-hosted LLM inference in air-gapped networks