OpenAI API alternative for air-gapped networks

Why a hosted OpenAI API cannot serve an air-gapped network and what replaces it

On a network where no request, prompt, response or update may cross the boundary, a hosted API is out of reach by definition. The OpenAI API runs in OpenAI's cloud, so every prompt leaves the enclave, and no configuration on your side changes that. The practical replacement is open-weight models served inside the enclave on vLLM, or on Ollama for a small team with light traffic, behind an authenticating reverse proxy that is the only path users can reach. Model weights and container images arrive through a controlled transfer that you verify before they are loaded. This page does not cover OpenAI's own contractual options for isolated or dedicated deployments. Confirm directly with OpenAI whether any such arrangement exists, what it includes and whether it satisfies your network rules, because the facts used here address only self-hosted options.

The serving layer needs a few decisions made early. vLLM exposes OpenAI-compatible endpoints such as /v1/chat/completions and /v1/embeddings, so existing applications can point at the enclave without a rewrite, but its --api-key option does not protect every route. The vLLM security documentation says it does not cover /metrics, /health or /invocations, and recommends a reverse proxy that allowlists only the endpoints meant for users. Ollama binds to 127.0.0.1 on port 11434 by default and its local API has no authentication, so exposing it requires a proxy with its own access control. For multi-node vLLM, the security documentation states that node-to-node traffic is insecure by default, so those nodes must sit on an isolated network. Your security team should test each allowed route from outside the proxy and confirm that every other path returns a refusal.

The transfer process and the update cycle carry the most ongoing work. Every model file, container image, Python package and Helm chart crossing the boundary is a supply chain input, and the NERC CIP-013-2 control on verifying software integrity and authenticity is a useful model for the checks you run on each one. vLLM ships new minor releases about every two weeks, so decide who reviews them, how they are tested against your prompts and how often the enclave takes a new build. Ask your implementation partner to document the import procedure, the offline reference material and the rollback steps before go-live, and confirm that your air-gapped support obligations, including any vendor offering for isolated deployment, are written into the contract.

What an air-gapped LLM deployment must satisfy

An air-gapped inference service fails in a few predictable places: a stray outbound call, an unprotected admin endpoint, traffic between cluster nodes, or an unverified package carried across the boundary. The cards below set out what each requirement means and what to check before go-live. The energy-sector example is included for readers whose rules come from NERC.

No outbound calls at runtime

Confirm that nothing in the serving stack reaches the internet while requests are running. Ollama binds to 127.0.0.1 on port 11434 by default, so remote access must be set deliberately through a proxy. Its cloud features can be disabled with OLLAMA_NO_CLOUD=1 or in the server configuration file. Test the claim with an egress-blocked network segment rather than trusting the configuration alone, and repeat the test after every upgrade.

Endpoint authentication beyond the API key

The --api-key option on vLLM protects only the /v1, /v2, /inference and /cohere paths. It does not protect /invocations, /metrics, /health, /pause, /resume or /abort_requests. vLLM's own security guidance recommends a reverse proxy that allowlists only the endpoints meant for users. Ask your team to list every route the server exposes and to confirm which of them the proxy blocks from application networks.

Isolation between cluster nodes

Multi-node vLLM deployments use Ray or a multiprocessing backend across machines. The vLLM security docs state that communication between nodes is insecure by default, with no authorization or encryption, and must be protected by placing nodes on an isolated network. Plan a dedicated inference network segment for node-to-node traffic, separate from user and management traffic, and verify that no other host can reach the cluster ports.

Verified software transfer

Model weights, container images, Python wheels and Helm charts all cross the boundary on removable media or a one-way transfer. NERC CIP-013-2 is a useful reference: its supply chain controls expect procurement to address verifying the integrity and authenticity of vendor software and patches. Record a checksum or signature for each artifact at the source, verify it on arrival, and keep the manifest with the transfer record so an auditor can trace what entered the network and when.

Offline documentation and interfaces

A chat front end and the serving engine often try to fetch documentation, fonts or scripts from public content delivery networks. vLLM provides an --enable-offline-docs option for air-gapped environments, and Open WebUI documents running fully offline with Ollama or OpenAI-compatible APIs. Load the interface in a browser on an isolated test host and watch for failed requests in the developer tools before you accept the deployment.

Remote access paths for energy sector

Energy-sector readers may face NERC CIP-005-7, which requires all Interactive Remote Access to high and medium impact BES Cyber Systems to pass through an Intermediate System with encryption and multi-factor authentication. The rule is an example of how regulators expect vendor access to be controlled. Confirm with your compliance team whether the inference service falls under a CIP scope before deciding how vendor support sessions reach it.

Which self-hosted serving options fit an isolated network

A production vLLM cluster is the option for many concurrent users on an isolated network. vLLM is an Apache-2.0 licensed serving engine with no paid edition, so the software itself carries no license fee; the cost sits in accelerator hardware, the surrounding infrastructure, implementation and the staff who operate it. Its vllm serve process exposes OpenAI-compatible endpoints, so applications written for the OpenAI API can point at a host inside the enclave without code rewrites. Because nothing needs to reach the internet, the cluster can run entirely on internal networks once model weights and container images are imported.

Authentication gap: The --api-key option does not cover every route. The vLLM security docs say it protects the /v1, /v2, /inference and /cohere paths but not /metrics, /health or several control endpoints, and they warn against relying on it alone.
Reverse proxy allowlist: Place a reverse proxy in front that exposes only the endpoints users need and blocks control routes from user networks.
Multi-node traffic: The security docs state node-to-node communication is insecure by default, so multi-node deployments need an isolated network segment.
Operations load: Upgrades arrive about every two weeks in minor releases, and each one must be imported, tested and approved inside the enclave.

Confirm with your engineering team which Kubernetes option (native manifests, Helm or another stack) fits your change control, and who signs off on each imported release.

Ollama suits a single host or a small team that needs a local model without building a serving cluster. It is MIT licensed, runs on Linux, macOS, Windows and Docker, and its local API at port 11434 works without internet access once models are on disk. Ollama states that local runs do not expose prompts to its servers, and setting OLLAMA_NO_CLOUD=1 turns off cloud features, which matters on a network where outbound calls are blocked or monitored. Its paid plans only add cloud models and team features, so running models on your own hardware carries no license charge.

Concurrency default: OLLAMA_NUM_PARALLEL defaults to 1 parallel request per model, so several users will queue behind each other unless you raise it and size memory to match.
Loaded model limit: OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE cap how many models stay in memory and how many requests wait.
No authentication locally: The local API does not require a key, so it binds to 127.0.0.1 by default and needs a proxy with access control before any other host can reach it.
Model storage: OLLAMA_MODELS moves the model directory to a volume you control, which simplifies offline model imports.

Check whether your expected concurrency stays within what one host can serve, and whether your model files are approved for transfer into the enclave.

Users often need a chat interface rather than raw API calls. Open WebUI can run entirely offline with Ollama or any OpenAI-compatible API, so it can sit in front of either the vLLM cluster or the Ollama host without external calls. It is a presentation layer, so it does not replace the access controls or logging of the serving stack. Treat it as an application that needs its own container image imported, its own authentication configuration and its own patch cycle inside the enclave.

Offline operation: Open WebUI documents running with local models and OpenAI-compatible APIs, so you can point it at internal endpoints only.
Offline documentation: vLLM provides an --enable-offline-docs option for environments without internet access, which keeps its API documentation available to internal users.
Image transfer: The Open WebUI container and its dependencies must pass the same integrity checks as the model server images.
Identity: Confirm how users sign in, and whether accounts are created locally or through an internal directory.

Verify with your security team that the front end makes no calls to external services in your configuration, and test this with network egress blocked.

Where the hosted OpenAI API stops short for isolated networks

The hosted OpenAI API cannot meet a true air-gap requirement by design. Every request is sent over the network to OpenAI's hosted endpoints, so any workload that must keep prompts, documents and responses inside a closed network has to be served from somewhere else. This is a property of how the service is delivered, not a judgment about its security. For an isolated enclave, the practical question is therefore not whether prompts are protected in transit, but whether the network can allow outbound calls at all, and most isolated environments by definition cannot.

The supplied research does not describe any isolated, offline or on-premises deployment option for OpenAI's models. Before you rule the hosted API in or out for a specific enclave, ask OpenAI directly whether it offers any form of disconnected or customer-hosted deployment, which regions and endpoints would be involved, what network paths it needs, and what the contract says about data flows. Get the answer in writing and check it against your network architecture team's egress rules, since a vendor-side option that still calls back to a service would not satisfy the constraint.

If the answer is no, the decision moves to self-hosting. You would need to plan for the work that moves to your team: selecting open-weight models that your licenses permit, transferring weights and container images across the boundary, keeping the serving stack and its dependencies current, and operating the endpoint. Treat the hosted API as a reference for application compatibility rather than as a component that can sit inside the enclave. The sections that follow describe how the open-source alternatives cover those needs.

How to deploy and update vLLM without internet access

This sequence takes a vLLM inference endpoint from image and weight transfer through verified, isolated serving, and it repeats the same path for every vLLM release and model upgrade, so the air-gapped environment stays current without an internet connection.

Build the Kubernetes control plane from a private registry
The platform team pulls the control plane images on a connected staging host using kubeadm config images list and kubeadm config images pull, then pushes them to an internal registry. Cluster bootstrap then runs kubeadm init with the --image-repository flag pointing at that registry, so no node reaches a public registry. Done means every node starts its control plane pods from the internal registry and the cluster reports healthy with no outbound connections. Confirm the Kubernetes version your image set matches the version you intend to run.
Stage vLLM images and model weights on removable media
On the connected side, the team pulls the vLLM container image from the release channel tied to the version under review, pulls the chosen open-weight model files, and records a checksum for every file. These are copied into the internal registry and an internal file store, and the transfer media is logged. If Ollama is used on a workstation or single host, set OLLAMA_MODELS to the directory where the transferred model files are placed so the runtime reads them from there. Done means each artifact has a recorded hash and loads from internal storage with no network access.
Verify every package before it crosses the boundary
Security engineers compare each image digest and model file checksum against the values recorded on the connected side, and they scan images with the scanner your organization already approves before they are admitted. This is the control point that supports the software integrity and authenticity expectations in NERC CIP-013-2 where that standard applies, and similar supply chain procurement requirements elsewhere. Done means a signed verification record lists each artifact, its hash, its scan result and the approver. Confirm with your compliance lead which records your assessor expects.
Deploy vLLM behind a reverse proxy with an allowlist
Platform engineers run vllm serve inside the cluster, with the --api-key option set, and place a reverse proxy in front of it. The proxy forwards only the endpoints meant for users, such as /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings if used, and rejects everything else. The vLLM security docs warn that --api-key alone does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, so those paths must be blocked at the proxy. Done means a request to an unlisted path returns a denial from the proxy and an authenticated request to a listed path returns a completion.
Protect metrics and health endpoints
vLLM serves Prometheus-format metrics at /metrics, which expose request counts, token throughput, latency and KV cache use. Keep that endpoint and /health reachable only from the internal monitoring network or the Prometheus server's address, never from user networks, and restrict scraping through network policy. Done means a scrape from the monitoring namespace succeeds, a request from a user subnet is refused, and the dashboards read the same series from the internal scraper.
Isolate node-to-node traffic in multi-node deployments
If vLLM spans several nodes using tensor or pipeline parallelism through Ray or its multiprocessing backend, the vLLM security docs state that node-to-node communication is insecure by default, with no authorization or encryption. Place these nodes on a dedicated isolated network segment with network policies that permit only the cluster peers that need to communicate. Done means a host outside the segment cannot reach the inter-node ports, and the inference service still starts across all nodes.
Repeat the transfer path for each vLLM release and model upgrade
vLLM ships new minor releases about every two weeks, with v0.31.0 dated October 5, 2026 among those listed in the researched release history. Plan a fixed review cadence: the team selects the release, pulls its image and any new dependencies on the connected side, runs the same verification and scanning steps, and stages the artifacts through the same controlled transfer. A model upgrade follows the same path with new weights, new checksums and a regression run of your prompt set before traffic moves. Done means the new version is signed off in a change record, running in a staging namespace, and the previous version remains available for rollback.

Evidence assessors and regulators ask for in an isolated LLM deployment

An air-gapped LLM deployment is judged on what it can prove: what entered the network, who allowed it, how traffic is confined, who may call the endpoint, and how changes were made and reversed. The six artifacts below connect those questions. DeployOpen prepares this evidence inside your environment, but independent auditors and certification bodies issue any assessment or certification.

Inventory of transferred models and images

Keep a manifest listing every model weight file, container image, Helm chart and package that crossed the boundary, with its version, source, size and cryptographic hash. Ollama stores models in a directory that OLLAMA_MODELS can relocate, and vLLM images are pulled from a registry, so both should be recorded at the point of transfer. Ask the assessor which hash algorithm and manifest format they accept before you build the list.

Approval record for each transfer

For every import, record who requested it, which technical reviewer checked the contents, the scan results, and who approved it for release across the isolated boundary. Assessors often ask for this trail under supply chain controls such as NERC CIP-013-2, which expects procurement processes to verify software integrity and authenticity of vendor software and patches. Confirm with your own compliance team which framework governs your environment.

Network diagrams and no-egress firewall rules

Provide diagrams that show every network segment, the inference hosts, the gateway and the transfer point, plus the firewall rules that block outbound traffic from the inference zone. Include the vLLM note that node-to-node traffic is unauthenticated and unencrypted by default, and show how an isolated network protects it. Test the rules and keep the test output, because a diagram alone does not prove that egress is blocked.

Endpoint authentication configuration

Document how clients authenticate to the inference endpoint. The vLLM --api-key option does not cover every route, including /metrics and /health, so the record should show the reverse proxy allowlist that exposes only user-facing endpoints, and the key or identity method behind it. Local Ollama has no authentication by default, so its configuration must show the proxy in front of it. Keep the configuration files under version control.

Log and metrics retention records

Show what the system logs, where those logs and vLLM Prometheus metrics at /metrics are stored, how long they are kept and who can read them. Prompt and response content is often the sensitive part, so state whether it is logged at all and how it is redacted or excluded. Set the retention period with your own security policy and confirm it with the assessor, rather than adopting a default from a tool.

Update and rollback history

Record each model, runtime and container change: the old and new versions, the test results, the approver and the rollback procedure tested before release. vLLM ships new minor releases about every two weeks, so an update log will fill quickly. Keep the previous image and model files staged inside the enclave so a rollback never requires an external download.

Questions buyers ask before moving an LLM service into an air-gapped network

What does it cost to run an LLM service in an air-gapped network?

The cost is mostly accelerator capacity, the enclave build, the transfer process and the people who operate it. vLLM and Ollama have no license fee to pay (vLLM is Apache-2.0 and Ollama is MIT), so the recurring spend is hardware, power, storage, staff time and any support contract you choose. The one-time costs are the isolated environment, the artifact pipeline that moves model weights and container images across the boundary, and the initial testing. Ask any implementation partner for a quote built from a written scope that lists each component separately, and ask which parts recur after handover. Hardware is the largest variable, and it depends on model size, concurrency and latency targets, so confirm those numbers in your own environment before you compare options.

How long does an air-gapped LLM deployment take?

The duration depends on how much of the enclave already exists, how many approvals the transfer process needs, and how many models and services must be validated. A team with an existing isolated Kubernetes cluster and an approved software intake process will move faster than one building the enclave from scratch. Time also goes to hardware procurement and delivery, which often sets the critical path. Ask for a timeline tied to written deliverables and to named dependencies, such as hardware arrival, security review and acceptance testing, rather than a single end date.

Can our own team build and run an air-gapped LLM service without a partner?

Yes, if you already operate Kubernetes or GPU servers, have a documented process for moving software across your boundary, and have staff who can patch and monitor inference services. vLLM documents its deployment options, including native manifests and Helm, and Ollama runs on Linux and Docker, so the software side is within reach of an experienced platform team. The usual gaps are GPU driver and capacity planning, endpoint hardening, and evidence collection for assessors. A partner is most useful for fixed-scope work such as the enclave build, the transfer pipeline and the compliance evidence, while your team keeps day-to-day operations.

How do model upgrades work when there is no internet access?

Every upgrade is an import. Someone downloads the new model weights, the container image and its dependencies on a connected system, verifies their integrity and authenticity, and carries them across the boundary on approved media. Inside the enclave, you load the new artifacts into your model store and registry, then test before routing traffic. vLLM ships new minor releases about every two weeks (for example, v0.31.0 on October 5, 2026), so you should decide an upgrade cadence and a skip policy in advance. For NERC CIP-013-2 style programs, the verification step is the control that matters most, so record the hashes and approvals for each import.

Is Ollama enough for a production LLM service in an isolated network?

Ollama is useful for single-user work, development and small internal tools, but production multi-user traffic usually needs vLLM or a similar serving engine. Ollama's defaults are conservative: OLLAMA_NUM_PARALLEL defaults to 1 parallel request per model and OLLAMA_MAX_QUEUE defaults to 512 queued requests. Its local API does not require authentication, so you must place a proxy in front of it with access control. vLLM offers OpenAI-compatible endpoints, Prometheus metrics at /metrics and Kubernetes deployment options, which give operators more control. Set OLLAMA_NO_CLOUD=1 on any air-gapped host so cloud features stay disabled.

Does an air-gapped LLM setup meet a specific compliance framework?

No deployment satisfies a framework by itself, and only the assessor or certifying body can decide. Your evidence for NERC CIP-005-7 would cover how remote access passes through an intermediate system with encryption and multi-factor authentication, and your evidence for CIP-013-2 would cover how you verify software integrity and authenticity of packages carried across the boundary. For CMMC under 32 CFR Part 170 or FedRAMP, confirm first whether your system boundary is in scope. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports or certifications. Confirm the control mapping with your assessor before you build.

What should we check on the endpoint before exposing an LLM API inside the enclave?

Treat the inference endpoint as a service that must be authenticated and isolated. The vLLM security documentation states that the --api-key option does not cover every route, including /metrics, /health and /invocations, and it recommends a reverse proxy that allowlists only the endpoints meant for users. Multi-node vLLM traffic is insecure by default, so nodes must sit on an isolated network. Confirm your proxy blocks administrative routes, your metrics endpoint is reachable only by monitoring, and your network policy denies all outbound traffic from the inference hosts.

How the work is scoped and handed over

Air-gapped engagements are fixed-scope. Before any work starts, the scope, deliverables and timelines are agreed in writing. For this topic, that scope should name the models to be served, the inference engine and version (vLLM or Ollama), the gateway and authentication design, the update path for model weights and container images, and the documentation handed over at the end. Because the network boundary decides almost every technical choice, the written scope should also state which environment is in use, such as a data center, a government enclave or a classified segment, since that determines what can be installed and how. Ask for the timeline to be tied to named milestones, including a hardware readiness check and a transfer rehearsal, rather than a single completion date.

The client supplies the hardware, the approvals and the transfer rules. The client provides the GPU or accelerator capacity, the servers, storage and the isolated network segment, along with the security approvals needed to bring software and model files into it. The transfer process is usually the hardest part to plan: confirm with your security team how media is scanned and logged on the way in, who signs off each import, and how the package checks described in NERC CIP-013-2 for software integrity and authenticity will be met for model weights, Python wheels and container images. Also confirm which staff can access the inference endpoints and the administrative interfaces, because those roles should be set before the first model is loaded.

Support and upgrades run as retainers after handover. Once the system is live, ongoing support, security patching, model and engine upgrades, and periodic testing of the endpoint are available as a retainer, which suits teams that cannot take a new inference release into production without a supervised rebuild. Ask whether the retainer covers offline update bundles, the time to validate a new vLLM or Ollama release before it reaches users, and who responds when the isolated system fails. Confirm these terms in writing alongside the fixed-scope deliverables so that handover and the ongoing service have clear owners.

How DeployOpen deploys open-source LLM inference under an air-gap constraint

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own air-gapped networks. For LLM inference, that means scoping which open-weight models, serving engine and gateway run in the isolated environment, then building and handing over the full offline stack.

How we scope

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
We map which applications call the endpoint, which models they need and which data classes may reach the isolated network.
The written scope names the serving engine, gateway, authentication method and update path so each decision is reviewable.

How we build

We stage model weights, container images and packages through a controlled transfer process with integrity checks before they enter the boundary.
We place vLLM or Ollama behind a reverse proxy that allowlists only the endpoints meant for users, and test that blocked routes stay closed.
We test network isolation, endpoint authentication and offline documentation, and record the results as evidence for your assessors.

How we hand over

We deliver runbooks for model transfer, version pinning, rollback and the offline update procedure your operators will repeat.
We document the architecture, the allowlisted endpoints, the configuration keys and the verification steps for each release.
Handover includes a walkthrough with your operations and security staff so they can run the stack without us.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainer work can cover re-validating a new model or serving engine release before it is carried across the boundary.
Support scope, response expectations and review cadence are agreed in writing, matching what your environment requires.