Amazon Bedrock alternative for air-gapped networks

Bedrock is reached only through AWS endpoints, so it cannot serve a disconnected network. Here is how open-weight models on vLLM with NeMo Guardrails meet that constraint, and what the move involves.

Book a callTalk to us
On this page

Why Bedrock cannot serve an air-gapped network and what replaces it

If your regulated network has no internet path, Amazon Bedrock cannot serve it. Bedrock is reachable only through AWS endpoints, and that remains true even when you connect through PrivateLink, because the service itself runs in AWS's cloud. The workable replacement is open-weight models served on vLLM with NeMo Guardrails, running fully inside your own boundary, with weights, container images and packages moved in by a controlled process. Nothing in that stack needs to call an external service at inference time.

The short answer has three parts. vLLM, licensed under Apache-2.0, exposes OpenAI-compatible endpoints such as /v1/chat/completions, so applications written for the OpenAI API can point at it. NeMo Guardrails, also Apache 2.0, provides the input, output and execution rails that replace Bedrock Guardrails for your policies, and it can run as a server from a Dockerfile. The main trade-off is that you take over the hardware, the patching and the model lifecycle. Confirm with your own architects whether your accelerator supply, staffing and change control can carry that work before you commit to a design.

Before any design work, confirm three things in writing: which Bedrock features your current workloads depend on, whether your classification or regulatory scope requires the disconnected boundary, and whether your procurement rules allow the open-source components you intend to use. The NERC CIP-005-7 and CIP-013-2 requirements, if they apply to your environment, shape how remote access and carried-in software must be controlled, so plan those checks alongside the model choice rather than after it.

What an air-gapped constraint requires from an LLM stack

A disconnected network removes the easy answers. Every requirement below has to be met inside your own boundary, with the same care you would give any other system that carries software across the perimeter. Taken together, they decide whether the model stack can run, be trusted and be kept current without internet access.

No runtime internet dependency

The serving stack must start, route requests and log events without calling any external host. Check each component for telemetry, model downloads, license checks and package installs at startup, and confirm the vLLM server, the guardrails service and the gateway all resolve only internal addresses. Test this by running the stack with outbound traffic blocked and watching for failures, not by reading the configuration alone.

Verified transfer of weights and images

Model weights, container images, Python wheels and Helm charts all cross the boundary on media you control, so each artifact needs a recorded checksum or signature checked on the inside before use. This is the practical form of the software integrity and authenticity verification that NERC CIP-013-2 asks procurement processes to address for vendor software and patches, where that standard applies. Keep a manifest of what came in, when and from which source.

Endpoint authentication beyond the API key

The vLLM --api-key option protects endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM docs warn against relying on it alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Place an authenticating proxy in front of the server, deny the rest by default, and keep administrative and metrics paths on a separate interface.

Isolated multi-node communication

When one model spans several nodes, vLLM's security docs state that communication between nodes is insecure by default, with no authorization or encryption. Treat the inter-node network as a closed segment that only the serving nodes can reach, and confirm that no user-facing route leads to it. This matters most when pipeline parallelism runs across machines that sit in different racks or subnets.

Offline monitoring and metrics

vLLM exposes Prometheus-format metrics at /metrics covering request counts, token throughput, latency histograms and KV cache use. Those scrapes, along with logs and traces, must stay on internal collectors and dashboards that need no external SaaS. Decide which metrics feed alerts, where the retention lives and who can read them, since /metrics also sits outside the --api-key protection.

Upgrades from offline media

Upgrades need a repeatable path: mirror the new release, verify it, test it in a staging enclave and then promote it. Release cadence matters here, because vLLM ships minor releases about every two weeks, so plan a regular intake window rather than ad hoc updates. Confirm with each vendor how it publishes offline bundles and whether security fixes arrive on the same schedule as feature releases.

Which open-source options fit a disconnected network

vLLM is the inference layer that replaces the model endpoint. It is licensed under Apache-2.0, has no paid edition, and runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, as well as on CPUs. The vllm serve command exposes OpenAI-compatible routes such as /v1/chat/completions and /v1/embeddings, so existing client code can point at an internal host. For a disconnected network, the model weights, container images and Python packages must be staged inside the boundary before installation.

  • Deployment options: The vLLM docs list native Kubernetes manifests, Helm, the vLLM production-stack, KServe, KubeRay and other options. Confirm which of these your platform team has already validated against your cluster version.
  • Multi-node traffic: The security docs state that node-to-node communication in a multi-node deployment is insecure by default, with no authorization or encryption. Place those nodes on an isolated network and plan for it explicitly.
  • Endpoint protection: The --api-key option does not cover /metrics, /health or /invocations. The docs recommend a reverse proxy that allowlists only user-facing endpoints.
  • Metrics: Prometheus metrics at /metrics cover request counts, throughput, latency and KV cache use, which supports monitoring without outbound telemetry.
  • Upgrades: New minor releases ship about every two weeks, so an upgrade process that works through an approved offline bundle matters more than in a connected environment.

Confirm with your platform team which vLLM release is approved for the boundary and who signs off on each re-imported image.

How to deploy vLLM and NeMo Guardrails without internet access

This sequence takes an empty isolated network to a working vLLM endpoint behind an allowlisting proxy, with NeMo Guardrails, metrics and a repeatable upgrade path, all without outbound internet access.

  1. Prepare the Kubernetes cluster offline

    The platform team builds the cluster from images held in a private registry rather than pulling from public sources. On a connected staging host, run kubeadm config images list to enumerate the control plane images, then kubeadm config images pull to fetch them. Push the images into the internal registry and run kubeadm init with the --image-repository flag pointing at that registry. Done means every node joins the cluster and all system pods reach Running without any external pull attempts in the registry logs.

  2. Verify and transfer vLLM and NeMo Guardrails artifacts

    Before anything crosses the boundary, the security team reviews each vLLM and NeMo Guardrails release you intend to use. Check the release notes and the license against your approved list, and confirm integrity and authenticity with the vendor or project, since NERC CIP-013-2 expects software integrity checks for vendor software and patches in regulated environments. Build or pull the container images, then mirror them, along with model weights and the nemoguardrails Python package and its dependencies, into the isolated registry and artifact store. Done means every artifact has a recorded checksum and the internal registry serves all of them.

  3. Load model weights into internal storage

    Copy the open-weight model files onto storage that the vLLM pods can mount, such as a persistent volume or an internal S3-compatible store. Record the model identifier, source, revision and checksum in your change log so the exact weights can be reproduced later. Confirm the model's license permits your use before the transfer. Done means a vllm serve test on a single GPU node loads the weights from local storage with no network calls made to any external host.

  4. Place vLLM nodes on an isolated network

    For a single-node deployment, keep the vLLM pods on a dedicated namespace with network policies that permit only the gateway and monitoring traffic. For multi-node runs that use tensor or pipeline parallelism, place every node on an isolated network, because the vLLM security documentation states that inter-node communication is insecure by default, with no authorization or encryption. Apply deny-by-default network policies and restrict the Ray or multiprocessing ports to the node subnet. Done means a port scan from a workload outside the subnet fails to reach the inter-node ports.

  5. Front vLLM with an allowlisting reverse proxy

    Do not expose the vLLM server directly. The vLLM security documentation says the --api-key option does not protect every route, including /metrics, /health, /pause and /abort_requests, and recommends a reverse proxy that allowlists only the endpoints meant for users. Configure the proxy to pass /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings to clients, authenticate every request with your identity provider or mTLS, and block all other paths. Deploy NeMo Guardrails behind the same proxy, serving its HTTP API from the guardrails server image. Done means a request to /pause or /abort_requests from a client returns a denied response, while a chat completion with a valid token succeeds.

  6. Wire Prometheus scraping and dashboards

    Configure Prometheus to scrape the /metrics endpoint on each vLLM pod from inside the cluster, not through the user-facing proxy. The vLLM metrics documentation lists request counts, token throughput, latency histograms and KV cache usage, and histogram buckets can be configured. Add alerts for KV cache saturation and rising latency, and store dashboards in the internal Grafana instance. Done means Prometheus shows active targets for every vLLM pod and a test request increments the request counter and token throughput series.

  7. Plan offline upgrades against the release cadence

    vLLM ships new minor releases about every two weeks, and NeMo Guardrails publishes tagged releases on GitHub, so plan a recurring intake rather than ad hoc imports. Each cycle, pick the release to adopt, run the same verification and transfer steps as before, and test it on a staging cluster using your own evaluation prompts and guardrail tests. Keep the previous image tags in the registry so you can roll back. Ask the vendor or project to confirm that release notes list any breaking API or configuration changes before you import. Done means a dated upgrade record exists with the tested tags, the test results and the rollback target.

How the rails and the serving layer split the work

The rails sit on both sides of the model call, and vLLM only generates tokens. In a NeMo Guardrails deployment, input rails inspect the user's message before it reaches the model, dialog rails steer the conversation flow, retrieval rails filter RAG chunks before they enter the prompt, execution rails check custom actions and tool calls, and output rails screen the completion before the application sees it. The guardrails server calls the model through its own configured endpoint, so in your network the call path is application, guardrails, then the vLLM server. Confirm which of these rail types your application actually needs, because each one adds a model or classifier call that must also run inside the disconnected boundary.

The policy itself lives in files, not in application code. A config.yml file defines the models and the rails to enable, Python actions hold custom checks and tool logic, and Colang files describe the dialog and flow rules. Colang 1.0 is the default, and Colang 2.0 is also supported, so confirm the version before you write new flows. Because these files define behavior, they should be versioned, signed or hash-checked and carried across the air gap with the same care as model weights. Changing a policy then becomes a reviewable change set, and the same files can be promoted from a test enclave to production.

Two deployment paths share one configuration model. You can call NeMo Guardrails from its Python API, with LangChain integration turned on only when you set NEMOGUARDRAILS_LLM_FRAMEWORK=langchain, or you can run the guardrails server, which loads configurations from a folder and exposes an HTTP API. NVIDIA also offers a production microservice built on the same model and deployed with Helm. Choose the library path for a small number of services and the server or microservice path when several applications need one shared policy. Confirm the microservice licensing and image delivery terms with NVIDIA before planning the air-gapped transfer.

Questions buyers ask before an air-gapped LLM build

What drives the cost of an air-gapped vLLM deployment?

The cost comes from GPU or other accelerator capacity, the surrounding infrastructure, implementation and the people who operate it. vLLM itself has no paid edition, so the software line is zero, and the open-source license is Apache-2.0. Hardware is usually the largest item and is bought or allocated once, with power, cooling and replacement recurring. Implementation is a one-time fixed-scope effort, while operating staff and support recur. Ask any partner for a quote that separates hardware, infrastructure, implementation, and ongoing operations and upgrades, so you can see which parts scale with model size and user load. Confirm whether the hardware is already in your data center, since that changes the first-year figure more than any other line.

How long does an air-gapped LLM build take?

The duration depends on how many models and endpoints are in scope, how mature your Kubernetes or container platform already is, and how quickly network and security approvals arrive. Hardware lead times, the number of guardrail policies to write and test, and the monitoring stack you need also move the schedule. Most engagements are fixed-scope, so the timeline is agreed in writing before work starts and tied to specific deliverables. Ask for a phased plan that separates the transfer process, the serving platform, guardrails, monitoring and handover, and confirm which phases depend on your change-control calendar.

What does our side need to provide before the build starts?

Three things usually gate the work. First, network approvals for the isolated segment, including who may move packages and model files across the boundary. Second, hardware that matches the chosen models, which for vLLM means supported GPUs or accelerators and enough memory for the weights and the KV cache. Third, a verified transfer process for bringing container images, Python packages and model weights into the network, with checksums and an approval record for each item. For regulated US energy entities, NERC CIP-013-2 expects procurement processes that address verification of software integrity and authenticity, and CIP-005-7 governs how remote access reaches systems in scope. Confirm with your compliance team which standards apply before scoping.

Can our internal team build and run this without outside help?

Yes, if the team has Kubernetes or container operations skills, GPU server administration experience and someone who can own the guardrail configuration. vLLM's docs cover Kubernetes deployment options, a Prometheus metrics endpoint and a security model that requires you to build your own reverse proxy allowlist. The harder parts are often the transfer process, offline package mirrors, and keeping several components patched without internet access. Map your skills to each component before deciding, and treat the on-call and upgrade rota as part of the staffing estimate rather than a side task. An outside partner can carry the build as a fixed-scope project and hand over documentation, or stay on a retainer for upgrades and testing.

How do we keep Colang rules and model versions current offline?

Set up a controlled update cycle that brings new images, packages and model files across the boundary in batches, with each batch verified before import. NVIDIA's NeMo Guardrails releases are published on GitHub, and the library supports both Colang 1.0 and 2.0, with 1.0 as the default. NVIDIA documents a conversion tool for moving from Colang 1.0 to 2.x, but says converted configurations still need manual review, so plan that review as a test cycle. Keep each guardrail configuration in version control inside the enclave, pin model and library versions, and run a regression set of prompts before each promotion. vLLM ships minor releases about every two weeks, so decide which versions you will adopt rather than following every release.

Can every Amazon Bedrock feature and model access be matched exactly?

No, not in every case, and you should verify each item against your current Bedrock usage. vLLM exposes OpenAI-compatible endpoints for chat, completions, responses, embeddings and audio transcription, so many applications need only a base URL change. Bedrock's model catalog, managed scaling, regional availability and AWS-hosted guardrail features are not replicated by a self-hosted stack. Some Bedrock-specific checks, such as Automated Reasoning, have no direct open-source equivalent. You will also own model selection, quantization and capacity planning that Bedrock currently handles. Build a feature inventory from your applications and mark each item as matched, partly matched or not matched before committing to scope.

Is an offline vLLM endpoint secure without internet access?

Air-gapping limits exposure, but it does not replace endpoint authentication. The vLLM security documentation states that the --api-key option does not protect every route, including /metrics, /health and several control endpoints, and advises against relying on it alone. The recommended pattern is a reverse proxy that allowlists only the endpoints meant for users, plus network controls around the cluster. Multi-node deployments also need isolated node-to-node traffic, since that communication is insecure by default. Have a security engineer review the allowlist and the list of exposed ports as part of acceptance testing.

Evidence assessors ask for in a disconnected LLM stack

A disconnected vLLM and NeMo Guardrails deployment is only as credible as the records behind it. The six items below are the records assessors usually request, and each one shows a specific control working in practice. DeployOpen prepares this evidence, but auditors and certification bodies decide whether it meets their requirements.

Inventory of weights and images

Keep a manifest of every model weight file, container image, Helm chart and Python wheel carried into the network. For each item, record the upstream source, the version, the publication date and a cryptographic hash computed before transfer. Assessors use this list to confirm that what runs in production is what was reviewed. Ask your team to name an owner for the manifest and to store it outside the disconnected network.

Records of each boundary transfer

Each time media or files cross into the isolated enclave, log who carried them, what was transferred, the hashes checked on arrival and the approver. The record should show that integrity checks ran before anything was loaded into the cluster. NERC CIP-013-2 addresses verification of software integrity and authenticity for vendor software and patches, which is the reason this record matters in energy environments. Confirm the procedure with your compliance lead before the first transfer.

Endpoint allowlist and authentication configuration

Export the reverse proxy rules and the vLLM launch settings that show which paths users can reach. The vLLM security documentation notes that the --api-key option does not protect /metrics, /health or several control endpoints, so the allowlist must block them for users. Keep the configuration in version control so reviewers can see when the rules changed and who approved the change. Ask a reviewer to test the allowlist from an unapproved client and keep the result.

Guardrail configuration versions and test results

Store each version of the config.yml file, Colang files and Python actions used by NeMo Guardrails, together with the test prompts and the pass or fail results for each rail. A Colang 1.0 to 2.x conversion needs documented manual review, so keep the review sign-off with the converted version. Assessors will ask whether a change to a rail was tested before it reached production. Tie each guardrail version to the model version it was tested against.

Monitoring and alert records from metrics

vLLM exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. Retain the scrape history and the alert rules that fired on thresholds you set, along with the on-call response notes. Because the metrics endpoint should stay off the user allowlist, confirm that the monitoring system can reach it from inside the enclave. Review these records on a schedule your security team documents.

Upgrade change records

Each vLLM, NeMo Guardrails or operating system update needs a change record that names the old and new versions, the release notes reviewed, the test run and the rollback plan. vLLM ships new minor releases about every two weeks, so a fixed review cadence keeps the backlog manageable. Record which packages were skipped and why. Assessors often ask how a patch reached an isolated system, so tie each change record to the transfer record and the inventory entry.

How DeployOpen deploys LLM services under an air-gap constraint

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including disconnected networks. For an Amazon Bedrock alternative in an air-gapped environment, that means scoping the model, inference, guardrail and monitoring stack, building it inside your boundary, and handing over the runbooks your team needs to operate it without internet access.

How we scope

  • Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
  • We inventory the Bedrock workloads, model use, data classes and network zones that the replacement must cover.
  • The written scope names which models, vLLM endpoints, NeMo Guardrails rails and monitoring components are in and out.

How we build

  • We plan how model weights, container images and packages are transferred across the isolated boundary and verified before use.
  • We place vLLM behind a reverse proxy that allowlists only user-facing endpoints, since the --api-key option alone does not cover every route.
  • We configure NeMo Guardrails rails, Prometheus metrics collection and offline private registries, then test them in your environment.

How we hand over

  • You receive documentation covering architecture, configuration, access control, backup and the offline transfer procedure.
  • We provide runbooks for upgrades, including how to bring new vLLM and NeMo Guardrails releases across the boundary.
  • Handover includes evidence of configuration and testing that your compliance team can use in its own assessments, without us issuing certifications.

How we support

  • Ongoing support, upgrades and continuous testing are available as retainers after handover.
  • Security testing of the LLM endpoints, guardrails and the surrounding APIs can be scoped separately from the build.
  • Support scope, response expectations and upgrade cadence are defined in the written agreement before any retainer starts.

Sources and further reading

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com