Self-hosted Portkey alternative for air-gapped networks

Can a self-hosted Portkey alternative run in an air-gapped network?

Yes. A self-hosted gateway such as LiteLLM, or the MIT-licensed Portkey AI Gateway, can sit in front of local vLLM or Ollama models on a network with no outbound path at all. The reason is structural: Portkey's hybrid model keeps the gateway in your VPC but has it call out over HTTPS to control.portkey.ai, so a disconnected network cannot run that split. A fully self-hosted gateway removes the control plane from the picture, so routing, keys, budgets and logs all live inside your boundary. DeployOpen implements this as fixed-scope work, with scope, deliverables and timelines agreed in writing before the build starts.

The first decision is which components the enclave needs. LiteLLM uses PostgreSQL for virtual keys, budgets and spend tracking, and its production guide requires Redis 7.0 or newer once more than one proxy instance runs. Ollama binds to 127.0.0.1 on port 11434 by default and its local API does not require authentication, so the OLLAMA_HOST setting and a reverse proxy in front of it are part of the design. vLLM's --api-key option does not cover /metrics or /health, so the docs recommend a reverse proxy that allowlists only user-facing endpoints. Confirm each of these in your own environment before you approve a topology.

What an air-gapped LLM gateway must satisfy

A gateway on a disconnected network has to handle its own identity, budgets and evidence without help from outside. The four requirements below connect directly to why a hybrid gateway that calls a vendor control plane cannot be used in that setting.

No calls to a vendor control plane

The gateway must start, route, authenticate and log without any outbound HTTPS to a vendor service. Portkey's hybrid deployment keeps the gateway in your VPC, but the gateway still calls out over HTTPS to control.portkey.ai, so it fails this requirement on a disconnected network. A self-hosted LiteLLM proxy or the MIT-licensed Portkey AI Gateway can run without that dependency. Confirm during testing by blocking all egress and checking that every feature still works, including startup, model calls and key checks.

Keys and budgets in local PostgreSQL

LiteLLM virtual keys need a PostgreSQL database, and that database holds per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions. Keep it inside the enclosure with backups, restore tests and access controls under your own administrators. LiteLLM's production guide names PostgreSQL as the primary supported database, so plan for it as a core component rather than an optional one. Ask whether any budget counters are cached elsewhere and where they are reset.

Redis for shared state across instances

Once more than one proxy instance runs, shared rate limits, router state and cache need a common store. LiteLLM's production guide requires Redis 7.0 or newer for that setup, so it must also run on the local network. Without it, each instance enforces limits independently and budgets can be exceeded in practice. Confirm that the Redis deployment has no replication or monitoring path that leaves the enclave.

Logs and metrics kept local

Request logs, token counts and spend records should land in systems you operate, such as a local Langfuse instance, OpenTelemetry collector or Prometheus. LiteLLM can send logs to these destinations, and vLLM exposes Prometheus-format metrics at /metrics. Check which fields contain prompts and responses, set retention, and confirm that dashboards and alert channels do not send data to external services. Prompt text is often the most sensitive content in the gateway.

Model endpoints reachable only internally

The inference servers behind the gateway, such as vLLM or Ollama, should accept traffic only from the gateway's network segment. The vLLM --api-key option does not protect every endpoint, including /metrics and /health, so the docs recommend a reverse proxy that allowlists only the endpoints users need. Ollama binds to 127.0.0.1 by default and its local API has no authentication, so network placement and proxy rules carry the access control. Test both with a scan from outside the gateway segment.

Where Portkey's hybrid and hosted models stop at the network boundary

In the hosted model, Portkey runs the gateway and the control plane in its own infrastructure. Application requests pass through Portkey before they reach the model provider, so prompts, responses, routing configuration and logs are handled outside your network by design. This model depends on continuous connectivity from your applications to Portkey's service, which makes it unsuitable for a disconnected environment regardless of the features it offers.

Request path: Prompts and completions transit Portkey's service before reaching the upstream provider.
Logs and configuration: Hosted logs, virtual keys and routing settings live in Portkey's environment.
Connectivity: Applications need a working route to Portkey at all times, so an air-gapped segment cannot use this model.
Enterprise features: Logging, access control and prompt management depend on your plan, and you should confirm how each is stored and where.
Data residency: Ask Portkey in writing which regions host the service and what retention applies to request data.

Confirm with Portkey whether any hosted feature can operate without outbound access, and get the answer in your contract or security documentation.

Which open-source gateway fits a disconnected network

This table compares LiteLLM proxy and the MIT-licensed Portkey AI Gateway for a network with no outbound path to a vendor control plane, with Kong and APISIX noted at the end. Read each row against your own air-gap rules, especially the licensing and database requirements, which decide what you can run without an enterprise contract.

Criterion	LiteLLM proxy	Portkey AI Gateway (open source)
Fit with an air-gapped constraint	Runs as a self-hosted server inside your network and calls only the local vLLM or Ollama endpoints you configure. No vendor control plane is required.	The gateway runs in your network with npx, Docker, Node.js or Kubernetes. Confirm that the open-source build starts and routes with no outbound calls to Portkey services in your setup.
Licensing	MIT except content under the enterprise directory, which has its own license. Core proxy features can be used under MIT. Check the LICENSE file before you adopt any enterprise module.	The gateway is MIT licensed. Prompt management, semantic caching, advanced logging and tracing, access control and PII redaction are listed as enterprise features outside it.
Virtual keys and budgets	Virtual keys require a PostgreSQL database. Per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions are available, all stored in your database.	Access control sits in the enterprise feature set, so key and budget controls in the open-source gateway are limited. Confirm what the open build supports before you plan key issuance.
Logging and audit storage	Logs can be sent to Langfuse, OpenTelemetry, Prometheus or Datadog. Choose a destination that also runs inside the network, since a hosted sink would reintroduce egress.	Advanced logging and tracing is an enterprise feature. Plan for your own logging stack, such as OpenTelemetry, and confirm how request records are captured without it.
SSO and audit licensing	The admin UI supports SSO with Google, Microsoft Entra ID, Okta and generic OIDC. From v1.76.0, SSO is free for up to 5 users, and an Enterprise license is needed beyond that. Audit logs are an Enterprise feature.	SSO and audit controls are not described as part of the open-source gateway. Confirm the authentication model with the project before you rely on it for regulated access.
Operating needs	Production guidance names PostgreSQL as the primary database and Redis 7.0 or newer once more than one proxy instance runs. You own both services, their backups and their patching.	Fewer required services for a single instance, but the routing configuration, upgrades and security patches are your responsibility. Confirm the operating model for multi-instance use.
Trade-offs	Broad provider coverage and a mature key model, offset by the database dependency and by the enterprise boundary for SSO beyond five users.	Permissive license and simple deployment, offset by fewer built-in controls for keys, budgets and audit, which you would need to build or add.
Kong or APISIX (note only)	Choose these only if you already run them for APIs. Their AI plugins are documented in open-source gateways, but multi-model features in some tiers depend on enterprise editions. Confirm which plugins your edition includes before you commit.	Not applicable

How to deploy LiteLLM in front of vLLM or Ollama without outside calls

This sequence walks an internal platform team through standing up a LiteLLM gateway in front of local vLLM or Ollama models, with every image, package, key and log kept inside the network. The end state is one OpenAI-compatible endpoint that issues keys, enforces budgets and writes logs to collectors you run.

Mirror images, packages and model weights inside
Before any cutover, the platform team moves container images, Python packages and model weights into the network through your approved transfer process, such as a scanned media or one-way transfer path. Confirm that the LiteLLM image, the vLLM image, the Ollama binary and the model files all come from sources your security team has reviewed, and that nothing in the runtime path fetches packages or weights from the internet. Done means the disconnected environment can install and start each component from internal repositories alone.
Deploy PostgreSQL and Redis internally
Platform engineering deploys PostgreSQL as the LiteLLM database, since virtual keys need it, and Redis as the shared store for rate limits, router state and cache. LiteLLM's production guide asks for Redis 7.0 or newer once more than one proxy instance runs, so pin that version or later in your internal registry. Done means both services accept connections only from the LiteLLM subnet and the backups for each run inside your own storage.
Store the master key in a local secret manager
The security team generates LITELLM_MASTER_KEY, which must start with sk-, and places it in a secret manager that runs inside the network, such as HashiCorp Vault or a Kubernetes secret backed by your own key management. LiteLLM also supports Vault and other secret manager integrations in its Enterprise tier, so confirm which option your license covers before relying on it. Done means the key is absent from configuration files, container images and shell history, and the proxy starts by reading it from the secret store.
Run vLLM behind an allowlisting reverse proxy
Platform engineering starts vllm serve with the --api-key option, then places a reverse proxy in front of it. vLLM's security docs say the key does not cover /metrics, /health, /invocations, /pause, /resume or /abort_requests, so the proxy should pass only the /v1 endpoints that users need and block the rest from user networks. Run one Uvicorn worker per pod if you deploy on Kubernetes, and keep the vLLM nodes on an isolated network, because multi-node traffic is unauthenticated by default. Done means a request to an unlisted path fails at the proxy and a request to /v1/chat/completions succeeds with the key.
Lock down Ollama bindings and cloud features
For Ollama, the team keeps the default 127.0.0.1 binding on port 11434 when the proxy runs on the same host, or sets OLLAMA_HOST deliberately to an internal interface when it does not. Set OLLAMA_NO_CLOUD=1 in the server environment so cloud features stay off, since the local API has no authentication and direct cloud inference needs an ollama.com API key. Add a reverse proxy that handles access control if other clients must reach it. Done means outbound connections from the Ollama host to ollama.com are blocked at the network firewall and a test request to the local endpoint still returns a model response.
Configure virtual keys, budgets and routing
The gateway team registers each local model in the LiteLLM configuration, then issues virtual keys per application, user or team with budgets, TPM and RPM limits and model restrictions. Routing uses the simple-shuffle default or another strategy the team tests, with retries, cooldowns and priority ordering for failover between vLLM and Ollama backends. Confirm which of these controls require an Enterprise license, such as SSO beyond the free user limit, audit logs and team guardrails. Done means a key limited to one model is refused for others and a deliberately exhausted budget blocks further calls.
Send logs to internal collectors and verify
The observability team points LiteLLM logging at a Prometheus endpoint or an OpenTelemetry collector that runs on internal hosts, and scrapes vLLM's /metrics from the monitoring network rather than through the user-facing proxy. Confirm that prompt and response content is either excluded or stored only where your retention policy allows, and that no logging destination is an external SaaS. Finish with a check that traffic captured at the perimeter shows no connections leaving the network from the gateway, vLLM or Ollama hosts. Done means the dashboards show request counts and spend, and the egress test is recorded as evidence for your assessment.

How keys, budgets and logs stay inside the network

In a self-hosted LiteLLM deployment, the gateway holds the key, budget and limit state itself. Virtual keys, along with their per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions, are stored in PostgreSQL, which LiteLLM names as its primary supported database and which virtual keys require. Because that database sits in your network, no key or spend record needs to leave it. Decide who owns the database, how it is backed up, and which team can issue or revoke virtual keys, since a revoked key is only as fast as the database write that records the change. Confirm with the LiteLLM documentation how budget and spend updates are batched at your scale before you set alerting thresholds.

Logs stay inside the network when you point the gateway at destinations you run. LiteLLM can emit logs to Prometheus and OpenTelemetry, and vLLM exposes Prometheus-format metrics at /metrics covering request counts, token throughput, latency and KV cache use. Keep that endpoint on an internal network, because vLLM's own security docs say the --api-key option does not protect /metrics or /health and recommend a reverse proxy that allowlists only the endpoints meant for users. If you add Langfuse for prompt and trace review, treat it as a second stateful system to self-host. Langfuse depends on PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage, so each of those needs its own backup, access control and patching plan.

Before go-live, verify three things in writing: that no component in the gateway path has an outbound dependency that fails in your network, that each log destination is internal and access-controlled, and that the key and spend data a reviewer would need can be exported from PostgreSQL. Also note that LiteLLM SSO beyond the free user limit, SCIM, audit logs and IP allowlisting sit behind an Enterprise license, so confirm your identity and audit requirements against the LiteLLM Enterprise documentation before you rely on them for controls.

Questions buyers ask before scoping an air-gapped gateway

What does an air-gapped LLM gateway cost to run, and what drives it?

The cost has four main parts: the people who build and operate the gateway, the compute that serves the gateway and logs, the accelerator capacity that runs the models, and any enterprise license you add. Open-source software such as vLLM carries no license fee, so its cost comes from GPU or other accelerator capacity, the surrounding infrastructure, implementation and the staff who operate it. Ollama's public pricing page states that running models on your own hardware is always unlimited, and its paid cloud tiers (Pro at $20 per month, Max at $100 per month, Team at $500 per month, Enterprise at custom pricing) do not apply to local models. Enterprise licenses matter if you need SSO beyond the free limit or audit logs. Ask any quote to split one-time build work from recurring operations and to name each component separately.

How long does an air-gapped gateway project take?

Duration depends on how much of the environment is already in place. The main drivers are whether you already run GPU hosts, how many model families and providers you need, how many teams and keys must be migrated, whether SSO and audit logging are in scope, and how fast your change process approves offline artifacts. A fixed-scope engagement agrees deliverables and timelines in writing before work starts, so ask for the schedule to be tied to named milestones such as the first model served, the first key issued and the first audit trail tested.

What must our team provide before the work starts?

Expect to supply four groups of input:

Network details: the isolated segment, host addresses, DNS, time sources and how traffic may reach the gateway.
Identity: the directory or OIDC provider for admin and user sign-in, and who approves access.
Approved artifacts: container images, Python packages, model weights, Helm charts and release bundles that your security team has scanned and accepted for transfer.
Hardware and owners: accelerator hosts, a PostgreSQL and Redis target, a secret store, and named owners for patching and on-call.

Confirm these with your own environment before scoping, because each one changes the design.

Can our own platform team build and run this without outside help?

Yes, many teams can, provided they already operate Kubernetes or a comparable container platform, PostgreSQL, a secret store and GPU hosts. The gateway itself is open source, and the vLLM and Ollama documentation covers installation and serving. The harder parts are usually offline supply chain handling, load testing, key and budget design, backup and restore of the database, and evidence collection for auditors. Ask a partner to scope only the gaps you cannot cover in-house, and decide early who carries the pager after go-live.

How do we bring model updates and LiteLLM releases into an offline network?

Each update is a transfer of signed or checksummed artifacts, not an online pull. Track the LiteLLM release list and its supported minor versions, and the vLLM and Ollama release pages for changes that affect your models. For each release, pull the image or package in a staging zone, scan it, run your regression tests against the gateway and a test model, then move the approved bundle across your transfer process. vLLM ships new minor releases about every two weeks, so set a cadence for deciding which ones to adopt, and keep the previous version available for rollback. Confirm the transfer and approval workflow with your security team before the first update.

Does the work produce audit evidence, or an audit report?

It produces evidence, not an audit report. DeployOpen prepares systems, controls and evidence for compliance assessments, such as configuration records, access reviews, key and budget settings, log retention tests and change records. Independent auditors and certification bodies issue audit reports and attestations, so an engagement never delivers either. Ask your auditor which evidence they expect for the controls in scope before the design is fixed, because log destinations, retention and access reviews are easier to build in than to retrofit.

What does an air-gapped gateway keep inside the network that a hybrid model does not?

A fully self-hosted gateway keeps the control plane, configuration, keys, budgets and logs inside your network. Portkey's hybrid deployment keeps the data plane in your VPC but has the gateway call out over HTTPS to control.portkey.ai, which is why it does not fit a disconnected network. Confirm the outbound list for each component, including telemetry and update checks, and disable any feature that calls the internet. For Ollama, the documentation states that local runs do not see your prompts or data and that cloud features can be turned off with OLLAMA_NO_CLOUD=1.

Evidence assessors ask for from a disconnected gateway

Assessors reviewing a disconnected LLM gateway want proof that no path leads out and that keys, budgets and model endpoints are controlled from inside. The cards below cover the artifacts that show this, and DeployOpen prepares them as part of an implementation, while independent auditors decide what they conclude.

Network diagrams with no egress path

Assessors start with a diagram that shows every segment the gateway, vLLM and Ollama hosts, the log store and the key database sit on, and every route between them. For this topic the diagram should show no default route to the internet and no DNS resolution for external hosts. Check that the diagram matches the live firewall and routing tables, not only the design intent. Portkey's hybrid model is ruled out here because its gateway calls out to control.portkey.ai, so the diagram must show that path absent.

Virtual key and budget configuration

Export the virtual key, budget and rate limit settings from the LiteLLM database, including per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions. Assessors will check that the master key is stored in a secret manager and that no application holds it. Keep a dated copy of the configuration so changes can be traced against it later.

Access logs and key issuance records

Keep a record of who requested each key, who approved it, when it was issued, and when it was rotated or revoked. Pair these with gateway access logs that show which key called which model and when. Confirm which log destinations run inside the network, such as Langfuse, OpenTelemetry or Prometheus, and that none forwards records outside it.

Artifact transfer and patch records

Because nothing downloads at runtime, every container image, Python package, model weight and vLLM or Ollama release must cross the boundary through a documented transfer. Record the source, checksum, approver and import date for each artifact, and keep the patch log for the gateway, the model servers and the operating system. Ask your team to confirm how new LiteLLM and vLLM releases are reviewed before import.

Endpoint allowlists for vLLM and Ollama

Document which paths on each model server are reachable from the gateway and which are blocked to users. The vLLM --api-key option does not cover /metrics, /health or /invocations, so the reverse proxy allowlist must exclude them from user traffic. Ollama binds to 127.0.0.1 by default, and its OLLAMA_HOST setting should be recorded alongside the proxy rules that expose it.

Negative test results for egress

Assessors look for proof that calls fail without an internal route. Run tests from the gateway host and from application subnets that try to reach external addresses, and record the failures. Also test that a request for a model the key is not allowed to use is rejected, and that a request over budget is stopped. Keep the test scripts and the dated results together.

How DeployOpen deploys a gateway inside an air-gapped network

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including disconnected networks. For an air-gapped LLM gateway, that means scoping the gateway, model servers, keys, budgets and logs as one system that never needs an outbound connection.

How we scope

Map every model endpoint, application and user group that will call the gateway inside the isolated network
Confirm which Portkey features are in use, such as virtual keys, fallbacks, guardrails and logs, and which must be replaced
Record the approved route for moving container images, model weights and licenses into the enclosure
Agree the written scope, deliverables and timelines before any work starts, with ongoing support available as a retainer

How we build

Deploy LiteLLM or the chosen open-source gateway in front of local vLLM or Ollama model servers with no internet egress
Configure virtual keys, per-team budgets and model restrictions backed by a PostgreSQL database inside the network
Put vLLM behind a reverse proxy that allowlists only the endpoints users need, since the API key option alone is not enough
Test routing, fallbacks and failure behavior with the network egress blocked and record the results

How we hand over

Document the architecture, network boundaries, key issuance steps and budget ownership for your operations team
Deliver runbooks for updates, model loading, backups of the database and log retention inside the enclosure
Provide evidence of configuration and access controls that your internal or independent auditors can review
Walk your operators through upgrades, since offline patching requires a scheduled process you own

How we support

Offer ongoing support, upgrades and continuous security testing as a retainer after handover
Re-test gateway access paths, allowlists and secret handling whenever the gateway or model servers change
Plan offline upgrade bundles with you so that new gateway and model server releases are reviewed before they enter the network
Confirm with us in writing which responsibilities stay with your team and which move under the retainer