vLLM behind a LiteLLM gateway

The vLLM --api-key option protects only part of the server's surface. According to the vLLM security docs, it covers endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. Some of the uncovered paths are operational controls. Pause and resume change how the engine serves traffic, and abort_requests cancels work in flight. If any client can reach vLLM directly, it can call these paths without the key, so the key cannot be the only control on the server.

The design rule that follows is that only LiteLLM should open connections to vLLM. Place vLLM on a private network segment or pod network with no ingress from user or application subnets, so the LiteLLM proxy is the only caller. LiteLLM then exposes only the paths applications need, such as /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/responses, where the deployed vLLM version supports them. Paths such as /metrics can be reached from a monitoring network that the team defines separately, not from the user-facing side of the gateway.

The vLLM docs recommend a reverse proxy that allowlists only the endpoints meant for users. Decide where that allowlist lives. Placing it at the LiteLLM layer controls what users can call, but it does not stop a direct call that bypasses LiteLLM. Placing a path allowlist in front of vLLM, combined with network policy, makes blocked paths fail even when a request arrives from an unexpected source. Assign an owner for this configuration and keep it in version control alongside the LiteLLM settings.

Confirm the exact path list for the vLLM version you run before writing the allowlist. Releases ship about every two weeks, and the set of routes can change between versions, so treat the allowlist as a versioned artifact. Build a test that sends a request to each blocked path from an application network and expects a denial, and a request to each allowed path with a LiteLLM-issued key that expects success. Repeat that test in every upgrade so a new route added upstream does not become reachable by accident.

The LiteLLM Router decides which vLLM deployment serves each request, and it is the component that makes retries, cooldowns and failover possible. It offers five routing strategies: simple-shuffle, usage-based, latency-based, least-busy and cost-based. Simple-shuffle is the recommended default, and it is the right starting point for most self-hosted pools of identical vLLM replicas. Usage-based, latency-based and least-busy strategies try to send traffic toward the replica with the most headroom, so they are worth testing once you have load data. Cost-based routing assumes each deployment carries a price per token, which a self-hosted replica does not have in the same way, so confirm what it optimizes in your LiteLLM version before choosing it. Make the choice with a load test against your own models and prompt lengths, not from the strategy name alone.

Retries and cooldowns determine how quickly a failing replica stops receiving traffic. When a call to one deployment fails, the Router can retry the request on another deployment in the same model group. After repeated failures, the Router places the deployment in cooldown and sends it no traffic until the cooldown ends. The supplied documentation confirms the mechanism but not the exact thresholds or cooldown length, so ask the LiteLLM project or read the settings reference for your release, then record the values you use. Test the behavior directly: stop one vLLM container under steady load and check whether clients see errors, how many retries occur and how long the replica stays out of rotation. Retries also add load to the surviving replicas, so confirm they have capacity for the retried traffic.

Priority ordering defines the failover sequence. You group deployments and assign priorities so the primary vLLM replicas serve traffic first and secondary deployments serve only when the primary group cannot. Fallback to a different model changes answer quality and behavior, so write the fallback map explicitly, get sign-off from application owners and log which deployment answered each request. Session affinity pins a conversation to one deployment. It can help multi-turn workloads on the same replica, but it works against even distribution and it can send a whole conversation to a degraded replica. Test it with recorded multi-turn transcripts at realistic concurrency, measuring latency, error rates and per-replica load with affinity on and off before deciding.

The gateway cannot fix a vLLM replica that is overloaded but still healthy. Such a replica still answers health checks and accepts requests, so the Router keeps sending it traffic and latency rises for every user routed there. Latency-based and least-busy strategies reduce the problem but do not remove it. Watch the metrics that vLLM exposes at its metrics endpoint, including request counts, latency histograms and KV cache use, and alert on them directly. Capacity planning for each replica is a separate task. Memory, KV cache size, concurrency limits and maximum sequence length are sized inside each vLLM deployment, and adding a gateway does not change them. If several proxy instances share router state, LiteLLM's production guide calls for Redis 7.0 or newer, so include that dependency in the design and in the failure tests.

The test plan should prove one thing: users and applications can reach vLLM only through the LiteLLM gateway, and every request that gets through carries a valid, budgeted key. Write each case as a script with an expected result, run it from the network positions that users, applications and the monitoring system actually occupy, and keep the output with the change record. A single failed case means the gateway has failed the review, even if every other case passes. The cases below cover the controls the gateway adds and the endpoints that vLLM's own --api-key option leaves unprotected. Before writing the scripts, have your team confirm the exact paths your installed vLLM version serves, because the vLLM security documentation is the baseline for that list.

Start with the key lifecycle. A request with a valid virtual key should succeed through the proxy, using the model route that the gateway maps to your vLLM deployment. Then present a key whose expiry date has passed and confirm the proxy refuses it. Repeat the check with a key whose budget is exhausted, using a small test budget so the refusal happens in a controlled run. For rate limits, set low RPM and TPM values on a dedicated test key and send a burst above each limit. Confirm that the proxy returns a rate-limit error and that the counters recover as the documentation describes. Also confirm that a key restricted to one model cannot call another model. Record the exact error codes, because client retry logic and cost reports depend on them.

Next, test failover and direct access. Run a steady request loop through the gateway, stop one vLLM replica, and watch what the client receives. The expected result is that LiteLLM retries or falls back to another deployment, the client sees no error, and the proxy logs show the stopped deployment entering cooldown. Confirm the priority order you configured, since failover only behaves as planned when the router knows which deployment is the backup. Then, from a host outside the gateway subnet, call the vLLM port directly on the inference paths and on the paths that the --api-key option does not cover, including /invocations, /pause, /resume, /abort_requests, /health and /metrics. Each call should fail at the network layer. Verify the block with a port check from that host, not only with an HTTP client.

Finish with metrics and upgrades. vLLM serves Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. Only the monitoring system should scrape them. Confirm that the scraper's source address returns the metrics, and that application subnets, developer workstations and gateway clients receive a refusal. Also confirm that the metric names your dashboards and alerts depend on are still present, because a silent rename breaks alerting without producing any error. Keep the whole suite in version control and rerun it after every vLLM upgrade, including each new minor release, and after every LiteLLM upgrade, since router defaults, key handling and proxy behavior can change between versions. Store each run's results with the change ticket.