vLLM load balancing with Nginx

How vLLM load balancing with Nginx works on Docker hosts

A team that has outgrown one vLLM container can add capacity by running several independent vLLM API servers on Docker hosts and placing one Nginx instance in front of them as the only address users reach. Nginx spreads requests across the replicas, stops routing to replicas that fail, and decides which API paths reach users at all. vLLM's security documentation recommends this kind of reverse proxy, one that allowlists only the endpoints meant for users, because the server's own --api-key option does not cover every route.

This setup covers four components. Balancing is an upstream group in Nginx that lists each replica, with the least-connection method chosen for long-running generation requests. Failure detection is the rule set that removes an unhealthy replica from rotation and returns it once it recovers. The shared model cache is a host directory or volume that all replicas read, so model weights are stored once per host instead of once per container. Endpoint allowlisting is a set of location rules that forward the /v1 paths and deny the operational routes such as /metrics, /health, /pause, /resume and /abort_requests to anyone outside the internal network.

Each replica keeps its own scheduler and KV cache, so the proxy balances whole requests and does not share state between servers. That shapes the first decisions: which GPUs each container receives, which port each one listens on, and how many replicas each host can hold given its memory and disk. Confirm with your vLLM version how the server handles client disconnects and streamed responses behind a proxy, and set Nginx timeouts for streaming and long generations to match the model's typical response length. Decide early whether Nginx runs on one of the GPU hosts or on a separate machine, since that choice changes the network boundary you must protect.

Kubernetes-based options are covered on other pages. Those pages address the production-stack router, KServe, and KubeRay for multi-node serving. The vLLM documentation lists several Kubernetes deployment routes, and this page does not compare them. The Docker host pattern keeps the replicas, the proxy and the cache on machines your team operates directly, which also means your team owns the failover logic that an orchestrator would otherwise provide.

Before building, confirm three things with your own environment. First, whether internal monitoring will scrape the /metrics endpoint directly from the replicas, since the proxy should block that path from users while still allowing your Prometheus server to reach it. Second, whether the deployment runs in an air-gapped network, where the --enable-offline-docs option may apply. Third, whether each model you serve includes a chat template or needs one supplied with --chat-template, because the chat completions route depends on it.

What you need before running several vLLM containers behind Nginx

Running several vLLM containers behind Nginx adds decisions that a single server hides: how much accelerator memory each replica gets, how the hosts are laid out, which vLLM release every replica runs, and where traffic terminates. Settle these items before any container starts, because the Nginx configuration and the failure behavior depend on them.

Accelerator capacity per container

Decide how many accelerators each vLLM container will own and how much model memory that leaves for weights and KV cache. Each replica loads its own copy of the model, so capacity is counted for each container separately. Confirm GPU driver and container runtime support for your accelerator against the vLLM installation docs before you size anything. Test the memory headroom with your chosen model and context length before fixing the replica count.

Docker host layout

Plan which hosts run replicas, how many containers each host carries, and whether the Nginx proxy shares a host with them or sits on its own. Give every container a fixed GPU assignment and a fixed published port so the upstream list in Nginx stays stable. Keep the host OS and container runtime versions recorded per host so a failed replica can be rebuilt to match its peers.

The vLLM release you pin

Pin one vLLM image tag for every replica so that all containers answer requests with the same behavior. vLLM ships new minor releases about every two weeks, for example v0.31.0 on October 5, 2026, so an unpinned tag can change under you between restarts. Read the release notes on the vLLM GitHub releases page for each candidate version, and check model support, API changes and security-related changes before you choose.

Network placement of replicas

Place replicas on a private subnet that only the Nginx host can reach, so that no user network can call a replica's API port directly. The vLLM security docs stress isolated networks for traffic between nodes, and the same principle keeps the proxy as the single entry point. Document the firewall rules and test them from a user network segment before go-live, and confirm that a direct call to a replica fails.

Certificates for the Nginx listener

Obtain a certificate and private key for the hostname that users will call, and decide who issues it and how it renews. Nginx terminates TLS at the listener, so the certificate chain must be complete and the private key must be readable only by the Nginx service account. Confirm your internal certificate authority or public issuer, and plan the reload step that applies a renewed certificate without interrupting active requests.

Access to the model files

Every replica needs the same model weights, tokenizer and chat template, so decide where those files live and how each container mounts them. A shared read-only volume and a synchronized local copy on each host are both workable, but each changes how a new model reaches the fleet. Confirm the chat template ships with the weights, because chat completions need one or a template supplied at startup.

Why the proxy must allowlist endpoints instead of forwarding everything

The Nginx proxy should forward only an explicit list of routes, because vLLM's own protection does not cover every endpoint the server exposes. According to the vLLM security documentation, the --api-key option protects paths under /v1, /v2, /inference and /cohere, but it does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health. A catch-all location block would pass those unprotected routes to every client along with the OpenAI-compatible API. The documentation also warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints meant for users. In this topic, the Nginx configuration is where the access boundary is enforced, so its rules need to be written from a route list, not from the assumption that the key covers everything.

The design principle is to expose only the OpenAI-compatible routes to users. These include /v1/completions, /v1/chat/completions, /v1/responses and /v1/embeddings, plus any audio transcription or translation endpoints your applications call. Control routes such as /pause, /resume and /abort_requests change the state of the engine, so they should stay on an internal network or sit behind separate credentials that only operators hold. The /metrics endpoint exposes request counts, token throughput, latency histograms and KV cache use. That data helps your platform team, but it also reveals load patterns to tenants, so it belongs on the monitoring network. Decide who needs each route before you write any rule, and record the decision next to the configuration.

In practice, the Nginx default should be deny. Any path that does not match an explicit rule should return an error without reaching a vLLM container. Use exact or anchored location matches for the allowed prefixes, and test how the rules handle trailing slashes, doubled slashes and URL-encoded variants, since a loose pattern can let a control route through. Health checks used by the load balancer should come from the internal side, and Prometheus should scrape the metrics endpoint from a monitoring address that users cannot reach. Keep in mind that the vLLM security documentation treats communication between nodes as insecure by default, so the proxy design should keep inter-node traffic off any path that users or the proxy can reach.

Confirm the full route list for your vLLM version before you write the rules. Route names and protections can change between releases, and vLLM ships new minor versions about every two weeks, so a list copied from an older deployment may be wrong. Obtain the list from the release you plan to run, check which paths the --api-key option covers in that release, and test each route from outside the network after the configuration is applied. Confirm that unlisted paths are refused and that the allowed paths work for a test client. Document the allowlist, assign an owner, and repeat this check at every vLLM upgrade and every change to the proxy configuration.

Configuring vLLM load balancing with Nginx in sequence

This sequence turns one vLLM container into several identical replicas behind Nginx, so requests spread by least connections, failed replicas drop out of rotation, and only the endpoints meant for users are reachable from outside. Each step ends with a check you can run before moving on.

Start and verify a single vLLM container
Start one vLLM container with vllm serve on the Docker host, using the model you intend to serve and a pinned model revision. Confirm it answers on its API port by sending a test request to the OpenAI-compatible completions or chat completions endpoint, and check that the response comes from the expected model. Do not add a second replica until this single instance behaves correctly, because later failures will be harder to trace. The step is done when one successful test request returns a complete response and the container logs show the model loaded without errors.
Replicate the container with identical settings
Create the additional replicas from the same image tag, the same model revision, and the same launch flags, including an identical chat template setting where the model needs one. Give each replica its own host port so Nginx can address them separately. Differences in image version, revision or template produce answers that vary by replica, which is hard to diagnose later. Send the same test request to each replica and compare the model name, response structure and template-formatted output. The step is done when every replica returns consistent results for the same prompt.
Share the model cache across replicas
Place the model weights and cache in a shared or read-only location that every replica mounts, so each container reads the same files instead of downloading its own copy. Confirm the mount path and permissions with the vLLM documentation for your release, and confirm that the replicas do not write conflicting files into the shared path. Verify that a restarted replica loads from the cache and that all replicas report the same model files. The step is done when each replica starts from the shared path and none of them attempts a fresh download.
Define the Nginx upstream group with least-connection balancing
Define an upstream block that lists every vLLM replica and set the least-connection balancing method. Confirm the directive names and the balancing behavior against the Nginx documentation for the version you run, rather than relying on examples from older guides. Point a server block at the upstream group and send a burst of test requests through the proxy. Watch the access logs or the replica metrics to confirm that requests reach all replicas. The step is done when traffic is distributed across every replica and no single replica receives all requests.
Add health-based failure handling
Configure how Nginx detects a failing replica and removes it from rotation, using the passive failure parameters for the upstream servers and, if your Nginx build includes it, the active health check features. Confirm which health check behavior your Nginx version supports and which requires a commercial edition. Stop one replica deliberately and send requests through the proxy to confirm that they continue to succeed on the remaining replicas. Then restart the stopped replica and confirm it returns to rotation. The step is done when stopping one container produces no failed user requests beyond those that fail during detection, and the restarted container receives traffic again.
Restrict user-facing locations to allowed endpoints
Because the vLLM --api-key option does not protect every endpoint, configure Nginx to proxy only the paths that users are meant to call, such as the inference endpoints under /v1, and return a refusal for everything else. Confirm the full list of endpoints the vLLM security documentation describes as unprotected by the API key, and make sure none of them is reachable from user-facing locations. Test the allowlist by sending requests to a blocked path such as the metrics or pause endpoints and confirming that Nginx returns a refusal. The step is done when every allowed path succeeds and every blocked path is refused at the proxy.
Route metrics collection to an internal scraper
Expose the Prometheus-format metrics that each vLLM replica publishes to an internal scraper only, and keep them off the user-facing proxy. Decide whether the scraper polls each replica directly or through a separate internal location, and confirm which approach your monitoring setup supports. Verify from the scraper host that every replica returns metrics, and from an outside client that the same paths are refused. Check that request counts and latency series from all replicas appear in the monitoring system. The step is done when the scraper collects from every replica and the public path to metrics returns a refusal.

Which vLLM endpoints each traffic class should reach through Nginx

This table sorts vLLM's HTTP routes into user-facing paths that Nginx should forward and operator paths that should stay internal or blocked. Read each row as the default for an allowlist that denies everything else. The --api-key option does not protect the operator routes, and endpoint names change between releases, so check every row against the vLLM version you run.

Endpoint	Purpose	Proxy treatment	Who should reach it
/v1/completions and /v1/chat/completions	Text generation for applications using the OpenAI-compatible API	Allow through Nginx on fixed locations, with the authorization header passed upstream	Application clients and end users through the gateway
/v1/responses and /v1/embeddings	Responses API calls and embedding vectors	Allow only if the deployed version serves them and an application uses them	Application clients that call these APIs
Audio transcription and translation routes	Speech-to-text and speech translation requests on the OpenAI-compatible audio API	Allow with an upload size limit suited to audio, after confirming the route names for your version	Applications that process audio, through the gateway only
/metrics	Prometheus-format request, throughput, latency and KV cache metrics	Deny on the public listener; allow only from the monitoring subnet	Prometheus scrapers and operators on the monitoring network
/health	Liveness check for the API server process	Internal only; check containers directly from the monitoring or proxy host	Load balancer checks and platform monitoring
/pause and /resume	Stop and restart request processing on a server	Block at the proxy; reach only through a management path on an isolated network	Platform operators during maintenance, never end users
/abort_requests	Cancels in-flight requests on the server	Block at the proxy; allow only from an operator network if the runbook requires it	Platform operators during incident response
/invocations	Generic inference route used by some serving integrations	Block by default; allow only for an integration that needs it, from that integration's network	Only the integration that documents its use

How least-connection balancing and failure detection behave with vLLM replicas

For vLLM traffic, the choice between round-robin and least-connection balancing matters more than it does for short web requests. A chat completion or long generation can hold an upstream connection open for a long time, so a replica that has received several long requests is busier than its turn in a rotation suggests. Nginx's least_conn directive sends each new request to the upstream with the fewest active connections, which tracks that load more closely. Whether this helps your workload is an empirical question. Run a load test that mixes short prompts with long streamed generations, record per-replica request counts and latency, and compare against plain rotation on the same hardware before you commit to a policy. Also confirm that least_conn counts only connections passing through the Nginx instance you are testing, since separate proxy instances do not share those counts.

Failure detection in open-source Nginx is passive by default. A server listed in the upstream block is marked unavailable after the number of failed attempts set by max_fails within the fail_timeout window, and it is tried again once that window passes. Confirm which active health check features your Nginx build and any add-on modules provide, because scheduled probing of a replica's health endpoint depends on them. vLLM's /health endpoint is not covered by the --api-key option, so probe it from inside your network and keep it off any path that users can reach. A replica that is restarting or loading model weights will refuse or time out connections, so it should leave rotation during that period and rejoin only after it answers again. Plan for that rejoin delay to be longer than a container restart alone would suggest.

Model loading time sets how long a replica stays out of rotation after each restart, image update or host reboot. Measure it on your hardware with your model, from container start to the first successful completion, and use that measurement when you size fail_timeout and decide how many replicas must stay up during a rolling restart. Also confirm that the proxy does not send traffic to a replica before its model is loaded, because a replica that accepts a TCP connection early may still fail the request. When you schedule maintenance, remember that each replica taken down reduces capacity by its share of the pool. In a two-replica pool, one replica down halves the capacity that is available to users, so maintenance windows should reflect that loss.

Test the slow replica case separately, because it is the failure most likely to go unnoticed. Start one replica behind a throttled or artificially delayed backend, or use a fault injection tool on its host, so that it accepts connections and returns tokens slowly without refusing anything. Then observe whether least_conn shifts new requests away from it, whether passive checks ever mark it unavailable, and how long users wait for the first token. Use those results to set proxy_connect_timeout, proxy_send_timeout and proxy_read_timeout. Confirm the values against your model's measured time to first token and time between tokens under peak load, not against generic defaults. Make sure a read timeout is not shorter than a legitimate long generation, or the proxy will cut off responses that were working correctly.

Hardening the Nginx layer for vLLM replicas

An Nginx front end is the only part of a multi-replica vLLM setup that users should reach, so most of the hardening effort goes into what it accepts, what it forwards and what it hides. The cards below cover the proxy controls and the replica-side settings that depend on them. Each one is a decision to make and a setting to verify before traffic arrives.

TLS termination at the proxy

Decide where TLS ends: at Nginx, at an upstream load balancer or inside each replica. Terminating at Nginx keeps certificate handling in one place, but the hop from Nginx to each container then needs its own answer. Confirm certificate chain, protocol versions and cipher policy with your security team, and test them against the hostname clients actually use.

Listener restricted to approved networks

Bind the Nginx listener to the interface and network the approved clients use, and add allow and deny rules for source ranges that should never reach the API. Check the rules from both inside and outside the approved network after each configuration change. A listener that answers on every interface exposes the service to networks nobody reviewed.

Replica ports off the public interface

Publish each vLLM container port only on a private Docker network or loopback address, so that only Nginx can reach it. Verify with a port scan from outside the host that no replica port answers directly. Docker's default port publishing can bypass host firewall rules, so confirm the published bindings in the container configuration rather than assuming the firewall covers them.

Blocking operator routes from user networks

vLLM serves endpoints for operators as well as users, including health, metrics, pause, resume and abort routes. Allowlist only the paths meant for user traffic, such as the OpenAI-compatible inference routes, and return a denial for everything else. Keep the metrics and health paths reachable only from monitoring and operations networks, and check the vLLM security documentation for the current list of routes.

Request size and timeout limits

Set maximum request body sizes and upstream timeouts in Nginx to match the longest prompts and streamed responses your models produce. Limits set too low cut off legitimate long generations, and limits set too high let oversized requests tie up replicas. Derive the values from tests with your own models and context lengths, and document them next to the model settings they depend on.

API keys at the proxy and in vLLM

The vLLM --api-key option protects only some endpoints, so the proxy must enforce authentication for the routes it exposes and vLLM should check the key as a second layer. Decide who issues keys, how they rotate and where they are stored, and keep keys out of logs and configuration files committed to version control. Confirm the exact behavior of the option for your vLLM version before relying on it.

Protected links between replicas on separate hosts

The vLLM security documentation states that node-to-node traffic in multi-node deployments is insecure by default, with no authorization or encryption. If replicas run on separate hosts, place their traffic on an isolated network or protect it with encryption and access controls. Read the current security documentation for the authoritative list of required settings before you expand beyond one host.

Common failures in vLLM behind Nginx and how to fix them

This table lists the failures teams most often hit when several vLLM containers sit behind Nginx, with the likely cause, a check to run and the fix for each. Read it as a triage guide: start from the symptom your clients report, then confirm the cause in the proxy and container logs before changing configuration. Most of these problems trace back to the allowlist, the health path, the model cache or the chat template requirement.

Symptom	Likely cause	Check to run	Fix
403 or 404 on legitimate routes after allowlisting	The Nginx location rules allow the /v1 prefix but not every path clients call, or a regex matches too narrowly and the request falls through to the default deny.	Run curl against each path your applications use, such as /v1/completions, /v1/chat/completions and /v1/embeddings, and read the Nginx access log for the matched location.	Add explicit location blocks for each approved OpenAI-compatible endpoint, keep /metrics, /health, /pause, /resume and /abort_requests blocked to clients, and reload Nginx after each change.
Health checks pass but clients get errors on some replicas	The upstream health path is a route that answers even when the model is not loaded or the engine has stalled, so Nginx keeps sending traffic to a degraded container.	Call the health endpoint and a short completion request directly on each container, bypassing Nginx, and compare the results.	Use the vLLM health endpoint for liveness, add a passive failure threshold such as max_fails and fail_timeout in the upstream block, and confirm the container is serving the model before it joins the pool.
Uneven load despite least-connection settings	Long generation requests hold connections open, and clients reuse keepalive connections, so the least-connection count does not reflect GPU queue depth.	Compare active connections per upstream in the Nginx status output with the running and waiting request gauges from /metrics on each container.	Set least_conn in the upstream block, tune keepalive connections and timeouts for streaming responses, and use the queue metrics to size the number of replicas per GPU host.
Requests fail during a replica restart	Nginx keeps routing to a container that is restarting or reloading weights, and in-flight streams are cut when the process stops.	Check the restart timing in the container logs against the 502 or 504 timestamps in the Nginx error log, and confirm whether the backend was marked down.	Drain the replica first by removing it from rotation or using the pause endpoint, restart one container at a time, and allow weight loading to finish before the health check returns success.
Replicas return different model outputs or fail to load the model	Each container reads a different local copy of the model files, or the shared cache volume has partial downloads or mismatched revisions.	Compare the model revision, file checksums and cache directory contents on each host, and check the startup log for the model path each container loaded.	Pin one model revision, populate a single shared cache or pre-download the files on every host with identical content, and restart replicas only after the cache matches.
Metrics missing from the Prometheus scraper	The scraper targets the Nginx port, and the proxy allowlist blocks /metrics, so Prometheus gets a 403 or an empty response.	Request /metrics directly from the scraper host, then from the container, and check the scrape error in the Prometheus target page.	Scrape each vLLM container directly on an internal network or a separate metrics listener that Nginx does not expose, and keep /metrics out of the public allowlist.
Chat completions fail while completions succeed	The served model has no chat template, so the chat completions endpoint has nothing to format messages with.	Send a request to /v1/chat/completions on a container directly and read the startup and error logs for a missing chat template message.	Confirm the model ships a chat template, or pass a template file with the chat template option at startup, and apply the same setting to every replica so behavior matches across the pool.
Authentication works on one replica only	The API key was set on some containers but not all, or the key passes through Nginx on one route and is stripped on another.	Send the same authenticated request to each container and to the Nginx port, and compare the status codes.	Set the same key on every replica through a shared secret source, and make the proxy forward the Authorization header on every approved route.

What does vLLM load balancing with Nginx cost and take to build?

Does vLLM itself add a license cost to a load-balanced setup?

No. vLLM is licensed under Apache-2.0 and has no paid edition, so the software line in this design is zero. The money goes to accelerator capacity for each replica, the Docker hosts and network that carry traffic, the implementation work that builds and tests the containers and Nginx configuration, and the people who operate the stack after go-live. Models carry their own licenses that are separate from vLLM, so confirm the terms for each model weight you plan to serve before you commit to a design.

What makes a vLLM Nginx build take more or less effort?

The main drivers are:

Number of replicas: each one needs its own container, health check, upstream entry and monitoring target.
Model size: larger models need more accelerators per replica, which shapes host layout and parallelism settings.
Access model: how clients authenticate, which keys or identity sources they use, and how the proxy enforces them.
Metrics integration: scraping the Prometheus-format metrics into your existing monitoring and alerting.
Number of teams: each team that calls the endpoint adds routes, keys, test cases and documentation.

Ask for each driver to be stated as a number or decision in the written scope, so the effort estimate can be checked against your environment.

Can our internal platform team build this without outside help?

Yes, if the team already runs Docker and Nginx and has GPU host experience. The pattern does not depend on unusual technology. The skills to check are container operations, Nginx upstream and health check configuration, GPU driver and container runtime setup, Prometheus scraping and proxy security. Gaps usually appear in two places: failure testing across replicas and the review of which endpoints the proxy exposes. Decide by checking that a named person owns each skill, and that someone has time for a full test cycle before users arrive.

What does our team need to provide before implementation starts?

Three things are needed early. Hardware access: GPU hosts with drivers and a container runtime, plus network paths between the proxy, the replicas and your monitoring. An endpoint inventory: a list of every application, client and path that must reach vLLM, including internal tools. Approval for routes: sign-off from the application and security owners on which paths the proxy will expose. This matters because vLLM's --api-key option does not cover paths such as /metrics, /health, /pause and /resume, and the vLLM security documentation recommends a proxy that allowlists only the endpoints meant for users. Also name an owner who can make decisions during the build.

What should a written quote for this work include?

A useful quote lists the scope in concrete terms: the number of replicas, the Docker hosts, the models, the endpoint allowlist, the metrics integration, the testing and the documentation handed over. It states deliverables, the client-provided items it assumes, and what is excluded. It separates one-time build work from recurring operations, such as patching containers and handling vLLM upgrades, and says how changes to scope are priced and approved. It should also state acceptance criteria, so both sides agree on what done means. Ask the vendor to explain each effort driver and how it was applied. Fixed-scope engagements are agreed in writing before work starts.

Which costs keep recurring after the Nginx setup goes live?

Recurring costs include accelerator time if you run in a cloud, host and network charges, storage for monitoring data and the staff time that keeps the stack current. Upgrades are the largest planned recurring item. vLLM ships new minor releases about every two weeks according to its GitHub release history, so decide in advance how often you will pin, test and roll out a new version. Each application change that affects routes also needs the allowlist re-tested. Ongoing support, upgrades and continuous testing are available as retainers, and you should confirm their terms against your own operating model.

Do we need Kubernetes to run several vLLM replicas?

No. This pattern runs several vLLM containers on Docker hosts behind Nginx without Kubernetes. The trade-off is operational: placement is manual, there is no automatic rescheduling after a host fails, and health checks come from Nginx rather than cluster probes. If your organization already runs Kubernetes, compare this design with the production-stack and other Kubernetes options listed in the vLLM documentation before you choose. Confirm your expected host count, failover expectations and patching process, because those decide whether the simpler design is enough for your environment.

How do we check that the proxy blocks endpoints we did not intend to expose?

Test from outside the trusted network. Send requests to each path that is not on the allowlist, including /metrics, /health, /pause, /resume and /abort_requests, and confirm the proxy denies them. Then send requests to the /v1 paths with and without a valid key, and confirm the expected responses. Repeat these tests after every configuration change and keep the results as evidence for your security review. Testing shows what the proxy does on the day it runs, so the test list should be updated whenever the endpoint inventory or the vLLM version changes.

How DeployOpen scopes and delivers vLLM load balancing with Nginx

DeployOpen approaches vLLM load balancing with Nginx as a fixed-scope engagement that starts with written scope covering the containers, hosts, proxy configuration and endpoint allowlist. Engineers build and test the Nginx upstream pool against your own GPU hosts, then document the operating procedures your team takes over. Ongoing support is available as a retainer.

How we scope

The written scope lists each vLLM container, host, GPU assignment and the Nginx upstream pool.
Scope confirms which /v1 routes users reach and which operational paths stay internal to the network.
Scope records the model cache location, model chat templates and the accelerator driver baseline each host needs.

How we build

Engineers configure Nginx least-connection balancing across the vLLM containers and verify traffic spreads under load.
Health checks and failure timeouts remove failed containers from rotation, tested by stopping containers on purpose.
Proxy allowlist exposes only the /v1 inference routes, while /metrics and /health stay on the internal network.

How we hand over

Handover includes the Nginx configuration in version control, with each change to upstreams documented.
Runbooks cover adding or draining a vLLM container, rotating API keys and reading Prometheus metrics from /metrics.
Your team receives test records for failover, allowlist behavior and load distribution, alongside the configuration.

How we support

Ongoing support is available as a retainer covering upgrades to vLLM and Nginx configuration changes.
Upgrades are tested on a staging pool before a container in production is replaced, one container at a time.
Changes to the allowlist and upstream pool are reviewed and tested before they go live.