Open WebUI with vLLM

Connect Open WebUI directly to vLLM when one vLLM server serves one team and nobody needs per-user budgets. In that setup, Open WebUI talks to the OpenAI-compatible endpoints that vLLM exposes, and there is one component to deploy, patch and monitor. Add LiteLLM only when a specific requirement appears, because it adds a database, a cache and a second service to operate. Before deciding, write down which of the triggers below apply today and which are likely within the next year.

LiteLLM earns its place in four situations. The first is several vLLM backends, or a mix of vLLM and hosted providers, that you want to expose behind one interface. The second is spending control: per-key, per-user or per-team budgets, TPM and RPM limits, temporary budget increases, and keys restricted to specific models. The third is routing, including retries, cooldowns after repeated failures and priority ordering for failover. The fourth is audit and observability across all model traffic. If none of these apply, the gateway is overhead. Confirm the exact control you need, such as whether a budget must be enforced per user or per team, because that decides the key design.

The gateway adds recurring infrastructure. LiteLLM's production guidance names PostgreSQL as the primary supported database, and virtual keys require it. Once more than one proxy instance runs, Redis 7.0 or newer is required for shared rate limits, router state and cache. On Kubernetes, run one Uvicorn worker per pod and scale horizontally by adding pods. LiteLLM suggests 1 vCPU and 4 GiB of memory per worker as a starting point, so size each pod from your expected concurrency and then measure. Your cloud or data center bills for these resources, while setup and operations are paid either to your own team or to a partner under a written scope. In a quote, ask for each database, cache and pod to be listed separately, with backup and upgrade work named.

Licensing and SSO also affect the decision. LiteLLM is MIT-licensed outside its enterprise/ directory, which is covered by a separate license file, so confirm which files your build and support terms touch. For the admin UI, SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. Check that threshold against your user count and your identity provider before you commit. The current release listed on LiteLLM's GitHub releases page is v1.105.1, published October 14, 2026, so confirm which version your deployment will pin and how it will be upgraded.

Whichever path you choose, keep vLLM off the user network. The vLLM --api-key option does not protect every endpoint, including /metrics and /health, and the vLLM security documentation recommends a reverse proxy that allowlists only the endpoints users need. With LiteLLM in place, the gateway becomes the single route users reach, so vLLM should accept traffic only from the gateway. Test this by calling vLLM directly from a user network segment and confirming that the request fails. Record the result in your change log so an auditor or security engineer can see the control.

Requests fail during model loading because a vLLM server takes time to move model weights onto its accelerators after it starts or restarts, and it does not accept inference traffic until that work finishes. Open WebUI does not know the difference. It sees a configured OpenAI-compatible connection, tries to send a chat request, and waits. If the server is still loading, the request stalls or is refused, and the user sees a timeout or an error in the chat. The failure is a timing problem, not a fault in the model, so the first step is to confirm which state the server is in before changing anything else.

From the Open WebUI side, a loading server usually looks like one of three things: a connection error when the port is not yet listening, a read timeout when the request is accepted but no tokens arrive in time, or a model that appears in the model list while chats to it hang. The model list can be misleading because Open WebUI populates it from the connection's model endpoint, so a model can show as available before it can generate. Check the vLLM startup log for lines reporting weight loading and the point where the API server begins serving, then query the /health endpoint on the same host and port that Open WebUI uses. If /health does not return success, the server is still starting, and no amount of retrying from Open WebUI will fix that. If /health succeeds and chats still time out, look at the request path, GPU memory pressure and queueing instead.

Fix the problem in this order. First, wait for readiness before adding a model to users' lists. Your startup script or orchestrator should poll /health and only then register the model connection or enable the model for groups in Open WebUI. This removes most of the errors because users never see a model that cannot answer. Second, raise client timeouts where the connection layer allows it. Check whether the Open WebUI connection settings for your version expose a request timeout, and confirm the same for any reverse proxy in front of both services, since a proxy with a shorter read timeout will cut requests before vLLM responds. Treat the longer timeout as a safety margin for cold starts, not a substitute for readiness checks.

Third, keep models warm on long-lived servers. If your deployment allows it, avoid restarting vLLM for routine updates to Open WebUI, and plan maintenance windows so that a freshly restarted model is loaded and checked before users return. Measure load time on your own hardware and with your own model, including the weight size, quantization, tensor parallel layout and the time needed for the first real request, because no load-time figures apply to your setup. Record those measurements so the timeout and warm-up decisions rest on your data. Fourth, when a second backend exists, put LiteLLM between Open WebUI and the model servers. Its router supports retries and cooldowns after repeated failures, and priority ordering can send traffic to a healthy backend while one restarts. Confirm the retry and cooldown behavior for your chosen LiteLLM release and routing strategy in a test before relying on it for users.

Run this verification plan before any user receives access. Its purpose is to show that the model server answers the way Open WebUI expects, that each group reaches only the models it should, and that nothing outside your trusted network can reach the server's internal endpoints. Point every test at the same base URL that Open WebUI uses, whether that is vLLM directly or the LiteLLM proxy in front of it, and use the same API key scope. Results from a path users never take prove little. Record each run with its date, the vLLM version, the model name, the serving flags and the Open WebUI version, so that a later failure can be traced to a specific change.

Start with functional tests. Send a non-streaming chat completion to /v1/chat/completions with a short prompt and confirm a well-formed response that includes token usage. Repeat the request with streaming enabled and confirm that tokens arrive incrementally, because a reverse proxy that buffers the stream makes Open WebUI appear to hang. If you enable embeddings for retrieval, call /v1/embeddings with the embedding model and check the vector length against what your vector database expects. Then send a request to every model that each user group is allowed to use, since group permissions fail per model. Sign in with a restricted test account and confirm that the model list shows only the models its group was granted.

Next, run a load-up test that begins with a cold server. Stop the vLLM process, start it again, and send traffic immediately so you observe what users experience while weights load. Record the time to first successful response and whether any Open WebUI or proxy timeout fires before it. Then raise concurrency in steps, holding each step long enough to reach a stable reading, and watch for queueing, rising time to first token and failed requests. Use the Prometheus metrics that vLLM exposes at /metrics to confirm what the clients report. Request counts, token throughput, latency histograms and KV cache usage show whether a slowdown comes from a saturated cache or from requests waiting for a free slot. Set the ramp steps, test duration and pass thresholds in the scope document.

If LiteLLM routes requests across two vLLM backends, test failover deliberately. With traffic running, stop one backend and confirm that requests continue to succeed through the second and that the failed deployment is taken out of rotation. The LiteLLM routing documentation describes retries, cooldowns after repeated failures and priority ordering for failover, so confirm which of these your configuration uses and check what you observe against that configuration. Count the user requests that fail during the switch and note whether Open WebUI shows an error or retries on its own. Restore the backend and confirm that it returns to rotation and that its metrics resume. Also test a long conversation that spans the failover, because session affinity, if configured, can keep a user pinned to a backend that is down.

Finish with a security test from outside the trusted network. From a host with no route into the internal network, try to reach the vLLM port directly and the user-facing address through any proxy. The vLLM security documentation states that --api-key protects the /v1, /v2, /inference and /cohere paths but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. It recommends a reverse proxy that allowlists only the endpoints meant for users. Confirm that the internal endpoints return no response from outside, and that the allowed /v1 paths reject requests without the key. Repeat this check after every upgrade or proxy change, because a new route or an edited rule can reopen an endpoint. Write the acceptance criteria and test volumes into the scope document before testing begins, so that the result is a recorded decision.