vLLM with Open WebUI

The request path starts when a staff member signs in to Open WebUI, which checks the session before any prompt reaches a model. When the user sends a message, Open WebUI builds a chat completion request and sends it to the vLLM server's /v1/chat/completions endpoint, using the base URL and API key stored in its connection settings. The vLLM server runs the model and returns output, which Open WebUI passes to the browser as it arrives. The browser does not call vLLM directly, so the vLLM address and key stay on the server side. Confirm in your own network that user machines cannot reach the vLLM port directly.

Open WebUI needs only the routes it uses. The vLLM server also exposes /v1/completions, /v1/responses, /v1/embeddings and audio routes, and the chat interface depends on /v1/chat/completions for conversations. Other routes matter only when something calls them. If you enable retrieval features that need embeddings, confirm whether vLLM serves the embedding model or another provider does, and allowlist /v1/embeddings only if vLLM is the source. Enforce the allowlist at a reverse proxy in front of vLLM, because the vLLM security documentation warns against relying on --api-key alone and says the key does not cover endpoints such as /metrics or /health. Recheck the allowlist whenever you change the Open WebUI or vLLM version.

vLLM does not know individual staff users. The API key in the Open WebUI connection is a shared credential for the whole deployment, so vLLM sees one client no matter who typed the prompt. It cannot enforce per-person access, attribute usage to a person, or revoke one user's access. User identity, roles, group permissions and any per-user API keys live in Open WebUI, which documents SSO through OIDC and LDAP, SCIM 2.0 provisioning and role-based access control with groups. Decide early where per-user usage records will come from, because vLLM's Prometheus metrics count requests and tokens without naming the person who made them.

Before go-live, run a short set of checks. Sign in with an account that has no model access and confirm Open WebUI refuses the request. Confirm the vLLM key comes from a managed secret store and is rotated on a schedule you have written down, not typed into a shared admin page. Send a direct request to the vLLM port from a user network and confirm it fails. Confirm the served model has a chat template, or that one is supplied with the --chat-template option, because the chat completions API requires it. Finally, record the Open WebUI and vLLM versions you tested together, since both projects publish releases on short cycles and each upgrade can change this path.

Access to each model is decided in Open WebUI, not in vLLM. Open WebUI documents single sign-on through OIDC and LDAP, user and group provisioning through SCIM 2.0, and role-based access control built on groups with per-resource permissions. Those settings determine which users can open which models. Before go-live, map your identity provider groups to Open WebUI groups and confirm, in the Open WebUI release you deploy, whether group membership arrives through SCIM, LDAP group lookup or OIDC claims. Test with one account from each group and confirm that a user outside every model group sees no models.

Open WebUI API keys and authenticator-app MFA are separate controls. An Open WebUI API key acts as the user who created it and inherits that user's model permissions, so it is a user credential. Decide who may create API keys, and whether most staff should have that ability at all. Authenticator-app MFA protects interactive sign-in to Open WebUI only. It does not apply to API calls made with a key, and it does not protect the vLLM endpoint. Confirm the exact setting names and the default state of each control in your installed version before you write the access policy.

The vLLM key is a service credential. Open WebUI holds it in its connection settings and sends it with every request to the OpenAI-compatible endpoint, so it identifies the Open WebUI service and no person. Store it in your secrets system, keep it out of browser-visible configuration, and plan rotation as a change with a maintenance window in Open WebUI and vLLM. The vLLM --api-key option covers only the /v1, /v2, /inference and /cohere paths. The vLLM security documentation warns against relying on it alone and recommends a reverse proxy that allowlists only the endpoints meant for users. Confirm that /metrics, /health, /pause and /abort_requests are unreachable from user networks.

A single shared vLLM key means the model server cannot attribute a request to a person, team or budget. Usage records at vLLM show the key holder, which is Open WebUI. Ask Open WebUI which per-user usage data it records and how long it keeps it, and decide whether that is enough for your audit and cost reviews. If you need per-team keys, spending limits or rate limits enforced before requests reach vLLM, you need a gateway layer between Open WebUI and vLLM. The sibling page on placing a LiteLLM gateway in front of vLLM covers that design. Settle this question before launch, because moving to a gateway later changes the endpoint and key that Open WebUI uses.

Interactive chat is judged on two things: time to first token, which is how long a user waits before text starts to appear, and sustained token throughput, which is how quickly the rest of the answer streams once it begins. Both are shaped by the accelerator capacity behind vLLM. Four inputs drive that need. Concurrency sets how many requests compete for the same GPUs at once. Prompt length increases the prefill work done before the first token. Response length extends decode time and keeps KV cache memory allocated for longer. Model size sets the memory needed for weights and the compute spent on every generated token. Open WebUI adds to prompt length because chat clients commonly resend earlier turns of a conversation, so long threads grow the input on every request. Confirm how your Open WebUI version builds requests before you assume prompt sizes.

Measure before you buy capacity. vLLM exposes Prometheus-format metrics at /metrics on its API server. Scrape that endpoint with your Prometheus instance and watch request counts, token throughput, latency histograms and KV cache use during a pilot with a representative group of staff from several departments, using the prompts and hours they actually work. Pay attention to peaks rather than averages, and note when KV cache use approaches its limit, because queuing and slow first tokens usually appear at that point. vLLM lets you configure histogram buckets, so set them around the latency targets your users care about. The output of this pilot is the traffic profile you will hand to any capacity estimate.

Tensor and pipeline parallelism change the hardware layout. Tensor parallelism splits a model across GPUs inside one node, which keeps the work on one machine. Pipeline parallelism spreads model layers across nodes, which adds dependence on the network between them. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default and must sit on an isolated network, so a multi-node layout also needs network design and access controls before it is run for staff. Ask your team whether one node with several accelerators can meet the measured concurrency. If it cannot, decide whether the added network and operational work of a second node is justified. Confirm the interconnect and GPU requirements for your chosen layout with the vLLM parallelism documentation and your accelerator vendor.

Open WebUI adds components to size once more than one replica is required. The scaling guide requires PostgreSQL instead of SQLite for multi-instance deployments, because SQLite does not handle concurrent writes from several processes well. Multiple replicas also need Redis as a shared store for sessions, websocket connections and application state, and shared file storage so uploads and knowledge stay reachable from every replica. Each of these has its own capacity needs for database connections, Redis memory and storage throughput, and each must be sized alongside the vLLM accelerators. When you request a capacity estimate, give the provider your measured profile: peak concurrent users, the spread of prompt and response lengths, peak hours, the models you plan to serve, and your latency targets. Ask for the estimate to show accelerator, database, cache and storage components separately, with the assumptions behind each.