How vLLM on KServe works and when to use it
KServe deploys vLLM as a model serving workload on Kubernetes and exposes its OpenAI-compatible API, so a team that already runs KServe for other models can add language models to the same serving control plane. The vLLM documentation lists KServe among its supported Kubernetes deployment options, alongside native manifests, Helm, the production-stack, KubeRay, LeaderWorkerSet and llm-d. This page covers two paths within that option. The first uses the Hugging Face serving runtime, where KServe starts vLLM for a model it loads. The second uses an LLMInferenceService resource with llm-d, which is aimed at distributed serving across replicas. Both expose the same OpenAI-compatible surface to applications, but they differ in how much routing and scheduling your team operates itself.
Choose the runtime path when each KServe service hosts one model on a predictable set of accelerators and your team wants the smallest change from how it already deploys other models. Choose the llm-d path when you need several replicas of the same model behind shared routing, or when the model needs capacity that spans nodes. Before you commit to either, confirm in the KServe documentation which release supports each path, and confirm that your model, accelerator and node layout are supported. Plan a review cadence for the vLLM image as well. vLLM ships new minor releases about every two weeks, so pinning the runtime image to a tested version and promoting upgrades through a staging cluster keeps model behavior from changing without notice.
Several vLLM behaviors need explicit handling in the KServe design. A model must provide a chat template, or you must supply one with --chat-template, before the chat completions endpoint works, so test each model in its final configuration. The --api-key option does not protect /metrics, /health, /invocations, /pause, /resume or /abort_requests, and the vLLM security documentation recommends a reverse proxy that allowlists only the endpoints users need. In KServe, decide whether the ingress or a gateway in front of the service performs that allowlisting. Metrics are exposed in Prometheus format at /metrics, so decide who scrapes them and keep that path unreachable from outside the cluster. For multi-node runs, vLLM's documentation states that inter-node communication is insecure by default, so those nodes need an isolated network.
The cluster underneath matters as much as the serving resource. Kubernetes stores Secrets unencrypted in etcd by default, and gated Hugging Face tokens or registry credentials used by KServe often end up in Secrets. Confirm that encryption at rest is enabled, restrict list and watch access with least-privilege RBAC, or move those credentials to an external store through the Secrets Store CSI Driver. People who deploy and manage models should authenticate through the cluster's identity integration, such as OpenID Connect, rather than through shared tokens. Upgrades need sequencing: kube-apiserver moves one minor version at a time before the controller manager, scheduler and kubelets, and kubelets may run up to three minor versions behind. Schedule KServe and vLLM upgrades around those windows, and check the Kubernetes release page for the currently supported minor versions before choosing a baseline.
What a KServe and vLLM cluster needs before installation
A KServe and vLLM cluster depends on a short list of prerequisites that are easy to underestimate. Each item sets a limit on what you can install, how the serving stack behaves under load, and what your security team will ask about later. Settle them before you order or schedule accelerator hardware, because changing the control plane, network or storage design after installation is costly.
Supported Kubernetes minor version
As of September 2026, the Kubernetes project maintains minor releases 1.37, 1.36 and 1.35. Choose one of them for a new cluster, and confirm it against the KServe compatibility matrix and the vLLM image you plan to run. End of life dates are October 28, 2027 for 1.37, June 28, 2027 for 1.36 and February 28, 2027 for 1.35, so your go-live date sets the upgrade window. The version skew policy requires kube-apiserver to move first, one minor version at a time.
Control plane and etcd
Run at least three control plane nodes for production, with a TCP load balancer in front of kube-apiserver on port 6443. The kubeadm high availability guide describes a stacked topology, with etcd on the control plane nodes, and an external etcd topology, so pick one before installation. etcd needs an odd number of members to keep quorum, and the Kubernetes documentation recommends five members for production. Cluster state, including KServe objects, lives in etcd, so schedule etcdctl snapshot save backups and test restores with etcdutl before go-live.
Accelerator nodes and GPU drivers
Accelerator nodes need the driver stack that your vLLM build targets: NVIDIA CUDA, AMD ROCm or Intel XPU. Match the container image and driver versions together, and confirm that your device plugin or operator exposes the GPUs to pods. Taint accelerator node pools so only inference workloads schedule there, and record driver versions per node pool so upgrades run in a known order. Other accelerators depend on third-party hardware plugins, which need their own review.
KServe version and serving path
Choose the KServe release from the KServe compatibility matrix against your Kubernetes minor version. Then decide which serving path you need: the Hugging Face serving runtime or the LLMInferenceService with llm-d. The two paths rely on different components, so the choice changes which controllers, gateway components and runtime images you install. Record both versions in your change plan, because KServe and vLLM each release on their own schedule and an upgrade can touch both.
Network, DNS and certificates
Set the network design before installation. The vLLM security documentation states that node-to-node traffic in multi-node deployments is insecure by default, so place those nodes on an isolated network. Reserve a DNS name for the inference endpoint and decide which certificate authority issues its TLS certificate and who renews it. Allowlist only user-facing routes at the ingress, because the --api-key option does not protect /metrics, /health or /invocations.
Model weight storage
Model weights are large files that each replica reads at startup, so choose the storage approach before installation. Decide whether weights come from a shared read-only volume, a persistent volume per model or a copy on each node, and confirm the choice against the vLLM and KServe documentation for your model format. Air-gapped clusters need weights moved in through an approved transfer path. Chat models also need a chat template, either bundled with the model or supplied through the chat template option.
Choosing between the Hugging Face runtime and LLMInferenceService with llm-d
Choose the Hugging Face serving runtime when you want KServe to manage one vLLM model service through the InferenceService objects your platform team already operates. Choose LLMInferenceService with llm-d when you need LLM-specific routing and scaling behavior that a generic model service does not expose. Start from the requirements, not the object. List the routing, scaling, metrics and authentication behavior your applications depend on, then check which path provides each one in the versions you plan to run.
The choice changes the custom resources your team writes and reviews. InferenceService manifests, runtime definitions and their fields are what your GitOps repositories, Helm values and admission policies already check. LLMInferenceService is a different resource with its own spec, and the llm-d components it brings in are additional workloads your cluster must run, secure and monitor. Review every template, generator and policy rule that touches model serving, because a change of resource type affects code review and access control as well as the cluster itself.
The upgrade path also differs. Kubernetes version skew rules apply to the cluster either way: kube-apiserver is upgraded first, one minor version at a time, and kubelets may trail it by up to three minor versions. On top of that, KServe, the serving runtime image, vLLM and llm-d each release on their own schedule. vLLM ships new minor releases about every two weeks, so pin versions explicitly and test each combination of KServe, runtime, vLLM and llm-d before you promote it. The llm-d path adds one more component to that matrix.
The skills your platform team needs also shift. The runtime path mainly requires KServe and serving runtime knowledge. The llm-d path adds gateway routing, scheduling and inference-specific metrics, so your team needs time to learn them and a named owner for on-call response. Ask for both designs to be demonstrated on a cluster that resembles yours before you commit, and record who will maintain each component after handover.
Do not assume feature parity between the two paths. Confirm current support for autoscaling, multi-node serving, authentication and Prometheus metrics against the KServe and llm-d documentation for your exact versions, and write the answers into the design record. Also check endpoint exposure on whichever path you choose. The vLLM --api-key option does not protect /metrics, /health or /invocations, so plan an allowlisting proxy or gateway rule for those paths, and verify it with a test from outside the cluster network.
Configuring the vLLM serving runtime in KServe, step by step
This sequence takes an approved model from stored weights to a vLLM endpoint that applications can call through a gateway. Each step ends with a check, so a fault is caught before the next layer depends on it. It assumes KServe is the serving control plane your team already runs or is adding for this model.
Confirm cluster and KServe versions
Your platform team compares the Kubernetes minor version with the releases the Kubernetes project still maintains, which were 1.35, 1.36 and 1.37 as of September 2026, and reads the version skew policy before changing anything. Confirm that the KServe release you plan to install supports that Kubernetes version, and that GPU nodes expose accelerator resources to the scheduler through the device plugin or operator your platform uses. Record the cluster version, KServe release and driver versions in the change record. This step is complete when every GPU node reports Ready and its allocatable resources include the GPU resource name your vLLM image expects.
Install KServe with the chosen serving path
Choose between the Hugging Face serving runtime, which uses an InferenceService, and the LLMInferenceService path with llm-d. Decide based on whether you need llm-d's routing and scheduling features for this model. Confirm the resource kinds, field names and install method in the KServe documentation for your release, since they change between versions. Install KServe with the method your team uses for cluster add-ons, and keep all manifests in version control. The step is complete when the controller pods are running in the KServe namespace, the serving resource definitions are registered, and the API server lists the resource kind you selected.
Stage model weights where the cluster can read them
Place the model weights in object storage, a persistent volume or a pre-populated node cache, depending on what your cluster can reach and how your storage team governs it. Confirm the license terms of the weights before copying them, and copy the configuration, tokenizer and chat template files with them. In air-gapped networks, mirror the weights and container images into internal storage and registries, and check the offline documentation option that vLLM provides for those environments. The step is complete when a temporary pod mounts the location, lists the expected files, and file sizes and checksums match the inventory from the source.
Define the serving resource and vLLM arguments
Write the InferenceService or LLMInferenceService manifest with the model location, served model name, maximum context length, tensor parallel size that matches the GPUs on each node, and the GPU request. Add the chat template option that points to the template file when the model's package does not include one, because the chat completions endpoint needs a template. Reference the API key from a Kubernetes Secret and keep the key value out of the manifest. Kubernetes stores Secrets unencrypted in etcd by default, so confirm that encryption at rest is configured. The step is complete when the resource reports a Ready condition and the pod is scheduled on a GPU node.
Confirm model load in pod logs
Read the logs of the serving pod after startup and confirm that the weight load finishes, the KV cache allocates, and the API server reports that it is listening. Typical failures at this point are out-of-memory errors during load, a missing chat template, or a tensor parallel size that does not match the GPUs present. Confirm that the Prometheus metrics endpoint is reachable from your monitoring namespace, since vLLM publishes request, throughput, latency and KV cache metrics there. The step is complete when the logs show a finished startup, the pod passes its readiness check, and metrics are being scraped.
Test a chat completion from inside the cluster
From a test pod in the same namespace, send a request to the chat completions path on the internal service address, with the bearer token and a short user message. Repeat the request without the token to confirm that the server refuses it. Because vLLM exposes OpenAI-compatible endpoints, standard client libraries can be pointed at the internal address for this test. The step is complete when the authenticated request returns generated text in the expected response structure and the unauthenticated request is refused.
Expose only allowlisted endpoints at the gateway
The vLLM API key does not protect /metrics, /health, /pause, /resume, /abort_requests or /invocations, so user traffic should reach the service only through the ingress or gateway. Configure an allowlist that passes only the endpoints users need, such as completions, chat completions, responses or embeddings, and keep the metrics and control endpoints on the internal network. Confirm that your gateway product supports path-level allowlisting before you rely on it. The step is complete when allowed paths return responses from outside the cluster, and denied paths are refused at the gateway without reaching the pod.
How API keys, models and metrics map between KServe and vLLM
This table maps each setting the team configures in vLLM to the matching KServe object or Kubernetes identity, and shows which endpoints the vLLM API key leaves open. Read each row as a pair: the vLLM column sets what the engine expects, and the KServe column shows where the team sets it and what to check. The main takeaway is that the vLLM API key covers only part of the surface, so the gateway must enforce access to the remaining endpoints.
| Element | vLLM behavior | KServe side and what to confirm |
|---|---|---|
| API key flag | The --api-key option protects the /v1, /v2, /inference and /cohere endpoints. | Store the key in a Kubernetes Secret and inject it into the serving container, not into the manifest text. Secrets are stored unencrypted in etcd by default, so confirm encryption at rest and least-privilege RBAC for reading them. |
| Model name on the OpenAI-compatible route | Clients send a model identifier in each request, and it must match the identifier vLLM was started with. | Confirm the path and model field your KServe release exposes, then test one request with the exact identifier clients will send. A mismatch returns errors that look like routing failures. |
| Chat template argument | A model must provide a chat template, or be given one with --chat-template, before the chat completions API works. | If the model repository lacks a template, supply one through the runtime arguments in the InferenceService or LLMInferenceService spec. Verify /v1/chat/completions returns a response before exposing the route. |
| Metrics endpoint | vLLM serves Prometheus-format metrics at /metrics on the API server, covering request counts, token throughput, latency and KV cache use. | Add the scrape job through the monitor resource your Prometheus setup uses, such as a ServiceMonitor or PodMonitor if you run the Prometheus Operator. Keep /metrics off the public route. |
| Kubernetes identities for callers | vLLM does not check Kubernetes identity itself. Access control comes from the API key and the layer in front of the server. | Kubernetes authenticates users with X.509 client certificates, OIDC or JWT bearer tokens, webhook token review or an authenticating proxy. Workloads use service accounts. Map which identity may call inference, which may scrape metrics and which may operate the server. |
| Endpoints the API key does not protect | /invocations, /pause, /resume, /abort_requests, /metrics and /health are outside the --api-key check. | The gateway must enforce access for these paths. Allow only the user-facing inference routes, block /pause, /resume and /abort_requests from users, and limit /metrics to the monitoring identity. Confirm which of these paths your application calls. |
| Health probes | The /health endpoint answers without the API key. | KServe probes must reach /health inside the cluster. Keep that path off external routes so probe access does not become a public path. |
Securing the vLLM endpoint behind KServe
Serving vLLM through KServe adds several layers that each need their own controls: the model server, the network around it, the Kubernetes API that stores its configuration, and the namespace that runs it. The cards below cover the settings that most often leave a gap, starting with the endpoint exposure that vLLM's own API key does not close.
Allowlist user-facing paths at the proxy
The vLLM --api-key option does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health. vLLM's security documentation recommends a reverse proxy or gateway that exposes only the endpoints meant for users, such as /v1 routes. Confirm in your KServe ingress configuration that management and metrics paths are unreachable from user networks.
Isolate node-to-node traffic in multi-node runs
vLLM's security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. If a model spans nodes through pipeline parallelism, place those pods on a network that only the serving nodes can reach. Check that no user-facing service or ingress route points at the inter-node ports.
Encrypt Kubernetes Secrets at rest
Kubernetes stores Secrets unencrypted in etcd by default, and a vLLM deployment usually holds an API key, registry credentials and possibly a Hugging Face token there. Configure encryption at rest for the API server and confirm it is active on your cluster distribution. Ask your platform team how etcd snapshots used for backup are protected, since they contain the same data.
Restrict access to Secrets with least-privilege RBAC
Encryption at rest does not limit who can read a Secret through the API. Grant get, list and watch on Secrets only to the service accounts and operators that need them, and keep KServe and vLLM workloads out of broad cluster-wide roles. Review bindings in the serving namespace after each KServe upgrade, since its controllers create and read objects there.
Apply Pod Security Standards to the serving namespace
Pod Security Standards define baseline and restricted profiles that control privileges, host access and volume types for pods. Enforce at least the baseline profile on the namespace that runs vLLM and check whether the restricted profile works with your GPU device plugin and model cache volumes. Record any exemptions with the reason, because GPU workloads often need settings that a stricter profile rejects.
Verify exposure after every change
Each KServe upgrade, runtime change or new route can reopen a path that was previously closed. Test the deployed endpoint from a user network and from outside the cluster, and confirm that /metrics, /health and the control paths return refusals. Keep these checks in a script that runs with the deployment pipeline, so the evidence is repeatable for your reviewers.
How to verify the OpenAI-compatible API, metrics and failover
Verification should prove three things before anyone calls the service production-ready: the OpenAI-compatible API answers correctly through the gateway, the metrics endpoint is reachable only where your monitoring stack expects it, and the service keeps answering while pods are replaced. Run the checks against the KServe route your applications will use, not against a port-forward into the pod, because a port-forward skips the gateway, its authentication and its routing rules. Record which endpoint URL, KServe InferenceService or LLMInferenceService name, model identifier and vLLM version each check ran against, so the results can be repeated later.
Start with the API. From a test client placed where your applications run, call /v1/models first and confirm that the served model name matches what your clients will send. Then call /v1/completions and /v1/chat/completions with a short prompt. The chat call only works if the model has a chat template, or if one was supplied with the --chat-template option, so a failure there is often a serving configuration issue rather than a gateway issue. Use the same API key or token your applications will hold, and confirm that a request without credentials is refused. Keep the request and response bodies as evidence, with timestamps and status codes.
Next, test the boundary. The vLLM security documentation warns that the --api-key option does not cover /metrics, /health, /pause, /resume, /abort_requests or /invocations, and recommends a reverse proxy that allowlists only the endpoints meant for users. From a client outside the cluster and outside the monitoring network, request /metrics and the other control paths through the public route. Each should return a denial from the gateway or proxy. Then confirm that the same paths still work from inside the monitoring namespace, so Prometheus can scrape them. Ask your security team which allowlist rules they expect to see before the test, and store the rule set with the results.
Then check observability. Scrape /metrics from the monitoring stack and confirm that series for request counts, token throughput, latency histograms and KV cache usage are present and change during a test run. Send a small batch of requests, then compare the counters before and after. If you use non-default histogram buckets, confirm the bucket boundaries match your latency targets, since the metrics documentation says they are configurable. Also confirm that the metrics series carry the labels your dashboards and alerts depend on. A missing series usually means a scrape configuration or a version difference, so check the vLLM metrics documentation for the release you deployed.
Finish with failover. Delete one vLLM pod while a steady stream of test requests runs, and record how many requests fail, how long the service takes to recover and whether the pod comes back ready only after the model has loaded. Then run a rolling update to a new image or configuration and repeat the request stream. Failed requests during a rolling update are a finding to plan for, not a pass, so decide with your platform team what error rate and recovery time are acceptable for your applications. Because vLLM ships new minor releases about every two weeks, rerun the full sequence after each minor release you adopt and keep the evidence in your change records.
Common KServe and vLLM failures and their fixes
This table lists the failures teams most often hit when serving vLLM through KServe, with the likely cause and the fix for each. Use each row as a check before go-live and again after a KServe, vLLM or Kubernetes upgrade. The main takeaway is that most of these failures trace to settings that vLLM, KServe or Kubernetes leave for the operator to configure.
| Failure | Likely cause | Fix |
|---|---|---|
| Chat completions call returns an error | The model has no chat template and none was supplied at serve time | Supply a chat template with the serving arguments, then test /v1/chat/completions before release. |
| Pod stays pending | The accelerator request, node selector or toleration does not match any node in the cluster | Compare the pod's GPU request and selectors with node labels and taints, then read the scheduler event for the exact reason. |
| Model fails to load | Weights cannot be reached from the pod, or GPU memory is too small for the model and its cache | Confirm the storage path and credentials from inside the pod, and size GPU memory and context length before choosing the accelerator. |
| The /metrics endpoint is reachable by users | The --api-key option protects /v1 paths but not /metrics, /health or /invocations | Allowlist only user-facing endpoints at a reverse proxy, and let Prometheus scrape /metrics over an internal network path. |
| Runtime breaks after a KServe upgrade | Control plane components were upgraded out of order or past the Kubernetes version skew limits | Upgrade kube-apiserver one minor version at a time before controller-manager, scheduler and kubelets, and check KServe and vLLM compatibility notes first. |
| Secrets or etcd data exposed | Kubernetes Secrets are stored unencrypted in etcd by default, and no encryption at rest is configured | Enable encryption at rest, restrict list and watch on Secrets with least-privilege RBAC, and protect etcd snapshots with access control. |
| Multi-node traffic unprotected | Node-to-node traffic in a multi-node vLLM deployment has no authorization or encryption by default | Place the vLLM nodes on an isolated network, and confirm the network policies and access rules that separate them from other workloads. |
What does a vLLM on KServe integration cost and take?
Does vLLM cost money to license for a KServe deployment?
No. vLLM has no paid edition. It is open-source software licensed under Apache-2.0, so the money you spend goes to three places: accelerator capacity, the surrounding Kubernetes and KServe infrastructure, and the people who build and operate the platform. Those are the cost lines to plan for. Ask any provider, including DeployOpen, for a written scope before you accept a figure, because the scope is what turns a general estimate into a number you can compare against your own team's effort.
What drives the cost of a vLLM on KServe integration?
The main drivers are accelerator capacity (how many GPUs or other accelerators your models need, and whether you already own them), platform work (the state of your Kubernetes cluster, the KServe installation, and how you expose the model through the chosen serving path), gateway and identity integration (connecting endpoints to your API management and single sign-on), security review (network isolation, endpoint allowlisting and secrets handling), and ongoing upgrades as vLLM, KServe and Kubernetes each release on their own cycles. Each driver rises with the number of models, environments and approval steps you have.
Which costs are one-time and which recur after go-live?
One-time costs usually include design, the initial build, integration with your identity provider and gateway, load and functional testing, and documentation. Recurring costs include accelerator capacity you keep running, cluster and storage operations, monitoring, backups, security patching and version upgrades. Upgrades deserve a separate line because vLLM ships new minor releases about every two weeks, and Kubernetes ships roughly three minor releases a year. Ask the provider to split the quote into one-time and recurring items so you can budget each correctly.
How long does a vLLM on KServe integration take?
Duration depends on two things: the current state of your cluster and the serving path you choose. A cluster that already runs KServe with working GPU scheduling, identity and ingress needs much less platform work than a cluster that needs KServe installed, GPU operators configured and networking approved first. The serving path also matters: the Hugging Face serving runtime and the LLMInferenceService with llm-d involve different configuration and different operational skills. Ask for a timeline tied to written milestones, and check which assumptions about your environment it depends on.
What must our team provide before an integration starts?
Plan to provide cluster access with the permissions needed to install and configure workloads, an inventory of accelerators by type, count and location, the list of models with their sizes, chat templates and expected traffic, and access to your identity provider and gateway. You also need network approvals for traffic between nodes and clients, plus named owners who can approve security decisions. Missing items are the most common source of schedule slips, so confirm each one in writing before the scope is signed.
Can an internal platform team implement vLLM on KServe alone?
Often yes, if the team already runs Kubernetes in production, operates KServe for other models, and has engineers who understand GPU scheduling and model serving. The harder parts are usually not the install itself but multi-node networking, endpoint protection (vLLM's own API key does not cover every endpoint, so a reverse proxy is recommended), monitoring through its Prometheus metrics, and upgrade discipline. Assess those skills honestly. If they are thin, a fixed-scope engagement can cover the build while your team takes over operations.
What should a written scope for this work include?
A useful scope names the Kubernetes distribution and KServe version, the serving path, the accelerator types and count, each model and its target endpoints, the identity and gateway integrations, the security controls and evidence you expect, the testing approach, the handover and documentation, and the post-launch support model. It should also state what is excluded, such as model selection, model training or application changes. DeployOpen plans, implements and documents this work in infrastructure you control and provides a quote from that written scope, so use the same checklist to compare any proposal you receive.
How DeployOpen scopes and delivers a KServe serving integration
DeployOpen plans, builds, documents and supports vLLM on KServe inside your own clusters and cloud accounts, so the serving control plane stays under your control. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- Inventory existing KServe installation, version, serving runtimes and the models already served on the cluster
- Confirm whether the Hugging Face serving runtime or the LLMInferenceService with llm-d fits your traffic and model mix
- Record accelerator types, GPU counts per replica and the node pools that will host vLLM pods
- Agree in the written scope which API endpoints, authentication layer and metrics exposure are in or out
How we build
- Deploy the chosen KServe resources with pinned vLLM images, chat templates and model storage configuration
- Place vLLM pods behind an allowlisting reverse proxy, since the --api-key option does not cover every endpoint
- Wire Prometheus scraping of the /metrics endpoint and set up dashboards for request, throughput, latency and KV cache signals
- Test model loading, OpenAI-compatible request paths and multi-node networking isolation where the scope requires it
How we hand over
- Provide manifests, Helm values or GitOps configuration stored in a repository you own
- Document the endpoint allowlist, network isolation rules, secret handling and the runbook for model rollouts
- Deliver a verification record showing the tests run, the results and any open items agreed in the scope
- Walk your platform and security teams through operating the serving stack before handover closes
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover
- Retainer work can cover KServe and vLLM version moves, which vLLM releases ship about every two weeks
- Planned upgrade windows, regression checks against your models and updates to documentation
- Ask for the retainer scope, response expectations and upgrade cadence in the written quote
