What a vLLM production-stack deployment on Kubernetes includes
A platform team that must serve open-weight models to internal applications inside its own cluster needs an inference layer it can install, upgrade and monitor like the rest of its Kubernetes estate. The vLLM production-stack is one of the Kubernetes deployment options listed in the vLLM documentation, and it is installed with a Helm chart. This section describes what that chart is meant to put in place, so the team can decide which parts it will run, which it will replace and which it will leave out. Because the stack is built around vLLM's OpenAI-compatible server, applications that already call the OpenAI API can be pointed at endpoints inside the cluster.
The stack has four main parts. A router sits in front of the model replicas and distributes incoming requests across them. Model replicas are vLLM server pods, each serving a model on accelerators the cluster provides. Grafana provides the observability layer, drawing on vLLM's Prometheus-format metrics, which cover request counts, token throughput, latency and KV cache use. LMCache provides KV cache offloading, which extends where cached attention state can be kept beyond accelerator memory. Each part is a separate decision. Confirm in the chart documentation which parts are enabled by default, which need their own resources and how the router finds the replicas it routes to.
vLLM is licensed under Apache-2.0 and has no paid edition, so the engine itself adds no license line to the budget. The cost comes from four places: accelerator capacity, the infrastructure around it (nodes, storage, networking and metrics retention), implementation by your team or a partner, and the people who operate upgrades, capacity changes and incidents. Grafana is a separate component with its own licensing. Grafana core is licensed under AGPLv3 and Grafana Enterprise is a commercial option, so confirm which edition your standards allow. Ask for each cost line to be quoted separately, and mark which lines are one-time and which recur.
Before committing, confirm the chart's value names, their defaults and the supported replica and router settings in the chart documentation, and check the vLLM Kubernetes deployment page for the supported arrangement of components. Confirm that the LMCache release you plan to use is compatible with your vLLM version. vLLM ships new minor releases about every two weeks, so a pinned chart should be matched to a tested engine version, and that pairing needs a named owner for each upgrade.
The router is the only component that users should reach directly. The vLLM --api-key option does not cover /metrics or /health, and the security documentation advises against relying on it alone, so plan a reverse proxy that allowlists only the endpoints meant for users. Secrets that hold keys should be encrypted at rest in Kubernetes, because they are stored unencrypted in etcd by default. Multi-node traffic between vLLM nodes is also insecure by default, so those nodes need an isolated network.
Prerequisites for a vLLM production-stack deployment on Kubernetes
Confirm these six items before you run the Helm install. Together they determine whether the router, model replicas and observability stack can start, stay current and reach the models they serve. Most delays come from gaps in Kubernetes versions, accelerator support or network placement, and each is inexpensive to check early.
Supported Kubernetes version
Run the production stack on a Kubernetes minor version the project still maintains. As of September 2026, the supported minors were 1.35, 1.36 and 1.37, with 1.35 reaching end of life on February 28, 2027 and 1.37 on October 28, 2027. Check the Kubernetes releases page on the day you install, because the list changes with each release cycle. Confirm that your managed or distribution provider offers the same minor version and that you have a plan for the upgrade path that follows.
Accelerator nodes vLLM supports
Provision nodes with a GPU or other accelerator that vLLM supports: NVIDIA CUDA, AMD ROCm or Intel XPU, or a third-party hardware plugin maintained outside the main repository. Check the vLLM installation documentation for the driver, runtime and model combinations you plan to run. Test one model replica on the target node type before you size the cluster, and confirm who owns driver upgrades, since the node driver stack and the vLLM image must match the hardware.
Helm client and chart pin
Install a Helm client on the workstation or pipeline that runs the deployment, and store the chart version as a pinned value in version control. vLLM ships minor releases about every two weeks, with v0.31.0 dated October 5, 2026, so a floating image tag or chart version can change behavior between installs. Confirm which vLLM version each chart release targets, set the image tag and chart version together, and record both in your change log before each upgrade.
Model artifact source
Decide where model weights and tokenizer files come from, and confirm the cluster can reach that source when pods start. A registry or object store inside your network keeps startup predictable and avoids pulling large files from outside your boundary on every replica restart. For air-gapped clusters, stage a verified copy of the model files and consider the --enable-offline-docs option so the server can run without external assets. Confirm the license terms for each model you copy.
Chat templates for chat models
Each chat model must provide a chat template, or you must supply one with the --chat-template option, before the chat completions API will serve it. Review the template shipped with each model revision, because a template mismatch can produce malformed prompts that look like model quality problems. Keep each template under version control beside the model reference, and test the chat completions endpoint with your own message format before routing user traffic to it.
Isolated network for multi-node traffic
If a model spans nodes through pipeline parallelism, place those nodes on a dedicated network segment. The vLLM security documentation states that communication between nodes is insecure by default, with no authorization or encryption, so the segment must be isolated from user traffic and other tenants. Plan firewall rules for the Ray or multiprocessing ports you use, and confirm that only the router and approved clients can reach the public API.
How the router, model replicas and Grafana fit together
Clients send every request to the router, which is the single entry point the rest of the design depends on. The router forwards each request to one of the vLLM model replicas. Each replica serves OpenAI-compatible paths such as /v1/chat/completions and /v1/completions, and the vLLM server also exposes /v1/responses and /v1/embeddings. Applications written for the OpenAI API can keep their client code and change the base URL. Before you set client timeouts, confirm in the production-stack documentation how the router selects replicas, how it handles unhealthy pods and how it treats streaming responses. Also write down which paths external users may reach, because the vLLM --api-key option does not cover every endpoint.
vLLM exposes Prometheus-format metrics at /metrics on its API server. The endpoint covers request counts, token throughput, latency histograms and KV cache use, and the histogram buckets can be configured. Grafana does not read the replicas directly in the way a dashboard reads a database. It reads the data through a metrics store, typically a Prometheus-compatible database that scrapes the endpoint, so retention and sizing of that store belong in your plan. Confirm which metrics component the chart deploys. Treat /metrics as an internal path, because the vLLM security documentation says --api-key does not protect it, so block it at the proxy for outside traffic. Grafana's configuration database is a separate decision. Grafana stores settings in SQLite by default, and its installation docs say SQLite does not scale for production, so plan MySQL 8.0 or later or PostgreSQL 12 or later for production and high availability.
LMCache sits beside the replicas as a KV cache offloading layer, which moves attention cache data out of GPU memory so that repeated context can be reused. Its behavior, supported backends and configuration must be confirmed against the production-stack documentation before you size storage or write the values file. Ask which backends the chart supports, where cached data is stored, what happens to cached entries when a replica restarts and how the cache interacts with the router's routing decisions. Grafana authentication is also a separate decision from the database choice. All Grafana editions support LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, auth proxy and JWT, while SAML, team sync, active sync and SCIM provisioning are available only in Grafana Enterprise and Grafana Cloud. Choose the identity method early so that dashboard access is decided with the same identity source as the rest of the platform.
The parallelism rule determines how each replica is built, and that determines what the router can treat as one unit. vLLM uses tensor parallelism within a node and pipeline parallelism across nodes. A replica that fits on one node needs only tensor parallelism, and a replica that spans nodes needs a multi-node backend, either Ray or vLLM's own multiprocessing backend. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, and that nodes must be protected by placing them on an isolated network. Plan node labels, scheduling and network policy for each multi-node replica before you register it behind the router, and confirm the chart's supported combinations with the production-stack documentation.
How to install the production stack in the right order
This sequence takes the production-stack chart from a pinned, reviewed install to a stack that serves users, with the control plane, secrets, routing, observability and load behavior each verified before the next layer depends on it. Each step ends with a check you can repeat, so the order also works as an acceptance checklist. Confirm exact chart and image versions against the vendor documentation for your environment.
Confirm the chart source and pin versions
The platform team confirms which production-stack Helm chart repository and release it will use, then records the chart version and the vLLM image tag in version control. Check both against the vLLM Kubernetes deployment documentation and the chart's release notes. vLLM ships new minor releases about every two weeks, so an unpinned image could change between environments without a review. Done when a manifest rendered from the pinned chart shows the same image tag your change record lists.
Build a highly available control plane first
The cluster team builds the control plane before any accelerator workload is scheduled. Use at least three control plane nodes behind a TCP load balancer on port 6443, with either a stacked etcd topology or an external etcd cluster. Kubernetes guidance calls for an odd number of etcd members, and recommends five in production, so confirm your topology with your own capacity review. Take an etcd snapshot with etcdctl snapshot save, store it off the cluster, and test a restore with etcdutl. Done when the load balancer health checks pass on every control plane node and a restore from snapshot has succeeded in a non-production cluster.
Enable Secret encryption at rest
Before the first model access token or registry credential is stored, the cluster team configures encryption at rest for Secrets in etcd. Kubernetes stores Secrets unencrypted by default, so this step cannot be left for later. Restrict list and watch access to Secrets with least-privilege RBAC, and decide whether an external store through the Secrets Store CSI Driver fits your key management model. Done when a test Secret is no longer readable in plain text in etcd and the RBAC review shows no broad list or watch grants on Secrets.
Install the chart and verify internally
The deployment engineer installs the pinned chart with one replica per model, setting accelerator resource requests and limits so each vLLM pod lands on a node with free capacity. Run the install in a namespace governed by your Pod Security Standards settings. Then open a debug pod inside the cluster and call the models list and the completions endpoint through the router service. Done when the completions request returns a completion from the expected model and every model pod reports ready.
Restrict the proxy to user endpoints
The platform team configures the router or ingress so that only the endpoints meant for users, such as completions and chat completions, are reachable from outside the cluster. The vLLM --api-key option does not protect /metrics, /health, /pause, /resume, /abort_requests or /invocations, and the vLLM security documentation recommends a reverse proxy that allowlists user endpoints, so do not rely on the key alone. Done when a request to /metrics from outside the allowlist is refused, and the same request from the monitoring namespace still succeeds.
Install Grafana on MySQL or PostgreSQL
The observability engineer installs Grafana with MySQL 8.0 or later, or PostgreSQL 12 or later, as its configuration database. The Grafana installation documentation says SQLite does not scale for production, so it is ruled out for this stack. Define the Prometheus data source and the vLLM dashboards as YAML or JSON provisioning files loaded at startup, and keep them in version control. Done when a panel shows request counts and token throughput scraped from /metrics after a test request, and a redeployed Grafana recreates its data source and dashboards from the files alone.
Enable LMCache after a baseline load test
Run a load test against the stack with LMCache disabled and record latency, throughput and KV cache use from /metrics as the reference point. Then enable LMCache KV cache offloading in a staging release, rerun the same test profile, and compare the two results. Keep LMCache only if the comparison supports it and the team that will operate the stack understands the cache behavior. Done when both result sets are stored together in version control and the production values file reflects the chosen setting.
Run functional and load tests before users
The test lead runs a functional suite covering completions, chat completions, embeddings and the model behavior your applications depend on. Confirm each chat model has a chat template, either built in or supplied with the chat template option, because chat completions require one. Then run a load test at the concurrency your capacity review expects, and check that the Grafana alerts you defined fire on the failure conditions you chose. Done when the functional suite passes on the pinned versions, the load test meets the targets in your written scope, and the sign-off is recorded before any user traffic reaches the proxy.
How the Helm chart compares with plain manifests
The production-stack Helm chart bundles parts of a vLLM deployment that a team would otherwise design and maintain by hand: a router in front of model replicas, observability wiring and LMCache options for KV cache offloading. With native manifests, each of these becomes a separate design task. Confirm the chart version you plan to use, because the bundled components and their defaults can change between releases.
- Router: the chart places a routing layer in front of model replicas so clients call one endpoint. Confirm how it detects unhealthy replicas and which authentication options it supports.
- Observability wiring: vLLM exposes Prometheus-format metrics at /metrics. Confirm which scrape configuration, dashboards and alert rules the chart includes.
- LMCache options: the chart exposes KV cache offloading settings. Confirm which options your accelerators and storage tiers support.
- Default values: the values file sets model, resource and replica defaults. Review each default before the first deployment.
Ask the chart maintainers for a written list of every packaged component, its version and its default settings before you estimate the effort of building the same pieces yourself.
Packaging moves work but does not remove it. Under either approach, your team owns the cluster, model storage, Secrets, network policy and the access path to the endpoint. The difference lies in how much of the vLLM configuration your staff read and maintain directly. With native manifests, your team writes and reviews every Deployment, Service, ConfigMap and autoscaling object. With the chart, your team reviews rendered templates and values and must still understand what those templates produce.
- Template review: render the chart with your values and review the output as you would hand-written manifests. Keep the rendered output in version control.
- Skills: Kubernetes networking, GPU scheduling and vLLM serving flags are needed in both cases. Helm chart internals are an extra skill the team must hold.
- Secrets: Kubernetes stores Secrets unencrypted in etcd by default, so encryption at rest, least-privilege RBAC and external secret stores are your responsibility under either approach.
- Support boundary: confirm who handles chart defects, how fixes reach your cluster and whether your support agreement covers the chart.
List which of these tasks your staff already perform today and which would be new to them before you choose an approach.
vLLM ships new minor releases about every two weeks, for example v0.31.0 on October 5, 2026, according to its GitHub releases. Each release can change images, flags and defaults, so the upgrade process matters as much as the first install. Chart upgrades carry the same risk, and the question is how easily your team can see what will change before it reaches production.
- Upgrade diffs: with the chart, render the output for the old and new chart versions and diff them before applying. With manifests, diff your own files against the flags of the new vLLM release.
- Drift control: compare live cluster state with version-controlled source on a schedule. Helm records release revisions, but manual edits can persist until the next upgrade.
- Release review cadence: set a regular review window for vLLM and chart releases so changes are reviewed in batches.
- Kubernetes skew: kube-apiserver must be upgraded one minor version at a time, and kubelets may be up to three minor versions older. GPU node pools need their own upgrade schedule.
- Pinning: pin chart, image and model versions explicitly in either approach.
Ask for the upgrade procedure the chart maintainers document, and run one full upgrade in staging before you rely on it in production.
The vLLM Kubernetes documentation lists native manifests, Helm, the production-stack, AIBrix, KServe, KubeRay, LeaderWorkerSet, llm-d and NVIDIA Dynamo. The chart is one option among these, and the best fit depends on what your platform team already runs. A team that operates KServe for other models may prefer the KServe path, which has its own sibling page in this group. A team that needs multi-node serving may evaluate KubeRay, because Ray clusters carry their own network and operations requirements.
- KServe: provides one serving control plane for several model types when the platform team already operates it.
- KubeRay: supports multi-node vLLM on Ray clusters, with network isolation requirements for Ray and PyTorch distributed traffic.
- Native manifests: give full visibility and few dependencies, at the cost of maintaining more YAML.
- Other projects: AIBrix, LeaderWorkerSet, llm-d and NVIDIA Dynamo each need separate evaluation of maturity, support and fit.
- Existing skills: tooling and staff experience often decide the outcome more than feature lists do.
Write down which serving control plane your organization already operates, then compare the options against that baseline.
Test the choice in a staging cluster that matches production as closely as possible: the same Kubernetes minor version, accelerator type, ingress path, model and traffic pattern. Neither approach removes the security steps on this page. vLLM's own security documentation says the --api-key option does not protect /metrics, /health or several other endpoints, so an allowlisting reverse proxy is still needed. Multi-node traffic also needs an isolated network.
- Staging parity: match the Kubernetes version, GPU nodes and ingress path to production before you measure anything.
- Measured outcomes: record rendered configuration size, upgrade diff time, rollback steps and the number of manual fixes needed during the test.
- Endpoint exposure: allowlist only the endpoints meant for users at a reverse proxy, under either approach.
- Pod security: apply the Pod Security Standards to the namespace that runs vLLM and its router.
- Observability access: if Grafana is deployed, require authentication, secure cookies and a Content Security Policy as its hardening documentation recommends.
Record the test results and the security checks in one document before you select an approach.
Identity, endpoints and data mapped between the stack components
This table maps each identity, credential, endpoint and data item in the production stack to the component that owns it and the setting that controls it. Read each row as a question to answer in your own cluster: who authenticates, which paths are reachable and where the data lives. The main takeaway is that the vLLM --api-key option does not cover /metrics or /invocations, so a proxy allowlist and network policy must close those gaps.
| Item | Owning component | Controlling setting or check | Team to confirm |
|---|---|---|---|
| End-user application credentials | vLLM server (--api-key) and the reverse proxy in front of it | --api-key protects /v1, /v2, /inference and /cohere only. Allowlist user-facing paths at the proxy. | Which paths the proxy exposes, and that /invocations, /metrics, /health and pause or abort paths are unreachable from user networks. |
| Cluster administrator identity | Kubernetes API server | X.509 client certificates, OIDC or JWT bearer tokens, webhook token authentication or an authenticating proxy. | Which method your distribution enables, and who holds cluster-admin bindings in the vLLM namespaces. |
| Model and serving secrets | Kubernetes Secrets in etcd | Secrets are stored unencrypted in etcd by default. Enable encryption at rest, restrict list and watch with RBAC, or mount from an external store through the Secrets Store CSI Driver. | That encryption at rest is enabled, and which service accounts can read Secrets in the router and model namespaces. |
| Grafana user identity | Grafana authentication settings | LDAP, generic OAuth, Entra ID, Okta, GitHub, GitLab, Google OAuth, auth proxy and JWT work in all editions. SAML and SCIM need Grafana Enterprise or Grafana Cloud. | That your edition supports the identity provider you use, and whether team sync is needed and available. |
| vLLM metrics | vLLM /metrics endpoint, Prometheus scraper and Grafana | Prometheus-format endpoint on the API server that the --api-key option does not protect. Restrict it with network policy so only the scraper can reach it. | The scraper's identity, and that users cannot reach /metrics through the ingress or the proxy. |
| Dashboards and data sources | Grafana provisioning files kept in version control | Data sources and dashboards are defined in YAML or JSON and loaded at startup. Terraform and Git Sync are further as-code options. | The Grafana database backend, which must be MySQL or PostgreSQL for production and high availability, and who can edit provisioned resources. |
How to harden the stack before users reach it
These controls sit between users and the router, the model replicas, the cluster API and the dashboards. Each one closes a path that default vLLM and Kubernetes settings leave open, so review them together before any user traffic reaches the stack. Treat each card as a check to run against your own manifests, proxy configuration and network design.
Allowlist user endpoints at the proxy
The --api-key option protects /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security docs warn against relying on the key alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Place the proxy in front of the router, allow the /v1 routes your applications call, and test every other path from outside the cluster to confirm it returns an error.
Keep operational endpoints off user paths
The /metrics, /health, /pause, /resume and /abort_requests endpoints expose telemetry and change serving state for every caller. Serve them on an internal service or port that only monitoring, health probes and operators can reach, and block them at the proxy and at any ingress. Confirm that Prometheus scrapes /metrics over the internal network and that health probes do not depend on any public route.
Isolate multi-node vLLM traffic
The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Place the nodes that exchange pipeline parallel traffic on a dedicated network segment, and use Kubernetes network policies or separate subnets so only those nodes reach each other's worker ports. Confirm that the router and clients cannot reach these internal ports, and document which ports your chosen backend opens.
Encrypt Secrets and limit list and watch
Kubernetes stores Secrets unencrypted in etcd by default, so configure encryption at rest for the API server before storing API keys or registry tokens. Grant list and watch on Secrets only to the service accounts and operators that need them, because these verbs can return secret values in bulk. Also decide whether an external store, reached through the Secrets Store CSI Driver, fits your key management policy.
Namespace policy and the CIS benchmark
Pod Security Standards define baseline and restricted policy levels that Kubernetes enforces through namespace labels. Enforce the most restrictive level your GPU workloads allow on the namespace holding the router, replicas and Grafana, and test warnings in a non-production namespace first. Use the CIS Kubernetes Benchmark as the reference to measure the cluster against, and record each deviation and its reason. Passing a benchmark check does not certify the platform.
Harden Grafana before exposing dashboards
The Grafana security hardening docs recommend secure and SameSite cookies, a Content Security Policy, Trusted Types, hiding the version from unauthenticated users, requiring authentication for the metrics endpoint and domain verification against DNS rebinding. Set each option in the Grafana configuration or provisioning files, then check response headers and login behavior from a browser. Keep dashboards off the public user path, since they display the same request data the operational endpoints expose.
Why the vLLM API key alone does not protect this stack
The vLLM --api-key option protects the /v1, /v2, /inference and /cohere endpoints. It does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security documentation warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints meant for users. In a production-stack deployment, the key is one control among several, not the boundary. A team that sets a key on the model replicas and assumes the stack is protected leaves pause and resume controls, request abort, and metrics reachable by anyone who can open a connection to the service.
The first decision is where the allowlist is enforced. Applications call the production-stack router, so the allowlist can sit in the router configuration, in an ingress or gateway in front of it, or in both. The vLLM documentation describes the allowlist pattern as the recommendation, but it does not state how the production-stack router treats these paths. Confirm in the chart documentation and chart values which paths the router forwards to model replicas, whether it forwards /pause, /resume or /metrics, and whether it adds any authentication of its own. Record the answer before deployment, because it determines whether the allowlist is a chart setting or a separate component your team operates.
Test the result from a machine on the user network, not from inside the cluster. Send a request to each non-allowlisted path (/invocations, /pause, /resume, /abort_requests, /metrics and /health) through the same host and port that application traffic uses. Use the HTTP method each endpoint expects, and repeat each call with and without a valid API key. Every call should be refused. Record the status code and response body for each path. Any response that returns model output, metrics data or a success code is a finding. Include a positive control, a valid request to /v1/chat/completions with the key, so a pass is not caused by the service being unreachable. Repeat the test after every chart upgrade and every ingress change.
The allowlist must not break in-cluster traffic. Kubelet health checks and Prometheus scraping originate inside the cluster, and they may call /health or /metrics. Confirm which paths the chart's liveness, readiness and scrape settings use, and whether those calls pass through the same proxy. If they do, limit them by source network, namespace or service account so the user network stays blocked. Keep the test results with the release record. An auditor or security reviewer who asks how the allowlist was verified should receive dated results from the user network, not a statement that the setting exists.
Common failures in production-stack deployments and their fixes
This table lists the failures teams most often hit when running the vLLM production stack on Kubernetes, with the cause and the documented fix for each. Read each row as a checklist item to verify in your cluster before go-live. The main takeaway is that most failures trace to a missing setting, an open endpoint or a single-node default that the documentation expects you to change.
| Failure | Cause | Fix |
|---|---|---|
| Chat completions fail for a model | The model does not provide a chat template, so the chat completions API cannot format messages. | Supply a template with the --chat-template option on vllm serve, then test /v1/chat/completions with a sample request. |
| User traffic reaches /metrics or /health | The reverse proxy in front of vLLM forwards all paths. The --api-key option does not cover /metrics, /health, /pause or /abort_requests. | Configure the proxy to allowlist only the /v1 paths meant for users, and scrape metrics from inside the cluster network. |
| Grafana dashboards are empty | The Prometheus scrape target points to the wrong port or path, or the Grafana data source does not match the Prometheus service. | Check that the vLLM /metrics target shows as up in Prometheus, then confirm the Grafana data source URL and provisioned dashboard references. |
| Grafana runs on SQLite in production | SQLite is the default database. Grafana's installation docs say it does not scale for production. | Point Grafana at MySQL 8.0 or later, or PostgreSQL 12 or later, before the stack takes real users. |
| Grafana HA sends duplicate alerts, loses Grafana Live features or fails logins across nodes | Two or more Grafana servers run without a shared database, so sessions are node-local, alerting is not set up for HA and Grafana Live has HA limits. | Run active-active servers behind a load balancer on a shared MySQL or PostgreSQL database with database-backed auth tokens, then follow the documented HA alerting guidance and check Grafana Live behavior. |
| Multi-node vLLM traffic is exposed | The vLLM security docs state that communication between nodes is insecure by default, with no authorization or encryption. | Place vLLM nodes on an isolated network, and restrict pod-to-pod traffic to the nodes that need it, confirming the policy with your network team. |
| Secrets are readable by broad RBAC or unencrypted in etcd | Kubernetes stores Secrets unencrypted in etcd by default, and broad list or watch permissions expose them to users and service accounts. | Enable encryption at rest for Secrets, restrict list and watch access with least-privilege RBAC, or use an external store through the Secrets Store CSI Driver. |
| Upgrades blocked by version skew | Kubelets are older than the allowed range relative to kube-apiserver, or components were upgraded out of order. | Upgrade kube-apiserver one minor version at a time first, then kube-controller-manager, kube-scheduler and kubelets, keeping kubelets within three minor versions of kube-apiserver. |
| etcd loses quorum | The etcd cluster has an even number of members, so a single member failure can prevent a majority from forming. | Run an odd number of members, use five members for production as the Kubernetes documentation recommends, and schedule etcdctl snapshot save backups with a tested restore using etcdutl. |
What a vLLM production-stack deployment on Kubernetes costs and needs from your team
What does a vLLM production-stack deployment on Kubernetes cost?
vLLM has no license fee and no paid edition, so the cost comes from what it runs on and the people who run it. The main cost components are:
- Accelerator capacity, such as NVIDIA CUDA, AMD ROCm or Intel XPU GPUs, or another supported accelerator
- The Kubernetes cluster, storage, networking and the infrastructure it runs on
- Grafana for observability, which is licensed under AGPLv3 in its open-source form, with Grafana Enterprise and Grafana Cloud as separate commercial options
- Implementation work and ongoing operations
Implementation is usually a one-time cost, while accelerators, infrastructure, upgrades and support recur. Costs rise with the number of models and replicas, larger models that need multi-GPU parallelism, and stricter identity or compliance requirements. DeployOpen does not publish a price list. A quote follows from a written scope that names your models, accelerators, cluster and identity needs.
How long does a vLLM production-stack deployment take to build?
No duration figure is available for this topic, so ask for a scoped timeline in writing before work starts. The main drivers are the number of models and replicas, whether accelerator nodes are installed and reachable from the cluster, and how long identity integration takes. Testing also adds time: checks on the router, failover between replicas, and verification of the OpenAI-compatible endpoints your applications will call. Ask the partner to split the work into phases with a clear handoff for each, and to state which steps depend on your team's availability. Timelines usually move when hardware arrives late or when network changes are waiting for approval, so ask how the plan handles those risks.
What does our team need to provide for a vLLM engagement?
Plan for five things from your side: cluster access, accelerator nodes, identity owners, a test plan and an upgrade owner. Cluster access should allow installing the Helm chart and managing the namespaces it uses. Accelerator nodes need drivers ready for the hardware you chose. Identity owners must be named for the sign-in systems that front the API and Grafana. The test plan owner sets acceptance criteria for response quality, latency and failover. The upgrade owner tracks releases, schedules staging tests and approves production changes. vLLM ships new minor releases about every two weeks, and Kubernetes ships about three minor releases a year. Confirm who owns each item before the scope is signed, because unassigned items stall work.
Can our own team deploy vLLM on Kubernetes without a partner?
Yes, if your team already runs Kubernetes in production and has GPU operations skills. Those skills include managing drivers and device plugins, scheduling accelerators, watching GPU memory and recovering from node failures. You also need time for testing, covering model compatibility, router behavior, failover and identity integration, and the discipline to repeat those tests after each upgrade. Check whether your team can keep that pace alongside vLLM's release cycle. A practical way to decide is to write down the skills, name the people who hold them and estimate the testing hours. Then compare that plan with a written scope from a partner. Many teams split the work, using a partner for the first build and keeping day-to-day operations in-house.
When are plain Kubernetes manifests enough for vLLM?
Plain manifests are enough when you serve one model with a fixed replica count, do not need a router across replicas, and are willing to build metrics dashboards and cache handling yourselves. The vLLM docs list native manifests and Helm among the Kubernetes options, alongside the production-stack, AIBrix, KServe, KubeRay, LeaderWorkerSet, llm-d and NVIDIA Dynamo. The production-stack chart adds a router in front of model replicas, Grafana observability and LMCache KV cache offloading, packaged and tested together. Choose it when you need those parts as one unit. Choose manifests when your platform team already has standards for them and wants fewer moving parts. Either way, someone must maintain the /metrics endpoint, the router and the dashboards.
How are vLLM upgrades and ongoing support handled after go-live?
Ongoing upgrades and support can run under a retainer, and the work covered is the work named in the agreement. Plan for several release streams. vLLM ships minor releases about every two weeks. Kubernetes ships about three minor releases a year, and its version skew policy requires kube-apiserver to be upgraded one minor version at a time, before kube-controller-manager, kube-scheduler and kubelets. Kubelets may run up to three minor versions behind kube-apiserver. Grafana minor releases arrive about every other month, patch releases at least monthly, and each minor release is supported for 9 months. Ask for the release intake process, the staging test procedure, the rollback steps and how security patches are handled outside the normal cycle.
Does DeployOpen certify our vLLM platform for compliance?
No. DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies do that work. For vLLM, the security docs say node-to-node traffic in multi-node deployments is insecure by default and must be placed on an isolated network. They also warn that the --api-key option should not be relied on alone, and recommend a reverse proxy that allowlists only the endpoints meant for users. Ask the partner which controls it will build, which evidence it will produce, and which items your auditor will need to review.
What should a vLLM deployment quote include?
A quote should follow a written scope that names the models, replica counts, accelerator types and counts, the Kubernetes distribution and version, the identity provider, the observability components, the test plan, the upgrade responsibilities and any retainer. Ask which costs are one-time and which recur, such as accelerator capacity, cloud or data center charges, Grafana licensing if you choose a commercial tier, and upgrade cycles. Ask how scope changes are priced, who owns the cluster and accelerators, and what the handover includes: runbooks, dashboards, configuration kept in version control and test results. DeployOpen does not publish a price list, so compare quotes only when they cover the same scope.
How DeployOpen scopes, delivers and supports this stack
DeployOpen treats a vLLM production-stack deployment on Kubernetes as a planned platform build inside your own clusters and network. The work starts with a written scope covering model replicas, the router, observability and cache offloading, then moves through build, documentation and handover to your team.
How we scope
- The written scope names the models, replicas, accelerator types and the Kubernetes distribution the stack will run on.
- Scope defines how the router, Grafana dashboards and LMCache offloading are configured, and which of them are in or out.
- Engagements are fixed-scope, so deliverables and timelines are agreed in writing before the build begins.
How we build
- Helm values are set per environment, with image sources, storage classes and accelerator scheduling defined before deployment.
- Access to vLLM endpoints is tested against the security docs, which say --api-key alone is not enough and a reverse proxy should allowlist user endpoints.
- Metrics from /metrics are checked in Grafana, and Grafana's own high availability and secrets settings are verified before handover.
How we hand over
- Handover includes architecture notes, Helm values, runbooks for scaling replicas and rotating keys, and the recorded test results.
- Your team receives documented steps for upgrading the chart, vLLM and Grafana, since vLLM ships minor releases about every two weeks.
- Knowledge transfer sessions walk operators through alerts, dashboards and the etcd and cluster backup procedures the platform depends on.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after the initial build is complete.
- Retainers can cover planned moves to newer vLLM releases, Kubernetes minor versions and Grafana patch releases.
- Ask any partner to confirm response scope, environment access and which support hours are included in writing.
