vLLM Kubernetes deployment

The accelerator request in the pod spec decides where a vLLM pod can run and whether it starts at all. A pod asks for GPUs through a resource limit exposed by the accelerator's device plugin, and the Kubernetes scheduler only places it on a node that advertises at least that many free devices. Node selectors and taints then narrow the choice to GPU node pools with the right accelerator model, memory and interconnect. If the requested count is higher than any single node provides, the pod stays Pending indefinitely. If the count is lower than what the model needs for its weights and KV cache, the pod may schedule but fail while loading the model or during the first requests, often with an out-of-memory error that looks like a runtime bug. Confirm the GPU count per node in your pool and size the request to it before you write the manifest.

Tensor parallelism is the setting that ties the GPU count to the model. vLLM splits a model across GPUs within one node through tensor parallelism, so the tensor parallel size in the server arguments must match the number of GPUs the pod requests and those GPUs must be on the same host. Spreading one replica across nodes is a different mode (pipeline parallelism with Ray or the built-in multiprocessing backend), and it needs a network that the vLLM security docs describe as insecure by default. Keep the GPU limit, the tensor parallel setting and any shared memory volume for inter-process communication in agreement. A mismatch in any of the three is a common cause of pods that crash at load time rather than failing to schedule.

Model weights can reach the pod in three ways, and the choice shapes cold start time, storage cost and version control. A persistent volume shared across replicas keeps one copy of the weights, so new pods start quickly and every replica runs the same files. It requires a storage class that supports the access mode you need, and it adds a recurring storage charge. A pre-populated cache, built into a node image or loaded onto local disks, gives the fastest and most predictable startup but ties weight updates to image or disk rebuilds. Downloading at startup needs no storage setup, yet every new pod pulls gigabytes of weights, which lengthens scale-out and makes restarts depend on the model host being reachable. Whichever path you choose, pin the model by revision or checksum so the running weights are known and reproducible.

The scheduling configuration depends on the accelerator type. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, and each one has its own device plugin, resource name, container image and driver or runtime stack. An NVIDIA pool typically uses the NVIDIA device plugin and a matching CUDA driver, while AMD and Intel pools use their vendors' plugins and resource names. The vLLM image and build must match the accelerator, so a manifest written for one vendor will not schedule or run on another without changes. Before writing the pod spec, confirm with your cluster team and the device plugin vendor the exact plugin version, driver version, runtime class and resource name that your nodes expose, and check them against the vLLM installation requirements for the release you plan to run.

A model that does not fit on one node cannot be served by a single pod. The model weights and the KV cache have to be split across GPUs on several machines, and vLLM can only do that if every participating process is up, reachable and started with the same configuration. For this reason the deployment has to treat a leader pod and its worker pods as one unit that starts together, becomes ready together and is replaced together. Scheduling them as independent Deployment replicas breaks that assumption, because a leader can come up while its workers are still pending and serve nothing useful. Inside a node, vLLM usually applies tensor parallelism, which splits each layer across the GPUs connected by a fast local interconnect, so every forward pass needs frequent communication between devices. Across nodes, the documented approach is pipeline parallelism, which assigns contiguous groups of layers to each node and passes activations along the chain. Pipeline stages communicate less often, but they still depend on the network between nodes for every step. Ask your platform team to measure the bandwidth and latency between nodes in the chosen cluster before you settle on a node count, because a slow path between stages often shows up as lower throughput rather than as an error. A multi-node group fails as a whole. If one worker crashes, loses its GPU or cannot reach the leader, the remaining processes block on collective operations and may hang rather than exit, so the leader can report a healthy state while no requests complete. Your runbook should define how the group is detected as unhealthy, how long it may remain stalled and whether the entire group is restarted. Confirm that your health checks test inference, not only process liveness, and that a single pod failure triggers recreation of the whole group. LeaderWorkerSet and KubeRay solve different parts of this problem, and most multi-node designs need one or the other, sometimes both. LeaderWorkerSet manages the group lifecycle: it creates a leader and a fixed set of workers as a unit, gives them stable identities for discovery and handles group-level rollout and restart. KubeRay manages a Ray cluster on Kubernetes, including head and worker pods, the Ray runtime and autoscaling of its workers. vLLM can use Ray as its multi-node backend, while it can also use its own multiprocessing backend, so the choice is partly a choice of runtime. Before you commit, confirm four things with your own environment and the vLLM documentation for your version. First, which multi-node backend your model and vLLM release support. Second, that the network path between nodes is isolated, because vLLM's security documentation states that inter-node traffic is insecure by default and has no authorization or encryption. Third, the restart behavior you want for a failed worker, including whether a KubeRay or LeaderWorkerSet restart recreates the whole group. Fourth, how the scheduler places pods on GPU nodes with the required interconnect, since a group split across poorly connected nodes will not perform as planned.

The acceptance suite for a vLLM Kubernetes deployment should prove three things: the model serves correct answers, the endpoints that must stay private are unreachable from outside the cluster, and the platform recovers from failure without someone repairing it by hand. Agree these tests with the client before go-live, because they become the definition of done for the implementation and the baseline for every later change. Start with readiness. The pod should report ready only after the model weights have loaded and the server can accept requests, so the test is to start a pod with a cold model cache, confirm it does not receive traffic while loading, and confirm that the readiness probe passes once loading completes. Then send one chat completion and one embeddings request through the gateway and check that the responses follow the OpenAI-style schema the client applications expect. Run the chat request against the model's chat template as well, using a fixed prompt with a known expected structure, because a missing or wrong template can produce output that looks valid but breaks the application's parsing. Confirm the running container's image tag matches the pinned release recorded in the change ticket, not the tag in the manifest alone. Next, test the boundary. From a host outside the cluster and outside the allowed network, request the control and metrics paths, including the paths the vLLM security documentation says the API key does not protect, such as /metrics, /health and /pause. Each request should be refused at the gateway or network layer. The vLLM documentation recommends a reverse proxy that allowlists only user-facing endpoints, so the test should confirm that the allowlist, not the API key, is what blocks these paths. Repeat the check from inside a user namespace to confirm that application teams cannot reach control paths either. The failure tests are the ones auditors and change reviewers ask about most. Delete a serving pod and confirm the replacement reloads the model and returns to service without manual steps. Drain a GPU node and confirm that pods reschedule to capacity that exists, that in-flight requests fail in a documented way, and that the service recovers. For a multi-node model deployed with LeaderWorkerSet or another group scheduler, stop one member and confirm the whole group either recovers or is marked failed together, never left serving partial results. Ask the platform team to confirm how the chosen group controller behaves, because this depends on the scheduler and the version in use. Record the evidence in a form that survives a change review: the test plan with expected results, dated run logs, the pinned image digest and model file checksums, the rendered manifests, screenshots or exported outputs of the refused requests, and the pod events from each failure test. Keep the results with the release record. Auditors and certification bodies decide whether this evidence is sufficient for a given control, so confirm the required format with them early.