vLLM multi-node serving with KubeRay

How multi-node vLLM on KubeRay works for a model too large for one node

When a model needs more accelerators than one node holds, the serving layer has to span several machines while still presenting one endpoint to applications. On a Kubernetes platform you already run, the vLLM project's documented KubeRay path is one way to do this. Instead of logging into each node to start processes by hand, you declare a Ray cluster in YAML and let the KubeRay operator schedule the pods, set up their networking, restart them after failures and carry out blue-green deployments. The manual SSH scripts that multi-node setups often start with are replaced by resources the cluster reconciles.

Two KubeRay custom resources carry most of the design. A RayCluster describes the head and worker groups, their accelerator requests and the Ray runtime that vLLM uses for pipeline parallelism across nodes. A RayService wraps a cluster and manages how serving changes over time, including the blue-green rollout that lets a new model version come up beside the running one before traffic moves. Your team needs to decide who owns these manifests, how they reach the cluster through your normal delivery path, and which namespace and node pool each group lands on.

This page covers four things: the setup sequence, the isolation the deployment needs, how to test it before users arrive, and the failure points that appear once it runs. The isolation point deserves early attention. vLLM's security documentation says communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, so the Ray cluster must sit on a network that only its own nodes and approved management paths can reach. It also describes a Ray trust model in which driver environment variables propagate to workers.

Confirm the exact KubeRay operator version, the Ray and vLLM image tags and the accelerator device plugin your platform uses against the vLLM KubeRay guide and the KubeRay project before you write manifests. vLLM releases ship about every two weeks, so pin the version you test and plan how it moves forward. Ask your platform team which Kubernetes version is in use and whether it is still within the upstream support window, because the operator and the scheduler both depend on it.

The rest of the page assumes you already run Kubernetes with accelerator scheduling and can say who operates the cluster. If that is not yet true, the work is mostly platform preparation, and the vLLM serving design should wait until the cluster, its identity model and its network policy are settled.

What you need in place before deploying vLLM with KubeRay

Multi-node vLLM on KubeRay depends on six layers working together: the Kubernetes cluster, the operator that manages the Ray clusters, accelerator nodes, model artifacts, an isolated network and an upgrade path. Check each layer before writing manifests, because a gap in any one usually shows up later as a failed rollout or an exposed endpoint.

Kubernetes cluster version

Start with a Kubernetes cluster, which is open source under the Apache 2.0 license. The project maintains the three most recent minor releases, which as of September 2026 are 1.37, 1.36 and 1.35. Pin a minor version that stays inside its support window for the expected life of the deployment. The Kubernetes releases page lists end-of-life dates: 1.35 on February 28, 2027, 1.36 on June 28, 2027 and 1.37 on October 28, 2027.

KubeRay operator and version pins

KubeRay reads RayCluster and RayService resources and handles pod scheduling, networking, restarts and blue-green rollouts, which replaces hand-run scripts on each node. Confirm the KubeRay operator version you will install and the vLLM release you will pin against the vLLM Kubernetes and KubeRay documentation. vLLM ships new minor releases about every two weeks, so record both pins in your change process before the first deployment.

Accelerator nodes and drivers

Node pools need accelerators with a driver stack vLLM supports: NVIDIA CUDA, AMD ROCm or Intel XPU. Confirm the driver, container runtime and device plugin versions against the vLLM installation requirements for your chosen pin. Every node in a multi-node group should run the same stack. vLLM uses tensor parallelism within a node and pipeline parallelism across nodes, so the accelerator count per node shapes the whole layout.

Model artifact source

Every Ray worker must load the same model weights and tokenizer, so decide where artifacts live before rollout. Options include a shared volume, an object store your team controls or a cache populated ahead of time, and each needs a named owner for writes. Confirm the model license terms. Models must include a chat template, or you must supply one with the chat template option, before the chat completions endpoint works.

Isolated node-to-node network

The vLLM security documentation states that traffic between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, so nodes must sit on an isolated network. Plan a segment that carries Ray and PyTorch distributed traffic, and restrict which pods can reach the Ray ports. Keep user-facing endpoints on a separate path that passes through a reverse proxy with an allowlist.

Upgradable Kubernetes distribution

Choose a distribution you can upgrade one minor version at a time. The version skew policy requires kube-apiserver to move first, and kubelets may run up to three minor versions behind it. Confirm that the distribution is CNCF Certified Kubernetes conformant, that accelerator nodes can be drained without losing a serving replica and that etcd has a tested backup plan before any node pool changes.

How vLLM splits a model across nodes with Ray

The layout decision comes first. vLLM splits a model with tensor parallelism inside a node and pipeline parallelism across nodes. Tensor parallelism shards each layer's weights across GPUs that share fast local links, so it stays within one machine. Pipeline parallelism assigns consecutive groups of layers to different nodes, which tolerates the slower network between them. For multi-node runs, vLLM coordinates GPUs through Ray or through its own multiprocessing backend. On KubeRay, Ray is the path the vLLM KubeRay guide describes, so the layout plan should start from a Ray cluster. Confirm in the vLLM parallelism and scaling docs, for your chosen release, which tensor and pipeline combinations and backends your model supports.

Every pod in the Ray cluster must agree on the model and the parallel layout. The head node and each worker load the same weights, use the same tensor and pipeline sizes, and run the same vLLM version. If the product of the tensor and pipeline sizes does not match the number of GPUs the cluster actually exposes, startup can fail or the engine can stall while it waits for ranks that never join. Keep the layout in the RayCluster spec and the serving arguments together, and review them as one change so a scaling edit cannot quietly break the match.

This changes how you size pods. Each worker group is usually planned around a fixed number of GPUs per pod, so the GPU request in the pod template, the CPU and memory requests, and the shared memory volume all have to fit the chosen layout. Storage matters because every node reads model weights at startup. A shared volume, or identically provisioned volumes, for weights and the model cache keeps workers from starting with different files. Ask your platform team to confirm the GPU count per node pool, the cold-start read performance of the storage class, and whether adding a worker group changes the pipeline size and therefore the layout.

Mixture-of-experts models add a second option. vLLM supports data parallel deployment for these models, which runs independent replicas that each hold a full copy of the model instead of spanning one copy across nodes. The choice affects pod count, GPU requests and network traffic, so decide it before you write manifests. Two security points also apply to any Ray layout. vLLM's security docs state that traffic between nodes is insecure by default and must sit on an isolated network. Ray also passes driver environment variables to workers, so keep the driver environment minimal. Verify the final layout against your model and vLLM release before sizing the cluster.

Configuration sequence for a multi-node vLLM RayService

This sequence takes a team from an empty Kubernetes cluster to a multi-node vLLM service that answers requests through a single user-facing endpoint. Each step ends with a check, so problems surface on the step that caused them. Confirm each flag and resource field against the vLLM and KubeRay versions you pin before you start.

Install the KubeRay operator
The platform team installs the KubeRay operator into the Kubernetes cluster that will host the model, using the chart or manifests that the KubeRay project publishes, and records the operator version in the same change record as the rest of the stack. The operator watches for RayCluster and RayService resources and handles pod scheduling, networking, restarts and blue-green deployments, which replaces hand-run scripts on each node. Check: the operator pod is running, its custom resource definitions are registered, and it reconciles a small test RayCluster without errors before any model work begins.
Declare head and worker groups
The platform engineer writes the RayService manifest with a head group and one or more worker groups. Each worker group requests the accelerator resource your GPU or accelerator plugin exposes and uses node selectors and tolerations so that pods land only on the accelerator node pool. Anti-affinity rules keep replicas on separate nodes where the topology requires it. Check: all head and worker pods reach the Running state on accelerator nodes, and the Ray dashboard or ray status shows the expected number of nodes joined to the cluster.
Set the vLLM serve command
The engineer sets the vllm serve command in the head and worker container definitions and chooses parallelism arguments. Tensor parallelism is set to the number of accelerators within each node, and pipeline parallelism spreads the model across nodes. The vLLM parallelism documentation describes this split, and the flag names should be confirmed against the pinned release. Check: the Ray cluster reports every node healthy, the expected accelerators are allocated to the engine, and the engine finishes initialization without out-of-memory errors in the logs.
Configure model storage and chat template
The team places the model weights where every node can read the same files, either on shared storage or on a volume prepared identically on each node, and points the serve command at that path. If the model does not ship a chat template, the engineer supplies one with the chat template option, because the chat completions API requires it. Check: the model loads on every node, the worker logs show the same weights and template path, and a chat completions request reaches the engine without a template error.
Expose only the user-facing endpoint
The engineer creates a Service and an ingress or reverse proxy that forwards only the OpenAI-compatible paths meant for users, such as the v1 completion and chat routes. The vLLM API key option does not cover every endpoint, including metrics, health, pause, resume and abort paths, so the proxy allowlist and network policies must block those paths from users. Ray ports and the Ray dashboard stay on the isolated cluster network. Check: an allowed client gets a response on a v1 route, and requests to the blocked paths and Ray ports fail from the user network.
Roll out and send a test request
The engineer applies the RayService and watches its status until the serve application reports ready. For later changes, the blue-green upgrade strategy brings up a second cluster before traffic moves, and the old cluster stays available until the new one passes checks. The team then sends a test request through the service address using an approved client. Check: the request returns a completion through the user-facing endpoint, the metrics endpoint is reachable only from the monitoring network, and the previous cluster is retired only after this check passes.

How Kubernetes, Ray and vLLM objects map to each other

This table links each Kubernetes resource in a KubeRay deployment to its runtime role in Ray and vLLM, and names the team that normally edits it. Read it top to bottom to see how a manifest change reaches the running model, and use the owner column to plan review responsibilities before the build starts.

Object	Runtime role	Who edits it	What to watch
RayCluster or RayService resource	Declares the Ray head and worker groups in YAML. The KubeRay operator reconciles it into pods, and RayService adds serve configuration and blue/green rollout.	Platform team owns the template; model team edits serve configuration	Confirm which KubeRay version and CRD your cluster supports before writing manifests
Ray head pod	Runs the Ray head process, the cluster's control point, and hosts the vLLM OpenAI-compatible server	Platform team sets resources and image; model team sets vLLM arguments	Head restarts affect the whole cluster, so confirm its resource requests and health probes
Ray worker pods	Join the head and hold the remaining GPU shards of a model that spans nodes	Platform team sets replica count, GPU requests and node selectors	Every worker must reach the head on Ray ports; confirm network policy allows that traffic
vLLM OpenAI-compatible server	Runs on the head and exposes endpoints such as /v1/chat/completions to applications	Model team sets model name, chat template and API key options	Endpoints outside /v1 are not protected by the API key option; confirm the proxy allowlist
Parallelism settings	Tensor parallel size spans GPUs within a node; pipeline parallel size spans nodes. Values must match the GPUs the pods request	Model team chooses values; platform team confirms GPU capacity	Mismatched sizes leave workers idle or fail at startup, so test after every change
Kubernetes Service	Gives application clients one stable address for the serving endpoint, usually pointing at the head	Platform team owns the Service and ingress; application team consumes the address	Expose only the user-facing port, not Ray dashboard or internal ports
/metrics endpoint	Serves Prometheus-format request, throughput, latency and KV cache metrics from the vLLM API server	Platform team owns scraping and alert rules; model team reviews metric meaning	Confirm the scrape path is reachable from Prometheus and not exposed to users

Isolating Ray and PyTorch traffic in a multi-node vLLM deployment

Multi-node vLLM depends on Ray and PyTorch traffic between nodes, and vLLM's security documentation says that traffic is insecure by default. The cards below cover the controls that protect it: network placement, the Ray trust model, the API key and proxy layer that faces users, and how Kubernetes stores the credentials involved.

Isolate node-to-node traffic

vLLM's security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Ray and PyTorch distributed traffic therefore needs its own network segment, separate from user traffic and from other tenants. For KubeRay, check that head and worker pods reach each other only over the cluster network you designed, and that no Ray or PyTorch port is published through a Service or Ingress.

Driver environment propagates to workers

In Ray's trust model, environment variables set on the driver propagate to the workers it starts. Any secret, token or setting placed in the driver environment therefore reaches every node in the cluster. Keep the driver environment minimal, passing only the variables a run needs, and review the head and worker group pod specs in your RayCluster manifest so that credentials are not inherited by accident.

Denylist configuration file

vLLM's guidance recommends a denylist configuration file that names environment variables which must not propagate to Ray workers. Keep the file in the same repository as the RayCluster manifests so changes are reviewed. Test it by starting a worker and inspecting the environment it received. Confirm the file format and variable names against the vLLM release you deploy, since the list must match the version in use.

API key scope and its gaps

The --api-key option protects the /v1, /v2, /inference and /cohere endpoints. It does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health. Treat it as one layer only. List each uncovered route in your threat model, then decide whether health probes, Prometheus scraping and operator actions need network-level restriction in addition to the key.

Reverse proxy allowlist

vLLM's security documentation recommends a reverse proxy that forwards only the endpoints intended for users. Build the allowlist from the routes your applications call, such as the OpenAI-compatible completions and chat endpoints, and deny everything else by default. Keep health checks, metrics scraping and pause or abort routes on a separate internal path, then test by requesting each blocked route from outside the cluster.

Kubernetes Secrets in etcd

Kubernetes stores Secrets unencrypted in etcd by default, and the Ray head and worker pods may receive API keys and model credentials through them. Configure encryption at rest for etcd, restrict get, list and watch on Secrets with least-privilege RBAC, and consider an external store through the Secrets Store CSI Driver. Ask your platform team to confirm which option is in place before the cluster serves production traffic.

Autoscaling and blue/green upgrades with RayService

The KubeRay operator handles blue/green deployments for RayService, so the main decision is how traffic moves from the running Ray cluster to a new one when the vLLM release changes. During a cutover, the operator brings up a pending RayCluster next to the active one and switches the serving endpoint once the new cluster is ready. Your team must decide what counts as ready for a vLLM model: a healthy Ray head, all workers joined, and the model loaded and answering test requests. Also decide how long the old cluster stays up. Keeping it running gives a fast rollback, but both clusters hold accelerators while they overlap, which affects capacity planning. Confirm in your operator version which health conditions gate the switch and how the previous cluster is retired.

Autoscaling is a sizing decision before it is a configuration setting. Start by choosing the scale signal, such as request backlog or in-flight requests, and confirm which metrics your KubeRay and Ray Serve versions expose for it. Set a minimum worker count that covers baseline traffic and the cold-start gap, because a model replica that spans several nodes under pipeline parallelism must scale as a whole group, not one node at a time. Set the maximum from the accelerators you can actually obtain. A scale-up request stays pending when the node pool has no free GPUs, so check quota and node pool limits with your cloud or data center team. Cold-start time covers pulling the image, loading weights and warming the engine. Measure it on your own hardware and model before you choose the minimum.

Autoscaling and blue/green settings must be confirmed against both the KubeRay operator version and the vLLM release, because the behavior you tested on one pair may not hold on another. vLLM ships new minor releases about every two weeks (v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026), so an upgrade decision comes up often. A new minor release may change parallelism defaults or serving behavior, so every upgrade needs a test cycle on the same node topology before traffic moves. Include a load test at your expected peak, a check of the chat template and OpenAI-compatible endpoints your applications call, and a comparison of the metrics your dashboards depend on.

Treat the green cluster as a second Ray deployment with the same security obligations as the first. The vLLM security documentation states that node-to-node traffic in a multi-node deployment is insecure by default, so both clusters must sit on an isolated network with policies that allow only Ray and PyTorch distributed traffic between their own nodes. Overlapping clusters double the number of nodes that can reach each other, so review network policies before the new cluster starts, not after. Before you approve a release change, ask your implementation partner or platform team to document the cutover steps, the rollback trigger, the expected accelerator footprint during overlap and the test results for each vLLM version in the plan.

How to test a multi-node vLLM RayService before users connect

Each card covers one group of checks that should pass before the RayService receives user traffic. The groups run in order: infrastructure, model behavior, load and failover, observability and an upgrade rehearsal. Together they show whether the cluster is isolated, the model answers correctly, the service recovers from failure and the team can change versions safely.

Infrastructure and network isolation

Confirm that every Ray pod is scheduled on the GPU nodes you intended, that the RayService reports a healthy status and that Ray head and worker traffic is reachable only inside the isolated network. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, so test from outside the cluster and from a pod in an unrelated namespace. Both attempts should fail. Save the network policy objects and test commands so the result can be repeated after each change.

Completions and chat behavior

Send a request to /v1/completions and a request to /v1/chat/completions, then compare the responses with outputs agreed in advance with the model owner. For chat, confirm that the model ships a chat template or that one was supplied with the chat template option, since the chat completions API needs one to serve. Repeat the same prompts through the external route users will take, so the check covers any gateway or allowlist in front of vLLM.

Load and worker failover

Under steady request load, delete one worker pod and observe the result. Confirm that KubeRay restarts it, that the Ray cluster accepts it back and that requests return to normal without manual steps. Record how many requests fail during the interruption and whether client retries are safe for your workload. Repeat the test with the head pod if your design depends on it. The results show whether your client timeouts and retry settings match how long recovery takes in your environment.

Metrics and KV cache visibility

Scrape /metrics from the vLLM API server using your Prometheus configuration and confirm that request counts, token throughput, latency histograms and KV cache usage all appear. The vLLM API key option does not protect /metrics, so decide who can reach that path and block it at the proxy or network layer if it should stay internal. Check dashboard and alert queries against the metric names in your installed vLLM version, because names can change between releases.

Blue/green switch and rollback

Rehearse an upgrade on a copy of the production manifest. Deploy the new RayService alongside the current one, send test traffic to it, shift the route and then roll back to the previous version. Time each step and note which requests fail during the switch. Include model weights, cache volumes and the vLLM image tag in the rehearsal, because a rollback only helps if the previous version can still start with the storage it depends on.

Common failures in KubeRay multi-node vLLM and how to fix them

This table lists the failures teams most often hit when running multi-node vLLM on KubeRay, with the likely cause, the fix and the first check to run. Read each row left to right and run the first check before you change any configuration. Many failures trace back to network isolation, scheduling labels, shared model storage or the exposed API surface, so verify those areas before any cutover.

Failure	Cause and fix	Check first
Workers cannot join the Ray head	Network policy blocks head-to-worker traffic, or the ports in the manifest do not match what Ray uses. Allow Ray traffic between cluster pods only.	Worker pod logs for connection errors, then the NetworkPolicy and port settings on head and workers.
Pods stay pending	Accelerator requests do not match node labels or taints, so the scheduler finds no eligible node. Align requests, selectors and tolerations with the actual node pool.	Scheduling events on the pending pod, using kubectl describe.
Model loading fails on some nodes	Model files are not on shared storage, or nodes run different vLLM or library versions. Mount one model store on all workers and pin a single image version.	The image tag and model path mounted in each worker pod.
Chat completions fail	The model has no chat template and none was set with --chat-template. Supply a template that matches the model and set it in the server arguments.	One request to /v1/chat/completions, then the error message the server returns.
Internal endpoints are reachable from users	--api-key was the only control, and it does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health. Place a reverse proxy that allowlists only user-facing endpoints.	Request each operational endpoint from outside the cluster network and confirm it is refused.
Driver environment leaks to workers	Ray propagates driver environment variables to workers, so secrets set on the driver reach every node. Keep the driver environment minimal and apply a denylist configuration file.	Environment variables in a worker pod, compared with those set on the driver.
Blue/green switch drops requests	Traffic moves to the new RayService before its workers are ready, or requests in flight end when old replicas stop. Gate the cutover on readiness and confirm how the old cluster drains.	RayService status and whether the new cluster passes health checks before traffic moves.

Questions buyers ask before hiring help with KubeRay and vLLM

How much does it cost to run vLLM across multiple nodes?

vLLM has no paid edition. It is open source under Apache-2.0, so the cost comes from four places: accelerator capacity (GPUs or other accelerators in each node), surrounding infrastructure (node-to-node networking, the Kubernetes cluster, storage, load balancing and monitoring), implementation work, and the staff who operate the stack after launch. Implementation is largely one-time. Accelerator capacity, infrastructure and operations recur every month. Costs rise with node count, model size, network design, identity and proxy requirements, and how often you upgrade. Ask any quote to show each line separately, including who pays for each item, so you can compare it with your own team's cost.

How long does a KubeRay and vLLM implementation take?

Duration depends on the scope agreed in writing, and no single timeline fits every environment. The main drivers are the number of nodes and accelerator types, the size of the models and whether their weights must move into your own registry, the network design between nodes, identity and proxy requirements, whether a suitable Kubernetes cluster already exists, and how long your approvals and hardware deliveries take. DeployOpen works to a fixed scope with agreed deliverables and timelines before work starts. Ask for milestones, the dependencies on your teams, and what changes if hardware arrives late.

What does our team need to provide for this engagement?

Plan to provide access and decisions as well as people. You will need a Kubernetes cluster or an agreed plan to build one, accelerator nodes with working drivers and a container runtime, network segments and a firewall change process, DNS names and TLS certificates, an identity provider for user access to the proxy, and the model artifacts with their licenses. You also need named staff for approvals, a technical owner who can make decisions during the build, and an on-call contact for cutover. DeployOpen implements inside your cloud accounts, data centers or air-gapped networks, so access to those environments is required from the start.

Can our internal platform team set up vLLM on KubeRay alone?

Yes, a team with solid Kubernetes, GPU operations and networking experience can build it, because both vLLM and KubeRay are open source and documented. The difficult parts are the ones teams often underestimate. vLLM's security documentation states that communication between nodes is insecure by default, with no authorization or encryption, so nodes must sit on an isolated network. The Ray trust model passes driver environment variables to workers, and the --api-key option does not cover every endpoint, so a reverse proxy is needed. Before deciding, list who will own networking, proxy and identity, GPU drivers, metrics, upgrades and on-call, and check whether each owner has the time and skills.

What does the KubeRay operator manage for a vLLM deployment?

The operator reconciles a declared RayCluster or RayService resource. According to the vLLM KubeRay documentation, it handles pod scheduling, networking, restarts and blue-green deployments, which replaces manual SSH scripts on every node. Your team still owns the decisions the operator cannot make: sizing head and worker groups, GPU resource requests, autoscaling bounds, model loading and caching, health checks, and the criteria for promoting a new version. Operator automation reduces manual steps, but it does not verify that a deployment serves correct output, so acceptance tests should be part of the scope.

Why does network isolation matter for multi-node vLLM?

Ray and PyTorch distributed traffic between nodes carries no authentication or encryption by default, so anyone who can reach a node on that network can interact with the cluster. vLLM's security documentation recommends placing nodes on an isolated network, limiting access to the Ray cluster, running drivers with a minimal environment, and using a denylist configuration file to block sensitive variables from propagating to workers. Network design is one of the largest effort drivers because it affects routing, firewall rules, certificates and how administrators reach the cluster. Ask the scope to include tests from outside the isolated segment that confirm unauthorized access fails.

What should we request in a written scope before hiring help?

Ask for a scope that names each deliverable and its acceptance test. Useful items include the RayCluster and RayService manifests, the network segmentation design, the reverse proxy allowlist for user-facing endpoints, metrics collection for the /metrics endpoint, backup and restore steps for configuration and state, and an upgrade runbook. Also request the node count, accelerator types, model list and sizes, the Kubernetes distribution and version, identity and proxy requirements, the split of responsibilities between your team and the implementer, explicit exclusions, change control rules, and the support terms after handover. A written scope makes quotes comparable and makes it clear what finished work means.

How do upgrades affect cost and effort after go-live?

Upgrades are a recurring cost because both vLLM and Kubernetes keep moving. According to vLLM's GitHub releases, the project ships new minor releases about every two weeks, and Kubernetes publishes roughly three minor releases per year, with each minor version receiving patch support for about one year. Each upgrade needs staff time for testing, compatibility checks against your model and API clients, and a blue-green rollout that may temporarily need extra accelerator capacity. Kubernetes version skew rules also set the order in which control plane and node components can change. Ask for the upgrade cadence, the test environment you will need, and who runs the work.

How DeployOpen scopes, builds and supports KubeRay vLLM deployments

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including vLLM multi-node serving on Ray clusters managed by KubeRay. Most engagements are fixed-scope, so the clusters, nodes, networks and endpoints involved are agreed in writing before work starts. Ongoing support, upgrades and testing are available as a retainer.

How we scope

The written scope names the Ray head and worker nodes, GPU pools and the network segment they share.
We confirm which vLLM endpoints users reach and which stay internal, such as metrics and health checks.
Model compatibility, chat template needs and accelerator capacity are checked before any cluster resources are defined.

How we build

We write RayCluster and RayService manifests as declarative code, reviewed and stored in your version control system.
We isolate node-to-node Ray and PyTorch traffic on a private network, since vLLM's docs say inter-node traffic is insecure by default.
We set a minimal driver environment and a denylist file, so fewer driver variables reach Ray workers.

How we hand over

Handover includes the manifests, network and allowlist configuration, runbooks for scaling and restarts, and the blue/green upgrade procedure.
Operators rehearse a node loss and a blue/green cutover in a test environment before go-live.
Architecture notes record the trust model, the Kubernetes and Ray versions in use, and the decisions behind them.

How we support

Retainers can cover Kubernetes version skew planning, etcd backups and encryption at rest for Secrets.
Upgrade work follows vLLM's roughly two-week minor release cadence, tested against your cluster before rollout.
Security testing of exposed endpoints can be added to a retainer, so allowlist changes are retested after upgrades.