When a single node, one node with several GPUs, or multiple nodes fits vLLM
Choose the smallest topology that holds the model in GPU memory: one GPU if the model fits on one card, tensor parallelism inside one node if it needs several GPUs in the same server, and tensor plus pipeline parallelism across nodes only when it does not fit on a single node. The vLLM parallelism and scaling docs give this rule directly, and they add that distributed inference is probably unnecessary when a model fits on one GPU. vLLM is Apache-2.0 licensed with no paid edition, so the choice is driven by hardware, interconnect and the operating effort of each layout, not by license cost.
Work through the decision in order. First, confirm the model's memory footprint at your target context length and batch size, then count the GPUs and nodes needed to hold it. The docs suggest setting tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes. Pipeline parallelism also accepts GPU counts that do not divide the model evenly. Ask your hardware vendor and the vLLM release notes for your chosen version which parallel sizes it supports for your model architecture, and test on your own hardware before you size production.
Multi-node serving adds three costs that single-node serving does not. Tensor parallelism needs fast internode communication, and the docs point to high-speed adapters such as InfiniBand, so confirm the network fabric before buying nodes. Multi-node runs use Ray by default, which is an optional dependency you must install explicitly. Node-to-node traffic is also unencrypted by default, so the docs say to set VLLM_HOST_IP to a private segment that untrusted parties cannot reach. Plan network isolation and on-call coverage for those nodes before you commit to a cross-node layout.
Three topologies and what each one asks of the team
vLLM offers a short ladder of layouts, and each step up adds hardware, networking and operating work. The right starting point is the smallest layout that holds the model with room for the KV cache. The cards below set out the condition for each option and the cost it brings with it.
One GPU, no distributed inference
Use a single GPU when the model and its serving overhead fit in one device. The vLLM docs say distributed inference is then probably unnecessary, so you avoid the extra moving parts entirely. The team's main cost is capacity planning: one GPU means one failure domain, so check how you will restart the server, route traffic during maintenance and keep an upgrade path to a larger layout before users outgrow it. Confirm memory headroom for your context lengths with a load test in your own environment.
Several GPUs in one node
When the model is too large for one GPU but fits within one node, vLLM recommends tensor parallelism across the GPUs in that node. Set tensor_parallel_size to the number of GPUs per node. The operating cost is mostly hardware and driver consistency: every GPU in the group must be healthy, and a single failed card takes the whole replica down. Ask your hardware vendor for the interconnect between GPUs in the node, since tensor parallelism depends on fast communication between them.
Multiple nodes with tensor and pipeline parallelism
Combine tensor parallelism with pipeline parallelism only when the model cannot fit on a single node. Set tensor_parallel_size to GPUs per node and pipeline_parallel_size to the number of nodes. The operating cost is the largest of the three: Ray is the default multi-node runtime and must be installed explicitly, efficient tensor parallelism needs fast internode networking such as InfiniBand, and node-to-node traffic is unencrypted by default. Plan an isolated network segment and confirm the network budget before you buy nodes.
Data parallel for mixture-of-experts models
vLLM supports data parallel deployment for mixture-of-experts models, which runs independent copies of the model to spread request load. Consider it when your model architecture suits it and the goal is serving more concurrent requests rather than fitting a larger model. Each replica needs its own full copy of the weights, so memory per replica still governs the design. Confirm with the vLLM docs and your model's documentation that data parallel is supported for your exact model and version before planning hardware around it.
Single-node and multi-node vLLM side by side
This table compares serving a model on one GPU, on several GPUs in one node, and across nodes with vLLM. Read each row as the question to settle before buying hardware: the first rows decide whether you need distributed serving at all, and the later rows show what each choice adds in operations and risk.
| Factor | Single GPU or one node (tensor parallelism) | Multi-node (tensor plus pipeline parallelism) |
|---|---|---|
| When it is needed | Use one GPU if the model fits. Use tensor parallelism across GPUs in one node if it does not fit on one GPU but fits on one node. | Needed only when the model is too large for a single node, per the vLLM parallelism docs. |
| Parallelism settings | tensor_parallel_size is set to the number of GPUs in the node. pipeline_parallel_size stays at 1. | tensor_parallel_size is set to the number of GPUs per node and pipeline_parallel_size to the number of nodes. |
| Interconnect requirement | NVLink or PCIe within the node carries the tensor parallel traffic. Confirm the link speed of your GPU topology. | Efficient tensor parallelism needs fast internode communication, preferably high-speed adapters such as InfiniBand. Plan the network before hardware. |
| Runtime | vLLM runs its own worker processes on the node. No Ray install is required for this layout. | Ray is the default runtime for multi-node runs and must be installed explicitly. vLLM also offers its own multiprocessing backend. |
| Node-to-node traffic security | Not applicable. All traffic stays inside one host. | Node-to-node traffic is unencrypted and unauthenticated by default. Set VLLM_HOST_IP to a private segment that untrusted parties cannot reach. |
| Failure domain | One host failure takes the whole model offline. Plan a second replica if you need availability. | A failure on any node can break the distributed group. Losing one node affects the model served across all nodes. |
| Upgrade and restart scope | Restarting one server restarts the model. Schedule maintenance on the single host. | Restarting or upgrading the cluster means coordinating every node and the Ray runtime together. Test the sequence before production. |
| Staff skills needed | Linux, GPU drivers and the vLLM server. Confirm the skill set your team already has. | GPU networking, Ray cluster operations, private network design and multi-host troubleshooting. Ask a partner to scope these skills explicitly. |
Why fast interconnect and Ray decide whether multi-node is worth it
Multi-node serving is only worth attempting when your network can carry the traffic that tensor parallelism generates. The vLLM parallelism docs say efficient tensor parallelism requires fast internode communication, preferably through high-speed network adapters such as InfiniBand. That makes tensor parallelism across nodes the expensive part of the design, because every layer's activations move between GPUs on different machines. Pipeline parallelism, which vLLM uses across nodes, sends less data between stages, but it is still a dependency on the link between machines. Before you commit, ask your hardware vendor and network team to confirm the adapter type, link bandwidth and latency between the nodes you plan to use, and whether that fabric is dedicated to the inference cluster or shared with other workloads. If the answers are weak, a larger single node with more GPUs will usually be the simpler route.
The runtime is the second decision. Ray is the default runtime for multi-node vLLM inference, but it is an optional dependency that you must install explicitly, so it does not arrive with a plain vLLM install. vLLM can also use its own multiprocessing backend for multi-node runs, which changes what your operators have to install, monitor and upgrade. Confirm which backend your chosen vLLM release supports for your topology, the Ray version it expects, and who owns the Ray cluster lifecycle inside your platform team. Plan for the operational load of a second distributed system, including its own health checks, logs and upgrade windows, and test a node failure in a non-production environment before you rely on the setup.
Treat the network and runtime checks as gates, not paperwork. If the fabric is not confirmed as fast and private, or if no one on your team will own the Ray or multiprocessing layer, multi-node serving adds cost and risk without a matching gain. Ask for a written statement of the interconnect specification from your vendor and a short test that measures throughput with and without the second node. That result, not the topology diagram, should decide whether the cross-node design goes into production. Keep the private network requirement from the security section in the same review, since the traffic these checks validate is also the traffic that must stay isolated.
How unencrypted node-to-node traffic changes the network design
The vLLM security docs state that all communication between nodes in a multi-node deployment is insecure by default, with no authorization and no encryption. That changes the network design before any model is loaded. The nodes that run tensor and pipeline parallel workers must sit on an isolated network segment that no untrusted party can reach. Set VLLM_HOST_IP to the address of that private segment so vLLM binds its inter-node traffic there rather than to a routable interface. Ask your network team to confirm which interface the address maps to on every node, because a mismatch often shows up as workers that fail to join rather than as an obvious security error.
Treat the serving endpoint and the cluster network as two separate exposure surfaces. The --api-key option protects the /v1, /v2, /inference and /cohere routes, but it does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health. The docs warn against relying on the key alone. Place a reverse proxy in front of the server that allowlists only the endpoints meant for users, and confirm that /metrics and /health are reachable only from your monitoring and load balancer addresses. Test this from outside the network, not only from inside the cluster, and record the result as evidence for your architecture review.
Multi-node serving also means that the Ray runtime, which is the default for multi-node vLLM inference, carries traffic between machines. Confirm with your platform team whether Ray's dashboard and worker ports are reachable outside the isolated segment, and whether the same private network is used for any fast interconnect such as InfiniBand. If a parallelism decision would require sending activations across a segment you cannot isolate, that is a reason to stay on one node, not a reason to open the network. Document the segment, the allowed endpoints and the owner of each firewall rule before the first production run.
What running each topology asks of your operations team
How often do vLLM upgrades arrive, and how should we handle them?
vLLM ships new minor releases about every two weeks, so the upgrade load is recurring rather than occasional. Recent minor releases in the vLLM GitHub releases history were v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026. Pin an exact version, run your own prompts and load tests against each candidate, and promote it on a schedule your team can staff. Multi-node upgrades touch every node and the Ray runtime together, so they take longer to test and roll back than a single-node change. Ask whether your team will own this cadence in-house or buy it as a retainer, since the work recurs for as long as the service runs.
How do we monitor a vLLM deployment with Prometheus?
The vLLM API server exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use, with configurable histogram buckets. Scraping it is inexpensive to set up, but the effort lies in dashboards, alert thresholds and tuning those thresholds per model and per GPU type. In a multi-node run, you must scrape every node that holds a shard of the model and correlate them, because a healthy head node does not prove that workers are healthy. Keep /metrics on the private network, since the vLLM security docs say it is not covered by --api-key. Confirm with your monitoring vendor how much series volume a multi-node layout adds before you size storage.
What fails when one node goes down in a multi-node vLLM run?
A single-node run has one failure domain: a GPU error or process crash takes the model offline until it restarts. A multi-node run with tensor and pipeline parallelism spreads one model across several machines, so the loss of one node usually stops inference for the whole model, not just the requests on that machine. Recovery means restarting the Ray cluster and reloading weights on every node, which adds time and makes capacity planning harder. Your runbook should cover node health checks, automated restart policy, and how long a full model reload takes on your storage. Ask your platform team which of these steps they can automate and which need an operator on call.
How do KubeRay and LeaderWorkerSet fit a multi-node vLLM deployment?
The vLLM Kubernetes docs list KubeRay and LeaderWorkerSet among the deployment options, next to native manifests, Helm and the vLLM production-stack. Ray is the default runtime for multi-node vLLM inference, so KubeRay is a natural fit when you want Kubernetes to manage Ray clusters, while LeaderWorkerSet groups a leader pod and its worker pods as one unit. Either option adds a control plane you must run, upgrade and secure. Kubernetes itself needs an etcd plan, backups and a version upgrade cadence, and the Kubernetes release cycle runs about three times per year. Single-node deployments can often stay on a simpler manifest or Helm chart. Confirm with your platform team whether you already run Kubernetes at the required maturity before adding this layer.
What skills does the team need to run multi-node vLLM?
A single-node deployment needs GPU driver management, container operations, and familiarity with the OpenAI-compatible API and its metrics. Multi-node serving adds the skills to operate Ray, configure fast interconnects such as InfiniBand, set VLLM_HOST_IP on an isolated network, and debug distributed failures across machines. Your team also needs network engineering to keep node-to-node traffic private, because the vLLM security docs say that traffic is unencrypted and insecure by default. Kubernetes operation, if you choose that path, adds etcd and RBAC skills. Skill gaps are usually the largest hidden cost, so ask whether you will hire, train or contract for these areas.
Who pays for ongoing vLLM operations, and what recurs?
vLLM has no paid edition, so the recurring costs come from GPU or accelerator capacity, the surrounding infrastructure, implementation and the people who operate it. Multi-node runs multiply the node count, so hardware, networking and idle capacity for failover all grow with the topology. Recurring work includes upgrade testing, monitoring tuning, capacity reviews and on-call coverage. One-time work includes the initial build, network isolation and the runbook. Ask any implementation partner for a written scope that separates build, handover and ongoing support, because DeployOpen works on fixed scopes agreed in writing, with ongoing support and upgrades available as retainers.
Can a small team run vLLM across nodes without outside help?
Whether a small team can run vLLM across nodes depends on its existing GPU, networking and Kubernetes experience, not on the vLLM license, which is Apache-2.0. A team that already operates GPU clusters with fast interconnects and a secured private network can take this on, but it should plan for upgrades every two weeks and a Ray-based failure domain. A team new to these areas should consider a single node for as long as the model fits, then add nodes only when the model requires them. If you engage outside help, ask for a fixed scope covering build, hardening, documentation and handover, and a retainer for ongoing upgrades if you need it.
Upgrades, monitoring and skills for each topology
This table compares what it takes to run vLLM on one GPU, several GPUs in one node, and multiple nodes with a Ray or multiprocessing runtime. Read each row as a question to answer before you pick a topology, since the operating burden grows with each step up in parallelism.
| Operating area | Single GPU or single node | Multi-node with tensor and pipeline parallelism |
|---|---|---|
| Release pinning | Pin one vLLM minor version and one CUDA or ROCm stack, then test before each upgrade. vLLM ships new minor releases about every two weeks, so pinning is a deliberate choice. | Pin vLLM, Ray or the multiprocessing backend, and driver versions identically on every node. A mismatch between nodes can fail at startup or during inference. |
| Rollout testing | Run a staging endpoint with the same model weights and request mix, then compare latency and output quality before switching traffic. | Stage on a cluster with the same node count and interconnect. A single-node test does not reproduce cross-node communication behavior or failure modes. |
| Metrics collection | Scrape the /metrics endpoint for request counts, token throughput, latency histograms and KV cache use. One scrape target is usually enough. | Scrape each node and the head process, and confirm that pipeline stage metrics line up. Ask the vendor or community docs which series are emitted per rank. |
| Network hardening | Keep the API behind a reverse proxy that allowlists only user-facing paths, because --api-key does not cover /metrics or /health. | Place all nodes on an isolated private segment and set VLLM_HOST_IP to that segment. vLLM multi-node traffic is unencrypted and insecure by default, so untrusted parties must not reach it. |
| Runtime management | No cluster runtime is needed beyond the serving process and its container or service manager. | Ray is the default multi-node runtime and must be installed explicitly. Plan who restarts failed workers and how Ray cluster health is monitored. |
| Hardware skills | Team needs GPU driver, memory sizing and model quantization skills to fit the model on one card or node. | Team needs InfiniBand or other high-speed interconnect skills, since efficient tensor parallelism requires fast internode communication. |
| Recurring cost drivers | Accelerator capacity for one node, plus the people who patch and monitor it. vLLM has no paid edition, so license fees are not part of the recurring cost. | Accelerator capacity across all nodes, high-speed network hardware, isolated network infrastructure and on-call coverage for a more complex failure surface. Ask any implementation partner to price each separately in writing. |
How to move from one GPU to multi-node without rework
This sequence grows a vLLM deployment one parallelism layer at a time, so each stage is proven before the next adds network and operational risk. The output is a multi-node cluster that serves the same OpenAI-compatible endpoints the applications already call.
Confirm the model fits on one GPU
The platform engineer loads the target model on a single accelerator at the context length and batch size the application needs, and records the memory headroom left for the KV cache. The vLLM parallelism docs say that a model which fits on one GPU probably does not need distributed inference, so this check decides whether any further steps are necessary. Exit check: the model starts, serves a test prompt and holds the expected latency under a representative load. Confirm in your own environment which quantization format, context length and GPU model you will standardize on, since these change the memory math.
Size tensor parallelism within one node
If the model does not fit on one GPU but fits on one node with several GPUs, the platform engineer sets tensor parallelism to the number of GPUs in that node and restarts the server. The vLLM docs recommend tensor parallelism for this case, and the GPU-to-GPU links inside a single node are the fastest path the deployment will use. Exit check: the model loads across all GPUs in the node, memory use stays within the planned margin on each device and throughput is recorded as the baseline for later comparison. Confirm the interconnect type inside your node, such as NVLink or PCIe, with your hardware vendor before you rely on this baseline.
Validate the OpenAI-compatible endpoints
The application team points a staging copy of its client at the vLLM server and exercises the endpoints it actually uses, such as chat completions, completions, embeddings or responses. vLLM exposes these routes under the OpenAI-compatible server, so the client code should not change. Exit check: the client passes its regression tests, response formats match what downstream code parses and the Prometheus metrics at /metrics are scraped by your monitoring stack. Confirm which endpoints you expose to users and which stay internal, because the vLLM security docs say the --api-key option does not cover every route.
Put an allowlisting reverse proxy in front
The security team places a reverse proxy in front of the vLLM server that forwards only the endpoints meant for users and blocks operational routes such as /pause, /resume, /abort_requests, /metrics and /health from untrusted callers. The vLLM security docs recommend this pattern and warn against relying on --api-key alone. Exit check: a test from outside the permitted network reaches only the allowlisted paths, and the proxy logs show denied requests. Confirm the proxy's TLS termination and authentication design with your security team, since vLLM does not provide that layer on its own.
Stand up the private network and Ray runtime
Before any cross-node run, the infrastructure team provisions a dedicated private network segment for node-to-node traffic and confirms that no untrusted party can reach it. The platform engineer then installs Ray explicitly, since the vLLM docs say it is the default multi-node runtime and an optional dependency. The vLLM docs also say to set VLLM_HOST_IP to an address on that private segment. Exit check: every node reaches every other node on the private segment, Ray reports all nodes as joined and the segment is verified as unreachable from user networks. Confirm the isolation design with your network team, because the vLLM security docs say node-to-node communication is insecure by default.
Measure the interconnect before adding nodes
The platform engineer benchmarks the inter-node network with the team's chosen tooling and compares the result with the throughput needs of the tensor-parallel layout. The vLLM docs say efficient tensor parallelism needs fast internode communication, preferably through high-speed adapters such as InfiniBand. Exit check: the measured bandwidth and latency meet the target agreed by the platform and performance owners. If the network does not meet it, the team should stop here and consider fewer GPUs per model or a different model size rather than pushing tensor parallelism across a slow link. Confirm the adapter type and fabric design with your hardware and network vendors.
Add pipeline parallelism across nodes
Once the network checks pass, the platform engineer sets tensor_parallel_size to the GPUs per node and pipeline_parallel_size to the number of nodes, following the vLLM rule of thumb for models too large for one node. The docs note that pipeline parallelism also handles GPU counts that do not divide the model evenly. Exit check: the cluster serves the same test suite used in the single-node stage, the endpoints return the same response shapes and the team has a documented rollback to the single-node deployment. Confirm the exact flags and the Ray launch procedure against the vLLM release your team has pinned, since vLLM ships minor releases about every two weeks.
Repeat the endpoint and security tests on the cluster
The security and application teams rerun the endpoint regression suite, the proxy allowlist test and a sample of production-like load against the multi-node service before cutover. This confirms that the new topology did not change what users can reach or how responses look. Exit check: all tests pass on the cluster, the rollback path has been exercised and the operations owner signs off in writing. Confirm the test scope and sign-off roles in your change process, and plan how the cluster will be patched and upgraded, since the same version pinning applies to every node.
Common questions about vLLM topology and effort
Is a single GPU enough to serve my model with vLLM?
If the model fits on one GPU with room left for the KV cache, a single GPU is usually enough, and the vLLM parallelism docs say distributed inference is probably unnecessary in that case. The deciding inputs are the model's weight size at the precision you plan to run, the context lengths your users need, and the concurrency you expect. A single GPU avoids inter-GPU communication entirely, so it is the simplest topology to operate. Ask your engineers to load-test one GPU against your real prompt mix before adding hardware, because memory for the KV cache often decides the answer more than the weights do.
When should I move from one node with several GPUs to multiple nodes?
Move to multiple nodes only when the model cannot fit in the GPU memory of one node. The vLLM docs advise using tensor parallelism inside a node when the model is too large for one GPU but fits on one node with several GPUs. Crossing nodes adds a second layer of complexity, so the usual trigger is a model or context size that a single node cannot hold, not a wish for more throughput. Throughput needs can often be met by running more replicas of a model that already fits.
What drives the cost of multi-node GPU hardware and networking?
Most of the cost comes from accelerator capacity, which you buy or rent by the node, and from the network between nodes. The vLLM docs say efficient tensor parallelism needs fast internode communication, preferably through high-speed adapters such as InfiniBand, so a node pair connected only by standard Ethernet can slow the model down. Costs that recur include cloud GPU hours or data center power and support contracts. Costs that are mostly one-time include cluster design, installation and the first tuning cycle. When you request a quote, ask for each node's GPU count and memory, the interconnect specification and whether the price covers spares, support and monitoring.
How long does a vLLM setup take, and what changes the timeline?
No duration is established in the sources for this page, so treat any number you hear as a claim to verify against your own scope. The timeline depends on whether you are deploying a single GPU or a multi-node cluster, whether the network and hardware already exist, how many models and endpoints are in scope, and how much integration with your identity, logging and gateway layers is required. Multi-node work also adds cluster setup and validation of the interconnect. Ask for a written scope and a milestone plan before work starts, since fixed-scope engagements agree deliverables and timelines in writing.
Can my own team run a multi-node vLLM deployment without outside help?
It is possible for a team with strong Linux, GPU driver and networking skills, because vLLM is Apache-2.0 licensed with no paid edition. The harder part is operations: multi-node serving uses Ray by default, which must be installed explicitly, and the team must keep the runtime, the interconnect and the model versions aligned. vLLM ships new minor releases about every two weeks, so someone must own upgrades and regression testing. Ask yourself whether you have on-call coverage for GPU failures and whether you can staff this alongside your other platforms. A fixed-scope engagement or a retainer can cover part of that work, but the staffing decision stays with you.
What network isolation does multi-node vLLM need for security?
Node-to-node traffic in a multi-node vLLM deployment is unencrypted and unauthenticated by default, so the nodes must sit on an isolated network. The vLLM docs say to set VLLM_HOST_IP to a private network segment and to make sure untrusted parties cannot reach that network. Apply the same discipline to the API: the --api-key option does not protect every endpoint, so place a reverse proxy in front that allowlists only the routes users need. Your security team should review segmentation, firewall rules and who can reach the metrics and health endpoints before go-live.
What do I need to supply before a vLLM engagement can be scoped?
Prepare the model list with weight size and target precision, expected context lengths and concurrency, the hardware you own or plan to rent, and the network between nodes. Also supply your target environment, whether cloud accounts, a data center or an air-gapped network, plus the identity provider, logging stack and any Kubernetes platform you run. The vLLM docs list several Kubernetes deployment options, including Helm, KubeRay and LeaderWorkerSet, so state which one you already operate. DeployOpen plans and implements inside your own infrastructure, so access rules, change windows and named owners on your side should be written into the scope.
What to confirm with your hardware and security teams before you choose
Start with the memory footprint of the model you plan to serve measured against the GPU memory you actually have. The vLLM docs advise adding GPUs and nodes until there is enough GPU memory for the model, so the first figure to get is the memory the weights and the KV cache need at your target context length and concurrency, not the parameter count on a model card. Ask your platform engineers to run a sizing test on the hardware you will use, and ask the model publisher or your vendor for the precision the weights ship in. If one node covers the requirement, the multi-node question never arises, and that is the cheapest outcome to operate.
Next, confirm the GPU vendor and driver stack on every node. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, so the answer may differ by hardware pool. Check the exact driver, CUDA or ROCm version and the vLLM release you intend to pin against the vLLM installation guide, and do not mix GPU families in one tensor-parallel group without testing it first. Also confirm the interconnect between nodes. The vLLM docs say efficient tensor parallelism needs fast internode communication, preferably high-speed adapters such as InfiniBand. Ask the hardware team for measured bandwidth and latency between the nodes, and treat ordinary Ethernet as a figure to benchmark, not an assumption.
Finally, settle the isolation of the node network and how the API is exposed. Multi-node traffic is unencrypted by default, so the node network must sit on a private segment that untrusted parties cannot reach, and VLLM_HOST_IP should point at that segment. Ask your security team to verify the segmentation with a test from outside it. Then decide which endpoints users reach. The --api-key option does not cover every route, so the docs recommend a reverse proxy that allowlists only user-facing endpoints. Ask for written evidence of both controls before go-live, because a multi-node cluster on a flat network is a finding waiting to be written.
How DeployOpen scopes and delivers vLLM deployments
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For vLLM, that means deciding with you whether a model needs one GPU, one node or several nodes, then building and handing over the deployment in writing against an agreed scope.
How we scope
- Confirm the model, GPU types, GPUs per node and whether the network between nodes is high-speed.
- Record the deployment shape (single GPU, tensor parallel in one node, or tensor plus pipeline across nodes) in the written scope.
- Agree whether the isolated network for node-to-node traffic is in scope or provided by your team.
How we build
- Install and configure vLLM with the parallelism settings agreed in scope, and document each setting and its reason.
- Set up Ray for multi-node runs only when the scope requires it, with the runtime pinned and tested.
- Place API access behind a reverse proxy that allowlists only the endpoints meant for users, and test that internal endpoints are unreachable.
How we hand over
- Deliver runbooks covering startup order, scaling steps,
VLLM_HOST_IPconfiguration and recovery from a failed node. - Provide evidence of network isolation, endpoint exposure tests and metrics collection from the
/metricsendpoint. - Walk your platform and security teams through the deployment and agree the upgrade plan for vLLM releases.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the build.
- Retainers can cover vLLM version upgrades, GPU driver changes and re-testing after infrastructure changes.
- Ask for the retainer scope in writing, including who handles node failures and how urgent requests are routed.
