vLLM on AMD ROCm GPUs

Running vLLM on AMD ROCm GPUs: what to decide first

Infrastructure teams with AMD accelerators can run vLLM as an OpenAI-compatible serving engine on hardware they control, but the important decisions come before any install. vLLM supports ROCm as one of several GPU targets, alongside NVIDIA CUDA and Intel XPU. The server started with vllm serve exposes the endpoints that applications written for the OpenAI API already call, so the choice is mainly about hardware and software compatibility. Before any deployment, four things must agree: the accelerator model, the ROCm version, the Python version and the install path. If one is chosen in isolation, the mismatch usually surfaces at the first model load, which is the most expensive point to diagnose it.

Start by recording the exact accelerator model and the ROCm release that the host runs, including the driver stack. Check both against the GPU support list in the vLLM installation documentation for the release you intend to deploy, and confirm the list with your hardware vendor for your specific card. Do not assume that a GPU working well for other ROCm workloads is covered by the vLLM build. Next, confirm the Python version. The topic's documented requirement is a Python 3.12 wheel, so a host running another interpreter will need a different path. The install path then follows from these choices: a prebuilt wheel, the ROCm container image, or a source build. A source build is needed only when the wheel and image do not match your hardware or your dependencies, and it adds a compiler toolchain and build time that someone must maintain.

vLLM has no paid edition. It is open source under Apache-2.0 according to its GitHub repository, so license fees are not the cost driver. The recurring costs are AMD accelerator capacity, the servers, storage and network around them, and the staff who operate the platform. One-time costs include the design work, validation that your chosen models run on your hardware, and the installation itself. When you request a quote from an implementation partner, ask for these components to be listed separately: hardware or hosting, implementation hours by role, validation effort, and ongoing support. Ask the partner to state which accelerator, ROCm, and Python baselines the quote assumes, because a change to any of them can change the work.

Plan for release cadence before you pin a version. vLLM ships new minor releases about every two weeks, for example v0.31.0 on October 5, 2026, and each release can change which accelerators, ROCm versions and Python versions it supports. Pin the vLLM version, the ROCm version and the Python version together, and record them in your change log. Before promoting a configuration to production, run a short validation on a staging host with the same accelerator model, driver stack and Python version. Confirm that the models you need load, produce expected output, and meet your latency targets. This staging run is the cheapest place to find a mismatch, and it gives you a tested baseline for the upgrade work that follows.

Which ROCm install path fits your team

This table compares the three ways to install vLLM on AMD ROCm GPUs across when each fits, what your team must maintain and what to verify before go-live. Read each row as a question about your team's skills and change process. The main takeaway is that the container image trades host-level control for a pinned stack, while a source build gives the most control and the most maintenance.

Decision factor	Official ROCm container image	Prebuilt wheel on a matching host	Source build
Best fit	Teams that want a pinned, vendor-packaged ROCm stack and can run containers on GPU hosts.	Teams with a matching host OS and ROCm driver stack who want a Python-native install without compiling vLLM.	Teams that need a custom build, unreleased changes or dependencies the other paths cannot provide.
What your team maintains	Image tag selection, host GPU driver, container runtime settings and rebuilds of any custom image layers.	Host ROCm user space and driver versions, a Python 3.12 environment and dependency pins that match the wheel.	Build toolchain, ROCm headers and libraries, build time, patched dependencies and every rebuild after an upgrade.
Host-level control	Limited to what the image contains. Host kernel and driver changes still apply on the GPU host.	Full control of the Python environment and host libraries, within the wheel's version constraints.	Full control of compile flags, kernels and dependency versions, at the cost of longer build cycles.
Reproducibility	High when the image tag is pinned and mirrored inside your boundary.	Moderate, since host packages and the Python environment must stay identical across nodes.	High only if build inputs, toolchain versions and the source revision are recorded and reproduced exactly.
Upgrade effort	Change the image tag, re-test the deployment and roll it out node by node.	Confirm ROCm and Python compatibility for the new release, then reinstall the wheel on each host.	Rebuild from the new source, repeat build verification and re-test kernels and dependencies.
Accelerator and ROCm support	Confirm the image's ROCm version and listed GPU families in the current vLLM installation docs for your release.	Confirm the wheel's ROCm target and supported GPUs in the installation docs before ordering or allocating hardware.	Confirm the build targets your GPU architecture and ROCm version, since source builds depend on those inputs.
Verify before go-live	Check GPU detection inside the container, the reported vLLM version and a test chat completion on each node.	Check GPU detection, the Python version, the reported vLLM version and a test completion on each host.	Check GPU detection, the built commit, the test suite results and a test completion on each node.

How vLLM targets AMD GPUs and what the docs cover

The vLLM installation documentation lists AMD ROCm among the GPU platforms it runs on, alongside NVIDIA CUDA and Intel XPU. For planning purposes, the choice of AMD hardware is a platform installation decision, not a change to how the service behaves. The serving interface stays the same across these accelerators, so the same vllm serve command, the same server options and the same request formats apply whether the host has Instinct or Radeon cards or NVIDIA GPUs. Your platform team should read the ROCm section of the installation page for the exact release you plan to deploy and record which accelerators and ROCm stack it names as supported.

The server exposes OpenAI-compatible endpoints regardless of the hardware underneath. These include /v1/completions, /v1/chat/completions, /v1/responses and /v1/embeddings, so applications already written against the OpenAI API can point at a ROCm host with little change. In most cases the client change is limited to the base URL and the model name. Do not treat "usually" as proof, though. Run your own client test suite against the ROCm deployment before cutover, especially for structured output, tool calling, streaming behavior and any model-specific chat template that your applications depend on.

The Python version and wheel requirements named in the topic brief must be checked against the installation page for the release you are installing. Pre-built wheels and their Python constraints can change from one release to the next, and vLLM ships new minor releases about every two weeks. For example, v0.31.0 was published on October 5, 2026, so a guide written for an earlier release may no longer match the wheels or the Python versions that the current documentation names. Confirm these requirements before you build the host image, and write the release tag into your change record so later rebuilds use the same combination.

The documentation will not settle every question for your environment. It does not decide which GPU models suit your latency targets, how much accelerator memory a given model needs at your context length, or which ROCm driver and kernel combinations your operating system supports. Ask your hardware vendor and your Linux distribution for their supported matrices, then run a benchmark on the exact accelerator and model you intend to serve. Keep the ROCm version, driver version, Python version and vLLM release together as one pinned set, and plan a repeat test each time any of them changes.

Setting up vLLM on an AMD ROCm host, step by step

Follow this sequence to take a vLLM server on an AMD ROCm host from hardware check to a working chat completions response. Each step ends with a check you can record, so a failure points to one layer: hardware, runtime, Python environment, install or server configuration. The result is a reproducible server your team can test, monitor and rebuild.

Confirm accelerator and host OS support
Check the accelerator model and host operating system against the vLLM installation documentation and the ROCm compatibility information before you order or install anything. Your platform engineers or your implementation partner should do this with the server vendor's specification sheet in hand. Record the GPU model, kernel version, OS release and Python version in your build record. The step is done when every item matches a supported combination in the documentation, and any gap that needs a hardware or OS change is written down with an owner.
Install or verify ROCm driver and runtime
Install the ROCm driver and runtime versions that the vLLM installation documentation specifies for your accelerator, or confirm the versions already on the host match that pairing. Run the ROCm command-line monitoring tools to list the GPUs and confirm the driver reports every device you expect. Then check from a plain shell, outside any container, that the GPUs are visible to the runtime. The step is done when the device list shows all GPUs in a healthy state and the runtime version matches the documented pairing.
Choose container image or Python environment
Pick one delivery path: the ROCm container image listed in the installation documentation, or a Python environment built on the Python version the ROCm wheel requires. The installation documentation names that version, and the topic guidance for this page points to Python 3.12, so confirm it there before you build. Create a virtual environment with that exact interpreter, and keep it separate from system packages. For a container, mount a persistent model cache and pass through the device access the runtime needs. The step is done when the interpreter reports the required version, or the container starts and lists the GPUs.
Install the vLLM ROCm build
Install the vLLM build for ROCm from the package index or image that the installation documentation names. Use a source build only if the documentation says your accelerator or OS needs one. Pin the exact version you test with and record it, because vLLM ships new minor releases about every two weeks and your environment should change only on purpose. Confirm the install by importing the package and printing its version inside the same environment that will run the server. The step is done when the import succeeds without missing ROCm library errors and the printed version matches your pin.
Start the model with vllm serve
Start the server with vllm serve, passing the model path or identifier and the options your hardware plan needs, such as the number of GPUs for tensor parallelism. Bind the server to an internal address and port that only your gateway or approved clients can reach, and set an API key as one layer of access control. Watch the startup log until it reports that the model weights are loaded onto the GPUs and the API server is listening. The step is done when the log shows the model loaded without out-of-memory errors and the health endpoint responds.
Call chat completions from a test client
Send a short prompt to /v1/chat/completions using curl or the OpenAI-compatible client your application uses, pointed at the internal server address. Check that the response contains a generated message and the token usage fields. If the model has no chat template, the chat endpoint will not return a usable answer until you supply one with --chat-template, so confirm the template in the model files and the vLLM documentation before you troubleshoot the server. The step is done when a test client receives a coherent reply from the chat endpoint.
Record configuration and confirm metrics
Save the driver, runtime, Python version, vLLM version, image reference and the exact vllm serve arguments in your build record. Query the /metrics endpoint to confirm Prometheus-format series, such as request counts and token throughput, appear after the test request. Restart the server once and repeat the test to confirm the configuration is reproducible. The step is done when the build record matches what ran and the metrics show the test request.

When a source build is needed on ROCm

Build vLLM from source on ROCm only when the prebuilt wheel or the ROCm container image cannot run the stack you need. Three situations usually justify it. The first is an unsupported combination of ROCm and Python: the ROCm release your accelerators require, or the Python 3.12 wheel requirement, does not line up with a published artifact. The second is a patched dependency, where a security or correctness fix must ship before the upstream build carries it. The third is a custom kernel or a hardware plugin, which the vLLM installation docs describe as code that can live outside the main repository. Name the specific component behind each reason in your change record. If the reason is only a wish for a newer feature, first check whether a published release or image already provides it, because a source build carries recurring work.

A source build moves the packaging work onto your team. You must manage the build toolchain: the ROCm SDK matching your target, a compiler set that the ROCm release supports, a Python build environment, and the system libraries the build links against. Reproducibility depends on pinning every input: the vLLM tag or commit, the Python and dependency versions, build flags, and the base image digest. Store the resulting wheel or image in an internal registry so that a later rebuild cannot silently change it. Confirm the current build steps in the official installation documentation for your target, because build requirements can change between releases.

The heaviest recurring cost is the rebuild cycle. vLLM ships new minor releases about every two weeks, for example v0.29.0 on September 9 and v0.30.0 on September 22, 2026. Each minor release you adopt means repeating the build, checking that your patches or custom code still apply, and retesting. Plan who owns that cycle and how long you will stay on a release before you move forward, because staying on an old tag can leave you without fixes that later releases receive. Treat this work as ongoing engineering time that continues after the first build succeeds.

Test every build against the ROCm version you chose before it reaches users. A practical gate loads the target model at the tensor parallel size you will run, sends requests to /v1/completions and /v1/chat/completions, confirms that /metrics reports request counts and KV cache use, and compares throughput and latency with a baseline from the previous build. Run the same gate for any custom kernel or plugin, since those components are the most likely to break when a new vLLM minor release arrives.

Record the decision next to the pinned versions. The record should name the trigger (the ROCm and Python combination, a patched dependency, or a custom kernel or plugin), the affected component, the vLLM tag and commit, the ROCm and Python versions, the build flags, the base image, the owner, and the condition for returning to a prebuilt artifact. A reviewer should be able to rebuild the artifact from that record alone. When you request a quote, ask the implementer to price the initial build, the test gate and the per-release rebuild as separate lines, because they recur on different schedules.

Security settings to keep on ROCm hosts

vLLM's security behavior does not depend on the GPU vendor, so the same controls apply on AMD ROCm hosts as on NVIDIA CUDA hosts. The cards below cover the API key option, the endpoints it leaves open, the proxy allowlist that should sit in front of the server, the operational endpoints that need network restriction, and the traffic between nodes in multi-node runs.

The api-key option is partial

The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere, but it does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health. A pause or abort endpoint left open can disrupt a shared ROCm server even when every chat request carries a valid key. Treat the key as one layer and confirm, by sending unauthenticated requests to each path, which endpoints actually reject them.

Api key is not sufficient alone

The vLLM security documentation warns against relying on --api-key as the only control. On a ROCm host that serves several applications, the key is shared secret material that tends to spread to clients, scripts and logs. Pair it with network placement and a gateway that enforces who may call which path, so a leaked key does not expose administrative functions.

Reverse proxy endpoint allowlist

Place a reverse proxy in front of the vLLM server and allow only the endpoints meant for users, such as /v1/chat/completions, /v1/completions, /v1/embeddings or /v1/responses, depending on what your applications call. Deny everything else by default, including the pause, resume and abort paths. Confirm the allowlist by requesting each blocked path through the proxy and checking the response. This also keeps future endpoints from being exposed by a version upgrade without review.

Restrict metrics and health endpoints

The /metrics endpoint publishes Prometheus data on request counts, token throughput, latency and KV cache use, and /health reports server state. Neither is covered by --api-key. Bind scraping to your monitoring network or allow only the Prometheus server address at the host firewall or proxy. Check that a request to /metrics from an application subnet is refused, while the monitoring server still receives samples.

Isolate inter-node traffic

In multi-node runs that use pipeline parallelism across nodes, the vLLM documentation states that communication between nodes is insecure by default, with no authorization or encryption. Place all nodes on a dedicated network segment that user traffic cannot reach, and restrict Ray or multiprocessing ports to the cluster members. Ask your network team to verify the segment with a scan from outside it, and plan encryption on that segment if your policy requires it.

Review settings at each upgrade

Endpoints, flags and default behavior can change between vLLM releases, and the project ships new minor versions about every two weeks. Re-run the unauthenticated endpoint checks, the proxy allowlist tests and the internal port scan after each upgrade on a ROCm host, before the new image or wheel reaches production. Keep the results as evidence for your change records.

Verifying throughput and metrics after the ROCm install

Before anyone sends production traffic to a vLLM server on ROCm, confirm three things: the model loaded on the GPUs you intended, the tensor parallel setting matches how many GPUs the model is split across within the node, and the server reports the metrics you will later use to judge it. Start with the startup log. It shows which model and dtype were loaded and how many devices each worker bound to. Then check the GPU view from the host, for example with the ROCm system management tooling in your image, and confirm that only the expected devices show memory allocation and that no other process is holding them. A model that loads on the wrong card, or that silently runs on a single device while you expected a split, will look healthy in a simple request test but will perform very differently under load.

For models that span more than one GPU inside a node, verify the tensor parallel size explicitly rather than trusting the default. The value should equal the number of accelerators you dedicated to that server instance, and it should divide evenly into the model's attention head layout as the engine reports at startup. Check that the server process sees the device indices you set through the ROCm visibility variables, and that a second instance on the same host does not claim devices already in use. Record the exact command line, the vLLM version, the ROCm release and the container image digest or wheel version in your change record. These are the values you will need when someone asks why throughput changed after an upgrade.

Next, scrape the Prometheus-format endpoint at /metrics on the API server. The endpoint exposes request counts, token throughput, latency histograms and KV cache usage, which are the signals that show whether capacity is adequate. Point a Prometheus instance you control at the server, confirm the series appear, and build a short dashboard covering queued and running requests, time to first token, time per output token and KV cache utilization. The histogram buckets are configurable, so decide them before you rely on percentile queries. Note that /metrics is not protected by the --api-key option, so the endpoint should be reachable only from your monitoring network, which the reverse proxy in front of the server should enforce.

Benchmark numbers from vendors, community posts or the vLLM project depend on the model, the accelerator generation and count, the ROCm and PyTorch builds, the input and output length mix, concurrency and the sampling settings. None of those are likely to match your environment exactly, so a published tokens-per-second figure is not an acceptance criterion. Write your own. Choose representative prompts from your real workload, define the concurrency levels you need to support, and set thresholds for latency percentiles, error rate and KV cache headroom at peak. Run the same test before and after every engine, driver or image change so the comparison is like for like. Ask your ROCm and vLLM contacts to confirm which versions were tested together for your accelerator model before you fix the thresholds.

Limits to confirm for your AMD GPU model

The vLLM release, the ROCm stack, the Python interpreter and your model are linked, so a mismatch in one usually breaks the others. Confirm each limit below against the vLLM installation documentation and the release notes for the exact version you plan to run, and record the results in your design.

Exact accelerator families

Confirm the specific AMD Instinct or Radeon models listed for the release you plan to run, down to the SKU. Support can differ between data center and workstation parts, and between generations within the same family. Check the ROCm section of the vLLM installation documentation, then compare it with the release notes for any changes since your last validation. Ask your hardware vendor to confirm that the SKU, firmware and driver you purchase match that list.

ROCm version pairing

Each vLLM release is built and tested against particular ROCm versions, so record the pair before you choose an operating system image or driver stack. The installation documentation names the supported ROCm versions for each release line. Read the release notes between your current and target versions, because the pairing can change within a minor release line. If you use the ROCm container image, confirm its ROCm version against the same list and check that it matches the host driver.

Python version for wheels

The ROCm wheel path requires Python 3.12, so the interpreter, base image and build environment all need to match that version. Check the installation documentation for the wheel source and the Python versions each release publishes. If your organization standardizes on another Python version, plan for a source build or a separately managed environment, and confirm the approach with the project guidance before the build starts. Pin the interpreter in your image so an operating system update cannot change it silently.

Model architecture and quantization

Support on ROCm depends on both the model architecture and the quantization method, and a model that serves well on NVIDIA CUDA may not have equivalent kernels on AMD. Check the supported models list and the quantization notes in the documentation for your release. Then load your exact weights, with your planned quantization and context length, on the target GPU before you size capacity. Record the failing cases as well, because they determine whether you need a different model or a different accelerator.

Multi-node backend choice

Multi-node runs use either Ray or vLLM's own multiprocessing backend, and the choice shapes how you deploy workers and operate the cluster. Read the parallelism and scaling documentation to confirm which backend options apply to ROCm in your release. Inter-node traffic is insecure by default, with no authorization or encryption, so place every node on an isolated network whichever backend you choose. Decide the backend before you write deployment manifests or scripts.

Third-party plugin support status

Hardware plugins outside the main repository are maintained separately, so the main project's installation and release notes may not cover them. For any plugin that supplies AMD or other accelerator support, confirm which vLLM versions it targets, which ROCm versions it tests, and who maintains it and how quickly it follows new releases. Request that support status in writing, and plan a test upgrade before each vLLM minor release, which ships about every two weeks.

Questions buyers ask about vLLM on AMD ROCm

What does it cost to run vLLM on AMD ROCm GPUs?

vLLM itself has no paid edition, so the cost comes from four places: the AMD GPU capacity, the infrastructure around it, the implementation work and the staff who operate the service. GPU capacity is usually the largest line, whether you buy hardware or use accelerators in your own cloud account or data center. Infrastructure covers hosts, storage for model weights, networking, load balancing and monitoring. Implementation is one-time effort to install, validate the model and connect the API. Operations recurs: patching, testing vLLM upgrades (minor releases ship about every two weeks), capacity planning and on-call coverage. When you request a quote, ask for the written scope, the named deliverables, what you must supply, and which costs are one-time versus recurring, including any retainer for ongoing support.

How long does a vLLM on ROCm setup take?

No single duration applies, because the timeline depends on version alignment and access more than on vLLM alone. The main driver is whether the GPU model, ROCm release, Linux kernel, Python version and vLLM release all match one tested combination. The vLLM wheel path requires Python 3.12, so the Python level is set before anything else. If you use the ROCm container image, much of the driver and library packaging is already done, which shortens the work. Time also goes to hardware access, staging model weights, network approvals and load testing. Ask any provider for a timeline tied to a written scope, with milestones and dependencies that show which items sit on your side.

What do we need to provide before a vLLM on ROCm engagement starts?

Plan to supply five things before work begins. First, hardware access: AMD GPU hosts in your cloud account or data center, with administrative access to install drivers and the container runtime. Second, a test model: one model you are licensed to use, with its weights staged inside your boundary. Third, a network plan showing which subnets reach the API, the reverse proxy and the metrics endpoint. Fourth, named operators who will own the service after handover and have authority to approve changes. Fifth, a change window and sign-off owner for validation. Confirm these items in the written scope so that delays on your side are visible in the timeline and the quote.

Can our own team run vLLM on AMD ROCm without a partner?

Yes, if the team already has ROCm, GPU driver and Linux operations skills, plus time to test. The vLLM installation path is documented, and the ROCm container image gives a tested starting point. Gaps usually appear in four places: matching driver and ROCm versions to the GPU, validating the chosen model on the hardware, securing the API with a reverse proxy because the --api-key option alone is not enough, and keeping up with a fast release cadence. Run a short skills check: can your engineers read driver and kernel logs, pin versions, roll back a container and write alerts on the Prometheus metrics? If two or more answers are no, a scoped engagement may be worth comparing against the learning time, and a written quote will show the difference.

When is a vLLM source build on ROCm worth the extra effort?

Build from source only when the prebuilt options do not fit your stack. Typical reasons include a GPU, ROCm release or Python version outside what the published wheels or the ROCm container image target, a required patch or custom dependency, or a need to build inside an air-gapped boundary with pinned inputs. A source build adds work: compiling against the exact ROCm toolchain, maintaining your own build pipeline, rebuilding for each vLLM release and recording build flags for audit evidence. Check the target GPU and ROCm combination in the vLLM installation documentation first. If a prebuilt wheel or the container image works, use it, and keep the source build for gaps you have confirmed.

What should a security review check first in a vLLM ROCm deployment?

Start with network exposure. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, so those nodes must sit on an isolated network. The --api-key option protects routes under /v1 and similar paths, but not /metrics, /health, /pause or /invocations, so place a reverse proxy that allowlists only the endpoints meant for users. Check who can reach /metrics, because Prometheus metrics expose request and KV cache activity. Then review image provenance, pinned versions, and where model weights and logs are stored. This work produces evidence for your assessor. DeployOpen prepares systems and controls for compliance assessments but does not issue audit reports or certifications; independent auditors do.

How DeployOpen scopes and delivers vLLM on ROCm

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including cloud accounts, data centers and air-gapped networks. For vLLM on AMD ROCm GPUs, the accelerator, ROCm, Python and container choices are settled in writing before the serving stack goes live.

How we scope

Confirm AMD GPU models, host operating system and ROCm release against vLLM's installation documentation before build.
Decide between the ROCm container image and a source build, based on the packages and kernels your hosts need.
Agree fixed deliverables, timelines and acceptance tests in writing before work starts, with ongoing support available as a retainer.

How we build

Install the Python version and vLLM wheel that match your ROCm build, and record each pinned version.
Test model loading, tensor parallel settings and the chat template on your own AMD hardware before any cutover.
Configure the OpenAI-compatible endpoints behind a reverse proxy that allowlists only the routes your users need.

How we hand over

Deliver the build record, pinned versions, configuration files and proxy allowlist as documented handover material.
Run a walkthrough with your platform team covering startup, the /metrics endpoint and the --api-key limits.
Document which endpoints the --api-key option does not protect, so operators know which routes need proxy controls.

How we support

Plan upgrades around vLLM's minor releases, which ship about every two weeks, and re-run acceptance tests on each target version.
Review ROCm and driver changes before upgrades, since accelerator support depends on the ROCm release you run.
Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing like the original build.