What a vLLM migration involves
A vLLM migration moves an application's model calls from a hosted API or another serving engine to a vLLM server that you run. The server started with vllm serve exposes OpenAI-compatible endpoints, including /v1/completions, /v1/chat/completions, /v1/responses and /v1/embeddings, so many applications built for the OpenAI API can change a base URL and key instead of their code. Teams often underestimate the work at this point. The important checks are which endpoints and request fields the application uses, how streaming, tool calls and structured output behave, and whether the new model's answers still meet the application's quality bar.
Three triggers usually start the project. The first is hosted API cost at your token volumes, or a data residency or contract requirement that keeps prompts and outputs inside your own network. The second is a move from another serving engine, which brings differences in chat templates, default sampling settings and model loading that must be matched and tested. The third is capacity, when hosted rate limits or quotas block growth. Write down which trigger applies, because each sets different priorities: residency puts logging and network design first, cost puts measured utilization first, and an engine change puts output parity first.
Teams often ask whether to do this in-house or with a partner. A team that already operates GPU nodes, can own model behavior and has time for load testing can usually run the migration itself. A partner is worth comparing when accelerator procurement, Kubernetes, access control or a documented cutover are new to the team, or when the date is fixed. vLLM is Apache-2.0 licensed and has no paid edition, so no license decision is involved. The decision is about infrastructure, testing and operations, and any quote should itemize those three areas.
What the work includes
The migration runs as a sequence of work packages, each ending with a artifact the client keeps. The list below shows what the engagement covers and what the client receives at the end.
Inventory of model calls
Every application, job and script that calls the current model API is listed with its endpoints, request fields, streaming use, tool or function calling, and token limits. This inventory defines the compatibility scope, so anything missed here tends to surface as a production failure after cutover. The client receives the inventory as a maintained document with owners for each caller.
Compatibility testing against vLLM
Request and response behavior is tested against the vLLM server's OpenAI-compatible endpoints, including /v1/chat/completions, /v1/completions and /v1/responses where the inventory uses them. Tests cover field handling, error shapes, streaming chunks and finish reasons, since small differences in these break client parsers. The client receives the test suite, which it can rerun on every vLLM upgrade.
Serving configuration and chat templates
Each target model gets a serving configuration covering parallelism, context length, and the chat template, which the chat completions API requires for the model to serve. Template choice changes output quality and tool-call formatting, so it is validated against outputs from the current API. The configuration is delivered as code in the client's repository.
Load testing with production-shaped traffic
Load tests replay traffic shaped like production: prompt and output length distributions, concurrency bursts, and the mix of models and endpoints from the inventory. Results are compared against the latency and throughput targets the client sets before testing starts. The client receives a load test report that states the conditions under which each result was measured.
Gradual traffic shift through a gateway
Traffic moves from the hosted API to vLLM in steps through the client's API gateway, with routing rules that can be changed without an application deploy. Each step has an agreed set of metrics to check before the next step is approved. The gateway configuration is part of the delivered configuration as code.
Tested fallback to the original API
A fallback path routes requests back to the original API if vLLM errors, times out or fails the agreed checks. The fallback is exercised in test and during the shift, not assumed to work. Its limits are recorded, including any requests that cannot be replayed safely and any differences in output that callers must accept during the fallback.
Cutover and rollback runbooks
Runbooks describe the cutover steps, the criteria for pausing or reversing a traffic step, the on-call contacts, and how to restart or drain vLLM replicas. They are written for the operators who will run the platform, not only for the migration team. The client receives them with a dry run record showing the rollback path was executed.
What drives effort and duration
This table lists the factors that most change the work of moving an application to self-hosted vLLM, why each matters and how to keep it under control.
| Driver | Why it matters | How to keep it down |
|---|---|---|
| Distinct model calls and endpoints | Each hosted API call the application makes must map to a vLLM endpoint such as chat completions, completions, responses or embeddings, and each mapping needs its own test cases. | Inventory every call site before scoping, and move one endpoint type at a time, starting with the chat completions calls used most. |
| Provider-specific features | Tool calling, structured output, streaming behavior, token limits and sampling parameters can differ from the hosted provider, so responses may change even when the request succeeds. | List which features the application depends on, then write compatibility tests that compare outputs against the current provider for each one. |
| Number of models to host | Every model needs its own serving configuration, memory sizing, warm-up and regression tests, so effort grows with each model added to the migration. | Limit the first wave to the models with the highest traffic or the fewest provider-specific features, and defer the rest to a later phase. |
| Chat template availability | The chat completions endpoint requires a chat template, either bundled with the model or supplied at serve time, so a missing or incorrect template can break prompts silently. | Confirm the template for each model before migration begins and add tests that check the rendered prompt format against a known-good reference. |
| Accelerator type and multi-node parallelism | Accelerator families differ in supported features and tuning, and models too large for one node need multi-node parallelism, which adds networking, isolation and failure-handling work. | Choose the accelerator family and node layout against the largest model in scope, and confirm current hardware support in the vLLM installation documentation before procurement. |
| Traffic volume and latency targets | Higher concurrency and tighter latency goals require load testing, capacity planning and possibly several replicas behind a load balancer, which extends test cycles. | Agree target latency and concurrency numbers in writing before testing, and run load tests with production-shaped prompts rather than synthetic short requests. |
| Number of client applications | Each application has its own SDK configuration, retry logic, timeouts and error handling, and each one needs to be switched and verified separately. | Consolidate calls behind a shared client library or gateway route where possible, and migrate applications in a planned order with a rollback path for each. |
| Output quality evaluation | Changing the model or serving engine can shift answer quality, and proving equivalence requires a representative evaluation set and agreed acceptance criteria. | Build a fixed evaluation set from real, approved prompts and agree pass criteria with the application owners before the traffic shift starts. |
| Approval and data residency requirements | Security reviews, data processing approvals and residency rules determine where models run, which logs may be kept and who can approve each stage of the traffic shift. | Gather approvals for the target environment early, and document the data flows so each review covers the same architecture rather than restarting for each change. |
Migration phases and outputs
The sequence moves traffic from the current model API or serving engine to self-hosted vLLM only after each stage has produced evidence that the new endpoint behaves acceptably.
Inventory calls and model dependencies
DeployOpen and your application owners list every place that calls the current model API or serving engine, including models, endpoints, request shapes, streaming use, tool calls, and token volumes. The output is a call inventory that names each consumer, its owner, and the vLLM endpoint it will need, such as /v1/chat/completions or /v1/embeddings. The step is done when the owners have signed off the inventory.
Prepare environment and pin the vLLM release
Your team and DeployOpen confirm accelerator capacity, drivers, container images or Python environments, and model file sources, then pin one vLLM release for the migration. Each supported model must have a chat template, or one supplied with the chat template option, before chat traffic can be tested. The output is a reproducible staging environment and a written version pin that will not change during the migration without a new test cycle.
Test compatibility and output quality
DeployOpen runs a reference set of prompts drawn from your inventory through the current service and the vLLM endpoints, then compares response structure, error handling, tool-call behavior, and output quality against criteria your owners set. Differences are logged and classified as acceptable, requiring prompt or parameter changes, or blocking. The output is a compatibility report with a pass or fail decision per endpoint and per model, reviewed by the application owners.
Run load and failure tests
DeployOpen drives representative concurrency and token volumes at the staging endpoint, and injects controlled failures such as a restarted server, a dropped GPU node, or an unavailable upstream gateway. Latency, throughput, queueing, and error rates are recorded against the targets agreed in the written scope. The output is a load and failure test report that states the observed limits and the recovery behavior, which feeds the capacity decision.
Shadow and run in parallel
Selected production requests are copied to vLLM while the current service still answers users, and the responses are compared offline without affecting users. Outputs, latency, and error patterns are reviewed over a period agreed in the written scope. The output is a parallel-run comparison that shows where the two services diverge and whether those differences were resolved or accepted by the owners.
Shift traffic gradually with a fallback
Traffic moves to vLLM in steps through the API gateway or routing layer, with each step held until agreed health checks and quality checks pass. The current service stays configured as a fallback route that can be restored by changing routing rules, and the rollback procedure is rehearsed before the first step. The output is a traffic-shift log with the share served by vLLM at each step and a tested rollback record.
Hand over with runbooks
DeployOpen delivers runbooks for deployment, version pinning, model changes, scaling, monitoring, incident response, and rollback, and your operations staff run a supervised handover exercise. Ownership of alerts, on-call duties, and the upgrade path is recorded. The output is a signed handover package and a documented support model, which determines whether ongoing support is covered by a retainer or by your own team.
What usually goes wrong
Most failed vLLM migrations trace back to a small set of assumptions about compatibility, access control, networking and release management. Each card names one risk and the check that prevents it.
Assuming the OpenAI contract is identical
vLLM implements OpenAI-compatible endpoints, but edge cases in request fields, streaming behavior, error shapes and token accounting can differ from the hosted API. Replay real request logs from the application against the vLLM endpoints and compare responses field by field before any traffic moves.
Models without a chat template
The chat completions API needs a chat template. A model that ships without one fails on /v1/chat/completions unless a template is supplied with the chat template option. Confirm each target model's template during compatibility testing, and use the completions endpoint only where the application's prompt format is known to match the model.
Metrics and health endpoints left reachable
The --api-key option does not cover /metrics or /health, so a server exposed with only that option can leak request volumes, latency and cache data to anyone who can reach it. Place a reverse proxy in front of the server that allowlists only the endpoints users need, and confirm from outside the network that the others return denied responses.
Treating --api-key as full access control
A shared key does not identify users, separate teams, or limit what a caller may do with the endpoints it covers. For a migration that replaces a hosted API with several internal consumers, decide who authenticates at the gateway, how keys are issued and rotated, and how usage is attributed before cutover.
Multi-node traffic without isolation
vLLM's multi-node communication is insecure by default, with no authorization or encryption between nodes. If the model spans machines for pipeline parallelism, place those nodes on an isolated network segment that clients and general corporate traffic cannot reach, and verify the segment's firewall rules during the build rather than after go-live.
Latency untested under real concurrency
Single-user tests rarely reflect production queueing, batching and long-prompt behavior. Run load tests that mirror the expected mix of prompt lengths, output lengths and concurrent sessions, and record time to first token and end-to-end latency at each level. Compare the results with the service levels the application owners actually need.
Breaks from fast minor releases
vLLM ships new minor releases about every two weeks, for example v0.31.0 on October 5, 2026, and a working configuration can change behavior between them. Pin the version in deployment manifests, run the compatibility and load suites against any candidate release in a staging environment, and roll forward only after those results pass.
What stays in-house and what goes to a partner
This table compares, task by task, the parts of a vLLM migration that teams usually keep internally with the parts they often hand to a partner, along with the skills each one needs, so you can judge where your own staff are strongest.
| Migration task | Usually kept in-house | Often outsourced and skills needed |
|---|---|---|
| Application ownership | Owning the client applications, deciding which call paths move first, and signing off on behavior changes for end users | Not usually outsourced. Application developers familiar with the OpenAI API client libraries and their own request patterns are needed to repoint code. |
| Prompt and product decisions | Writing system prompts, deciding what the model may answer, and approving changes to user-facing behavior after the switch | Not usually outsourced. Product and domain owners are needed, with prompt versioning in your own source control. |
| Model selection | Choosing which open-weight model replaces the hosted model and accepting its quality, licence and data-handling trade-offs | Partly outsourced. A partner can run comparative tests, but a model owner in your organization should own the decision. |
| Quality acceptance | Defining pass criteria for answers, tone, refusals and structured output, and deciding when results are good enough to cut over | Keep the acceptance decision. Evaluation tooling (for example, a test runner over saved prompts) can be built by a partner and run by your team. |
| Compatibility harness | Reviewing results and owning the test cases that reflect your business rules | Often outsourced. Python API client work against the OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/responses), including chat template and tool-call checks. |
| Serving configuration | Setting the model, context length and access policy requirements that the platform must meet | Often outsourced. vllm serve flag tuning, chat template selection, GPU and Linux operations, and Kubernetes or VM operations for the vLLM pods or processes. |
| Load testing | Supplying representative prompts, concurrency targets and latency expectations from real traffic | Often outsourced. Load testing tools such as k6, Locust or vegeta, plus reading vLLM Prometheus metrics during each run. |
| Gateway and traffic shift | Approving the rollout steps, the percentage schedule and the rollback criteria for user traffic | Often outsourced. Gateway configuration (for example, Apache APISIX or LiteLLM) for weighted routes, endpoint allowlists and fallback to the hosted API. |
| Runbooks and handover | Operating the platform after go-live, on-call rota and incident ownership | Often outsourced for drafting. Writing runbooks for restarts, model swaps, GPU failures and fallback, which your operators then rehearse. |
How the migration changes by starting point
Applications already written for a hosted API move most easily, because the vLLM server exposes OpenAI-compatible endpoints such as /v1/chat/completions and /v1/responses. Client code often stays largely the same, so the work concentrates on model behavior and configuration.
- Testing: replay recorded prompts through both endpoints and compare output structure, tool call formats, stop reasons and token limits.
- Configuration: confirm the served model has a chat template, or supply one with --chat-template, before chat traffic arrives.
- Cutover: change the base URL and key behind a feature flag, shift traffic in steps, and keep the hosted API available as a fallback.
- Confirm: which endpoints your applications call and whether they depend on parameters that only the hosted provider supports.
Moving from another open-source engine may look familiar at the API level, but default sampling settings, batching behavior and supported model formats can differ. Testing needs to compare the engine you are leaving with vLLM under the same conditions.
- Testing: generate outputs with fixed sampling settings on both engines, then run the same load profile against each and compare latency and error rates.
- Configuration: map existing engine flags to vLLM server flags, and check model format and quantization support for every model you serve.
- Cutover: run both engines in parallel, route a small share of requests to vLLM, and expand only after results match your acceptance criteria.
- Confirm: which vLLM release you will pin, since minor releases arrive about every two weeks, and whether that release supports your model architectures.
The deployment target changes how GPUs are scheduled, how the server restarts and how metrics are collected. Model load time and health checks need to match the platform, because the server is not ready until the model has loaded.
- Testing: verify GPU detection, driver and runtime versions, and model load time after a node restart on the target platform.
- Configuration: on Kubernetes, choose native manifests, Helm or another option from the vLLM Kubernetes documentation; on VMs, use a pinned install under a service manager.
- Cutover: on Kubernetes, use readiness probes that wait for the model to load; on VMs, drain traffic from a host before restarting it.
- Confirm: which Kubernetes option your platform team supports, whether GPU device plugins are already installed, and how the /metrics endpoint will be scraped without public exposure.
Models too large for one node use tensor parallelism within a node and pipeline parallelism across nodes. vLLM's security documentation states that communication between nodes is insecure by default, so network isolation is part of the migration plan from the start.
- Testing: measure throughput and latency at the chosen parallelism sizes, and test service behavior when one node in the group fails.
- Configuration: select Ray or the built-in multiprocessing backend, and keep all inter-node traffic on an isolated network segment.
- Cutover: send traffic only after the full multi-node group passes readiness checks, and keep a single-node or hosted fallback for the first stage.
- Confirm: that your network design isolates node communication, and that your allocated GPUs provide enough memory for the chosen model and parallelism settings.
Air-gapped networks change where model weights, documentation and dependencies come from. The server cannot fetch anything at startup, so every artifact must be staged in advance through your approved transfer process.
- Testing: start the server with no outbound network access and confirm it loads the model from local storage.
- Configuration: use --enable-offline-docs so API documentation is served without external assets, and point the server at a local model path.
- Cutover: stage container images, Python packages, model files and chat templates, verify checksums on arrival, and record which artifacts each release uses.
- Confirm: whether your transfer process accepts the vLLM release you plan to pin, and how model file provenance will be documented for your assessors.
What you need to provide
Migration starts with access to the applications that call the hosted model API and to their model call code. Your application owners should provide read access to the repositories, a staging environment, and the configuration that sets the base URL, model name, timeouts and retries. Compatibility testing begins from a list of the request fields and response features each call depends on, such as streaming, tool or function calling, and structured output. Only the people who own those services can confirm that list accurately.
Your product or domain experts should provide a representative set of prompts and the behavior they expect from each. The set should reflect production traffic, including edge cases and the inputs that caused past incidents, with sanitized data where needed. Pair it with a capacity decision from your infrastructure owner: which accelerators to use, whether to buy or rent them, and the supplier lead time. Confirm that your target model and its chat template are supported on the hardware you plan to use before procurement starts.
Your network and identity teams need to approve the paths from each application to the new serving endpoints, including DNS, TLS, firewall rules and the identity each caller uses. vLLM's own --api-key option does not cover every endpoint, so plan for a reverse proxy that exposes only the paths users need. Name one owner for quality acceptance. That person signs off on compatibility and load test results and on the thresholds that allow each traffic step to proceed.
Three decisions close the list. First, decide whether the hosted API stays available as a fallback, for how long, and who can route traffic back. Second, obtain data handling approvals from privacy, legal and security for moving prompts and outputs onto your infrastructure, including what is logged and how long it is kept. Third, name the team that carries on-call duty after cutover, with access to runbooks. DeployOpen can help prepare these runbooks, but the on-call rotation and its escalation path should belong to your organization.
Frequently asked questions
How much does a migration to self-hosted vLLM cost?
The cost depends on the written scope, not on a list price. vLLM itself is open source with no license fee, so the spend comes from accelerator capacity, the surrounding infrastructure, implementation work and the staff who operate the service. Implementation is one-time: compatibility testing, load testing, the traffic shift and fallback design. Accelerator capacity, hosting, monitoring and on-call support recur. A quote should separate these lines, state which models and applications are in scope, and name the number of endpoints and environments involved. DeployOpen does not publish prices.
How long does a vLLM migration take?
Duration depends on scope, so no single timeline applies to every migration. The main drivers are how many applications call the current API, how many models must be validated, whether the accelerator capacity is already in place, and how much compatibility work the applications need. A single application using standard chat completions usually needs less work than several services relying on embeddings, tool calls or provider-specific behavior. Ask for a timeline tied to a written scope with named phases and acceptance checks.
Can my existing application switch to vLLM by changing the base URL?
Often the base URL and model name are the first changes, but it is rarely the only one. vLLM exposes OpenAI-compatible endpoints such as chat completions, completions, responses and embeddings, so many OpenAI SDK clients can point at it. Check the endpoints your code calls, how it handles streaming, token limits, function or tool calling, and error formats. Models also need a chat template for chat completions, or one supplied at startup. Confirm each of these against your application before assuming a configuration-only change.
What does the migration need from our team?
Your team supplies the knowledge of the applications and their traffic, access to the target cloud accounts, data centers or private networks, and a decision-maker who signs off on acceptance criteria. Expect input on which prompts and workflows matter most, expected concurrency and latency targets, model choices and data handling rules. Your staff also needs to own the eventual production system, so plan time for review sessions, change approvals and handover of runbooks and monitoring.
Can our own team do the migration without a partner?
Yes, a capable internal team can run a migration, provided it has experience with GPU operations, Kubernetes or virtual machines, API gateways and load testing. The questions to answer honestly are whether those skills are available for the migration period and for ongoing operation, and whether the team can build compatibility tests and a fallback path before traffic moves. A partner is most useful when those skills are scarce or the deadline is fixed. Either way, the written scope should define who owns each deliverable.
Does vLLM return exactly the same output as the hosted model API?
No, do not assume identical output. vLLM serves the open-weight model you choose with its own runtime, so the model, its version, quantization, sampling settings and chat template all affect the response. Hosted APIs may use different model versions or serving behavior that you cannot inspect. The migration should compare outputs on your own test set, measuring the quality, format and safety checks that matter to your applications, and accept differences only where the business case allows them.
How does fallback work if the vLLM service fails?
Fallback is a routing decision made at the gateway or application layer. A common approach keeps the hosted API configured as a secondary route, sends a small share of traffic to vLLM first, and switches back when error rates or latency cross thresholds you define in advance. Confirm that the fallback model produces acceptable output for each use case, that credentials and data handling rules still apply, and that the switch is tested in practice. The design should be agreed in writing before traffic moves.
How DeployOpen delivers this work
DeployOpen plans and implements self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and treats the move from a hosted model API or another serving engine as a staged cutover with a fallback path. Engagements are fixed-scope, with the scope agreed in writing before work starts.
How we scope
- Inventory every application call to the current hosted API or serving engine, including request shapes, streaming use and embeddings.
- Map each endpoint to the vLLM OpenAI-compatible routes, such as chat completions, completions, responses and embeddings, and record gaps.
- Define the target models, accelerator capacity, chat templates and API key or gateway needs that the written scope will cover.
- Agree the traffic shift plan, rollback trigger and fallback target in writing before any production traffic moves.
How we build
- Run compatibility tests that send recorded production-style requests to the self-hosted endpoints and compare response structure and behavior.
- Execute load tests against the planned accelerator sizing and record latency, throughput and queueing before any user traffic is routed.
- Configure the gateway or reverse proxy to allowlist only the endpoints meant for users, since the vLLM API key does not cover every route.
- Shift traffic in gradual steps with monitoring on request counts, latency and KV cache use, keeping the previous provider available as fallback.
How we hand over
- Document the deployed configuration, server flags, model versions, endpoint allowlist and fallback procedure for your operations team.
- Provide runbooks for traffic rollback, replica restarts and model changes, written against your chosen Kubernetes or VM deployment.
- Leave the test suites used for compatibility and load testing so your team can rerun them before future changes.
- Walk your engineers through the metrics dashboards and alert rules that show whether the migrated applications behave as expected.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after the migration is complete.
- Retainer work can include regression testing of each new vLLM minor release before your team decides whether to adopt it.
- Confirm with your team how often support hours, response expectations and release testing will be needed before signing a retainer.
- Ask for the retainer scope in writing so that the upgrade, testing and incident responsibilities are clear to both parties.
