Migrate from Azure OpenAI to self-hosted models on vLLM

Application code, prompts and evaluation datasets move with you, but model weights do not. Applications written against the OpenAI request format can point at a vLLM server, which exposes /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, so the main code change is usually the base URL and the model name, routed through a LiteLLM gateway. Your evaluation datasets, prompt templates and test cases are also portable, because they are your own files. Weights, however, cannot be exported from Azure OpenAI. Each open-weight model you run must be sourced from its publisher under that publisher's license, and your legal team should review those terms before any production traffic is planned.

Replacement models behave differently, so every prompt, output parser and guardrail needs revalidation. A different model family, tokenizer or version can change formatting, refusal behavior, tool-call structure and JSON reliability, which can break downstream parsers without any error from the server. Run your existing evaluation set against each candidate on your own prompts, record pass rates per workflow, and pin the exact model revision you approve. Treat this as a gate in the migration plan, not a final check. Confirm with your security and compliance owners whether any content filters or Azure-side safety settings you relied on need an equivalent in your gateway or output checks.

Azure-side logs and stored data do not follow the workload automatically. Any prompt, completion, deployment metadata or audit record you need for investigations, retention obligations or quality history must be exported through Azure's own tools before the resource is decommissioned. Confirm with your Azure administrators which diagnostic settings, log destinations and retention periods are in place, and which data is held under abuse monitoring. Plan the export as its own work item with an owner, a storage location in your environment and a sign-off, because once the deployment is removed, the data may no longer be retrievable by your team.

Run both paths at once and move traffic in steps, so Azure OpenAI stays in service until your own evaluation results hold. In LiteLLM, define the vLLM deployment and the existing Azure deployment under the same model group, then use priority ordering so Azure serves as the failover target. Start with a small share of traffic on vLLM using simple-shuffle routing, which LiteLLM recommends as its default, and compare error rates, latency and evaluation scores per route before raising that share. Latency-based routing can help once both backends have been measured under real load, but confirm its behavior with your own traffic before relying on it. Keep the Azure deployment, its keys and its fallback entry in place until the cutover criteria you agreed with the evaluation team have been met for a sustained period, and record the date you remove it so the decision is auditable.

The vLLM server needs its own protection before any application traffic reaches it. The --api-key option does not cover every endpoint: according to the vLLM security documentation, it protects the /v1, /v2, /inference and /cohere paths but not /metrics, /health, /pause, /resume, /abort_requests or /invocations. Put a reverse proxy in front of the server and allowlist only the endpoints your applications call, and block the administrative and metrics paths from the application network. If you run a multi-node deployment, the same documentation states that traffic between nodes is insecure by default, with no authorization or encryption, so place the nodes on an isolated network and confirm that Ray or the multiprocessing backend cannot be reached from outside it. Treat these as acceptance tests in your migration plan, not as settings you assume are present.

vLLM ships new minor releases roughly every two weeks, so a deployment that tracks the latest tag will change behavior often. Before the first production route goes live, choose one vLLM release, pin the container image or package version, and test the same evaluation suite against it. Ask your platform team to confirm which release your GPU driver, CUDA or ROCm stack and model architecture support, and schedule upgrades as deliberate revalidation events with their own evaluation run and rollback plan. Pinning also gives you a stable reference when comparing results against Azure, where model versions have their own retirement dates.

Before asking for a quote, build a complete inventory of every Azure OpenAI deployment in scope: the deployment name, the underlying model and version, the deployment type (Standard, Global, Data Zone or provisioned), the api-version strings your applications send, and the rate or capacity each deployment carries. Pair that inventory with the contract terms from your Azure agreement, including any provisioned throughput commitment and the retirement date Microsoft has published for each model version. Ask your account team to confirm those dates in writing, because they determine how much time the migration has and which deployments must move first. Also list every consumer: applications, internal tools, batch jobs, evaluation pipelines and third-party products that call the endpoint, along with the owner of each. Missing consumers are the most common source of cutover surprises.

Next, document the residency and logging requirements that the replacement must meet. Confirm which data classes flow through the prompts and completions, where they may be processed, how long request and response content may be retained, and whether your security team needs prompt logging for incident response or must avoid it entirely. These answers decide where the vLLM servers and the LiteLLM gateway run, which log sinks such as Langfuse or Opik receive payloads, and whether the /metrics endpoint or request bodies may leave your network. Record who approves each answer, since a quote should reflect those controls rather than assumptions. Finally, confirm the accelerator side with your infrastructure team before the quote is scoped. Check the vLLM installation documentation against the GPU vendor and driver stack you can actually buy or already operate, and confirm the lead times and export or procurement rules that apply to your region. Verify whether your environment needs a LiteLLM Enterprise license for SSO beyond the free user limit, audit logs, SCIM or secret manager integration, and confirm the current edition terms on the LiteLLM enterprise documentation. Those answers change both the licensing line and the implementation effort.