Why Azure OpenAI retirement dates force revalidation work on your schedule
Self-hosting open-weight models on vLLM behind a gateway lets you choose when a model version changes, at the cost of running the serving stack yourself. Azure OpenAI general availability models retire 18 months after launch, retirement dates cannot be extended, and provisioned deployments must be migrated by hand, so the date is set by Microsoft and the work lands on your team. A version change is not a single swap. It touches the model weights, the vLLM server version that loads them, the gateway configuration that maps application model names to deployments, the prompts tuned against that model, and the evaluation baselines and dashboards that show whether quality moved. If any one of these is left out of the change record, the failure tends to surface later in production.
The recurring cost is revalidation, and most of it is engineering time. Each retirement forces a team to rerun its evaluation set on the replacement, compare outputs on prompts that matter to the business, update regression tests, and record the decision for the security and compliance reviewers who will ask about it. Ask anyone quoting this work to scope these steps explicitly, because they repeat every time a model is retired. vLLM ships new minor releases about every two weeks, so an unpinned serving image would change far more often than any retirement notice does. Pin the vLLM version in your image tag, and move to newer vLLM releases through the same test gate that a model change passes.
Pinning changes who decides. Once model weights are stored in your own object storage with recorded checksums, the vLLM version is fixed in your deployment manifests, and the gateway maps each application alias to a named deployment, a version change becomes a planned change that you schedule and approve. The trade-off is that patches, GPU capacity, failover and on-call now sit with you. Keep the gateway current as well, since LiteLLM releases are supported for its four most recent stable minor versions, which makes that upgrade cadence a planning input. Confirm the license terms of each open-weight model with its publisher before you commit, because permission to self-host varies by model and is not implied by the serving engine's Apache-2.0 license.
What a model retirement cadence requires from your stack
A retirement cadence forces a version change on a fixed schedule, whether or not your team is ready. The four requirements below decide whether that change is a routine swap or an emergency migration, and each one depends on the others.
A stable endpoint applications call
Applications should call one fixed, OpenAI-compatible address, never a model-specific URL. vLLM's server exposes /v1/chat/completions, /v1/responses and /v1/embeddings, so client code written for the OpenAI API can stay unchanged. Behind that address, you swap the model without touching every application. Confirm which endpoints your client libraries call, including any that vLLM does not serve in your version.
A gateway that maps names to models
A gateway such as LiteLLM gives applications a logical model name, for example a name your team controls, and maps it to a concrete open-weight model behind the scenes. Changing the mapping is then a configuration change with a rollback path, not a code release. Ask whether your gateway can route one logical name to a canary share of traffic before full cutover.
An evaluation set from your own prompts
Vendor benchmarks do not show how a replacement model handles your invoices, tickets or clinical notes. Build a fixed set of representative prompts with expected outputs or grading rules, and run every candidate version against it before a switch. Store the scores with the version so a later retirement can be compared against a known baseline. Keep the set private, since it often contains regulated data.
Per-application model allowlists
Not every application should reach every model. LiteLLM virtual keys can restrict a key to specific models, and they support per-key, per-user and per-team budgets and rate limits. Allowlists let you move one application to a new version while others stay pinned, which limits the blast radius of a retirement. Virtual keys require a PostgreSQL database, so plan that dependency with the gateway.
A record of which version served each request
Audit questions after a model change depend on knowing which model and version produced each answer. Log the model identifier, gateway route and request time with every call, and send those traces to your observability stack. vLLM exposes Prometheus metrics at /metrics, which show throughput and latency per server, but per-request attribution still depends on your gateway logs. Confirm the retention period for these records with your compliance owner.
Where the revalidation cost actually sits
The expensive part of a model retirement is rarely the endpoint change. Pointing a deployment name at a new model is a configuration edit. The work that follows is revalidation: re-running your prompts against the replacement, comparing outputs with the retired model, updating runbooks and alert thresholds, and getting the change re-approved under internal model risk or security review. Each step is paid for in people's time, and most of that time sits outside the platform team that performs the migration.
Platform engineers own the inventory of deployments, the migration itself and the rollback path. Application owners own the evaluation sets, the judgment on whether a changed answer is acceptable, and the retest of their integrations, including any structured output parsers and prompts tuned to one model's quirks. Risk and compliance own the re-approval, which may require new documentation of the model, its version and its intended use. Ask each group to estimate its share before the next retirement date is announced, because the largest share is often the one nobody budgeted for.
Several factors push this effort up. The number of deployments matters, but so does the number of applications sharing one deployment, since a single model change then forces retests across every consumer at once. The volume of output that must be re-reviewed by people is another driver, especially for regulated content where a sample review is not enough. Provisioned deployments add work because they are migrated by hand. Measure this on your own workloads: count the deployments and consuming applications, time one dry-run revalidation on a representative prompt set, and record how many reviewer hours the last change required. No effort figures are supplied here, so treat any external estimate as a starting point to test against those numbers.
Which open-source options hold a model version still
This table compares vLLM, LiteLLM and three observability options against the job of keeping one open-weight model version fixed and routing traffic around it. Read each row for what the component pins, what it leaves to you, and where its license or access controls need extra work.
| Component | How it helps pin or route versions | What it does not do by itself | Trade-offs to weigh |
|---|---|---|---|
| vLLM (serving engine) | Serves the exact weights you choose from a fixed release tag through OpenAI-compatible endpoints, so the model only changes when you change the deployment. Apache-2.0 licensed. | The --api-key option does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, so a reverse proxy allowlisting user endpoints is still needed. | Minor releases arrive about every two weeks, so pin the version and schedule upgrades. You own GPU capacity, patching and on-call work. |
| LiteLLM (gateway) | Gives applications one endpoint and can map a logical model name to a specific vLLM deployment, so application code does not change when the backend version does. MIT licensed outside its enterprise directory. | Routing and fallbacks do not validate that a replacement model behaves the same. Virtual keys need PostgreSQL, and multi-instance setups need Redis 7.0 or newer. | Some items, including SSO beyond five users and SCIM, need an enterprise license. Confirm which features your environment needs before you choose. |
| Langfuse (tracing and evaluation) | Records model name, version and prompt per request, so you can compare outputs across versions before cutover. Core is MIT licensed outside its ee directories. | Does not decide which version to use or run regression suites by default. Advanced features need an enterprise license key. | Needs PostgreSQL, ClickHouse, Redis or Valkey and object storage, which adds stateful services to operate. |
| Opik (tracing and evaluation) | Covers tracing, evaluations and prompt management for side-by-side tests of a pinned model against its replacement. Apache-2.0 licensed, and the README states the full platform can be self-hosted without a commercial license. | Does not pin model weights or manage GPU serving. Confirm the self-hosted deployment footprint with the project before you size it. | Smaller ecosystem of integrations than some alternatives, so confirm connectors for your gateway and frameworks. |
| Phoenix (tracing and evaluation) | Built on OpenTelemetry and OpenInference, so traces can record model versions for comparison across retirements. Note: licensed under Elastic License 2.0, which is not OSI-approved and bars offering it as a hosted or managed service. | Does not provide serving, routing or version pinning. Internal self-hosting is permitted, but confirm that your use fits the license terms with counsel. | If your policy requires OSI-approved licenses, Langfuse or Opik may fit better. Confirm which model versions you will pin, since this page does not list model availability. |
How to pin and operate a model version under a retirement schedule
This sequence moves each Azure OpenAI workload onto a pinned open-weight model that your team controls, so version changes happen on your schedule after your own evaluation, not on a vendor retirement date.
Inventory deployments and the applications behind them
The platform owner lists every Azure OpenAI deployment, its model and version, its API version, its request volume, and the applications, scripts and agents that call it. Each caller is tagged with an owner and a business function, and provisioned deployments are flagged separately because they carry their own migration constraints. The step is done when the inventory names every caller and every deployment, and each owner has confirmed the list. Ask your Azure account team to confirm the retirement dates that apply to each model version you run, since those dates are the trigger for this work.
Select candidate open-weight models and pin exact revisions
The ML platform lead shortlists open-weight models that fit each workload's task, language needs and hardware budget, then records the exact model identifier and the commit or revision hash downloaded from the model repository. Weights are stored in your own artifact store with checksums, and the license terms of each model are reviewed by legal before use. The output is a model register with one pinned revision per use case. Pinning the revision, not a floating tag, is what keeps the model from changing underneath your applications.
Stand up vLLM and expose only allowlisted endpoints
The platform engineers start vLLM with the pinned revision inside your own network, using the vllm serve command with an API key set. vLLM's security documentation says the --api-key option does not cover every endpoint, so a reverse proxy sits in front and forwards only the routes that users and gateways need, such as the OpenAI-compatible chat and embeddings paths. Administrative and metrics routes stay reachable only from the monitoring network. The step is complete when a request to an unlisted route is refused at the proxy and the pinned model answers an allowed request.
Place LiteLLM in front with virtual keys and fallbacks
The platform team deploys the LiteLLM proxy as the single OpenAI-compatible endpoint for applications. Each application or team receives a virtual key restricted to specific models, with budgets and rate limits. Fallbacks are defined only to models that have passed evaluation, so traffic never shifts silently to an unvetted version. LiteLLM virtual keys need a PostgreSQL database, and once more than one proxy instance runs, Redis 7.0 or newer is needed for shared rate limits, router state and cache. LITELLM_MASTER_KEY, which must start with sk-, is kept in a secret manager, not in configuration files. The step closes when a test key can call the pinned model and is blocked from models outside its list.
Build an evaluation set from your own prompts
Application owners and domain experts collect representative prompts from production logs, with personal data removed, and label expected behavior for each. The set covers edge cases and the outputs your applications depend on, such as JSON structure or refusal behavior. The current Azure OpenAI version and each candidate open-weight revision answer the same prompts, and reviewers score the differences. The output is a signed evaluation report per use case that names the pinned revision it approves. Public benchmarks alone do not establish that a model behaves correctly for your workload.
Scrape vLLM metrics for latency and KV cache use
The observability engineer configures Prometheus to scrape the vLLM /metrics endpoint from the monitoring network, then builds dashboards for request counts, token throughput, latency histograms and KV cache usage. Alerts go to the platform on-call channel when latency or cache pressure crosses thresholds that the team sets from its own evaluation results. Histogram buckets can be configured in vLLM, so the team confirms they match the latency targets of each workload. The step is done when dashboards show live data from every serving instance and an alert has been tested end to end.
Define a version change procedure with approvals
The change board writes a runbook for replacing a pinned revision: a new revision is evaluated against the set, performance is measured on staging, the gateway routing is changed, and a rollback path to the prior revision is kept. Approvals come from the application owner, the security reviewer and the platform lead, and each change is recorded with the revision hash, evaluation report and approver names. The procedure is complete when a dry run on a non-production route has been recorded from start to finish, including the rollback.
Schedule a review ahead of any upstream change
The platform owner keeps a calendar of upstream events that matter, including vLLM minor releases, which the project ships about every two weeks, and any security advisories for vLLM or LiteLLM. Each review checks whether a release affects the pinned stack, and upgrades are planned as normal change requests rather than emergency work. LiteLLM supports its four most recent stable minor versions, so the team confirms its pinned version stays in support before the next planned upgrade window. The output is a dated review log that shows each decision and the owner who made it.
How the change differs for standard and provisioned workloads
Standard deployments are the simplest case. Applications call a named deployment, and the model behind it stops serving on the retirement date for its version. Nothing in the commercial arrangement changes, so the work arrives as a deadline: someone must point the deployment at a successor model and revalidate outputs before the old model stops answering.
- Retirement date: The deployment stops serving on the published retirement date, and the date cannot be extended. Check the schedule for every model in use, not only the newest one.
- What to test: Replay a fixed set of production-like prompts against the successor and compare refusal behavior, JSON or tool-call structure, response length and the answers your business rules depend on.
- Parameter drift: A successor can respond differently to the same system prompt, so retest temperature, output limits and any structured output settings along with the text.
- What self-hosting changes: The team pins a specific open-weight model and version on vLLM, so the date comes from your own change calendar. Upgrades happen once your evaluation passes, on a schedule your change process sets.
- Ongoing effort: Pinning moves the work to your side. You choose when to adopt a new vLLM minor release, and the project ships them about every two weeks, so the evaluation suite must be rerun each time.
Confirm which deployments still reference a model version with a published retirement date, and how long your evaluation set takes to run end to end.
Provisioned deployments reserve throughput, and they must be migrated by hand. When the model retires, the reserved capacity does not move to the successor on its own. The team creates a new deployment, confirms capacity, switches traffic and retires the old one inside the change window. Commitment terms and migration effort run in parallel, so a late cutover can leave paid capacity serving a model that has stopped answering.
- Manual migration: Create the successor deployment, confirm it has the capacity your workload needs and move each client over. Nothing is moved for you.
- What to test: Load-test the successor with a request mix that matches peak traffic, including prompt length, output length and concurrency, because throughput depends on all three.
- Cutover risk: Run old and new deployments side by side, shift a small share of traffic first and keep a rollback path until error rates and quality checks match the baseline.
- Commercial terms: Ask Microsoft in writing how reservation terms apply during a model change, and whether any capacity is credited or must be purchased again, before the retirement window opens.
- What self-hosting changes: The team sizes accelerators for a model it selected, so the replacement is a planned capacity change on hardware it controls. The team sets the date, and the capacity planning is done in-house, so it must be repeated whenever the model changes.
Confirm in writing what happens to reserved capacity at retirement, and how much parallel running your change process needs before the old deployment is removed.
A single deployment often serves several applications, teams or internal tools. A retirement then touches every consumer at once, and each owner may test on a different schedule. The platform team usually owns the deployment name and acts as the coordination point, but each application owner should own the acceptance decision for their own prompts.
- Consumer inventory: List every application, script and partner integration that calls the deployment, with its owner and the prompts it depends on. Many shared deployments have no current list.
- Per-consumer tests: Each owner signs off on their own evaluation set. A successor that passes one team's summarization tests can still fail another team's classification or extraction work.
- Single change window: Agree one cutover date that every owner accepts, because a shared name cannot point at two models at once without a routing layer.
- Routing layer: A gateway such as LiteLLM can map one public model name to a pinned open-weight version, so consumers keep their configuration while the platform team controls the change. Confirm which routing and key features your edition includes, since some enterprise features need a license. See the LiteLLM routing documentation for the options.
- What self-hosting changes: The platform team sets the version and the date for every consumer, and it can keep the previous version online under a separate name until the last owner signs off.
Ask each consumer owner to name their acceptance test and sign-off date before the cutover is scheduled.
Evidence model risk and security reviewers ask for
Every model version change on self-hosted infrastructure should leave a trail a reviewer can follow from the pinned weights to the traffic they served. The six records below connect the vLLM server, the LiteLLM gateway and the observability stack, so the team can show what changed, who approved it, and what users were routed to during the period.
Pinned model identifier and revision
Record the exact model repository, the commit or revision hash of the weights, the tokenizer and the vLLM version that served them. Azure OpenAI retirement dates cannot be extended, so the record should state which version is in use and when it was pinned. Store the identifier in the deployment manifest and in version control, not only in a chat thread. Confirm that the weights checksum matches the copy in your artifact store before each rollout.
Evaluation results on your own prompts
Keep the scored output of your regression set for the outgoing and incoming model, run against the same prompts, parameters and grading rules. Store the prompt set version, the scores per task and any cases where behavior changed. Reviewers will ask whether the replacement was tested on the workload that matters to the business, not on public benchmarks. Observability tools such as Langfuse or Opik can hold the traces and evaluation runs that produced these scores.
Approval record for the change
Record who reviewed the evaluation results, who approved the switch, the date, the scope of the change and the rollback target. Separate the person who runs the rollout from the person who signs off on it where your change policy requires it. This record lives outside the serving stack, in your ticketing or change management system, and should link to the evaluation run and the pinned revision. DeployOpen can help define the workflow, but the approvals are your organization's decisions.
Routing and key configuration in effect
Export the LiteLLM model list, router strategy, fallback order, and virtual key restrictions that were active during the period. A key limited to specific models, or a fallback that silently sends traffic to a different model, changes what was served. Store the configuration as versioned files and record the hash of each applied version. Confirm that the master key sits in a secret manager and that its rotation is logged.
Access logs for the period
Collect gateway request logs that show the caller, the virtual key, the model name, the timestamp and the outcome for each call, along with the vLLM server logs. Confirm that the vLLM API key protects only the endpoints meant for users and that a reverse proxy allowlists the rest. Access logs should be retained for the period your policy requires and stored where administrators cannot edit them without leaving a trace.
Serving metrics for the period
Export the vLLM Prometheus metrics for request counts, token throughput, latency histograms and KV cache use, covering the window before and after each change. These figures show whether the new version performed within the capacity and latency the team accepted. Keep the dashboards and the exported series alongside the approval record. Ask your monitoring team to confirm retention and to store the metric definitions that were in effect at the time.
How much effort does a model retirement cycle take, and what do we need to provide?
How much effort does a model retirement cost us in our environment?
The effort depends on how many deployments you run, how many applications call them, and how much revalidation each one needs before traffic moves. Each retirement triggers the same chain: pick a replacement, run your own prompts against it, compare outputs, update configuration and redeploy. Provisioned deployments are migrated by hand, so their cost grows with the number of workloads tied to them. Self-hosting changes the cadence because you choose when a pinned open-weight version is replaced, which turns a forced event into a planned release. Ask any outside firm to size the work from your deployment inventory, your evaluation coverage and your change-control rules, not from a generic estimate.
How long does a first self-hosted deployment take?
The timeline depends on the written scope, not on a fixed duration that applies to every organization. The main variables are the number of models, the GPU capacity you already have, the number of gateway routes, the identity and logging integrations, and how much evaluation work you want before cutover. A scope document should list each deliverable, the environment it lands in, the acceptance tests and the dates both sides agree to. Most DeployOpen engagements are fixed-scope, so the timeline is agreed in writing before work starts. Ask for the milestones and the conditions that would move them.
What does the work need from our own team?
Plan for five groups of inputs. First, GPU or other accelerator capacity in a location you control, sized to the models you pin. Second, identity: an SSO provider such as Microsoft Entra ID, Google, Okta or a generic OIDC source for the gateway admin UI. Third, network: a private subnet for vLLM nodes, a reverse proxy that allowlists only user-facing endpoints, and egress rules for any providers the gateway still calls. Fourth, a secret manager for LITELLM_MASTER_KEY and provider credentials. Fifth, named people on call for the GPU hosts, the gateway and the database. The vLLM security documentation says inter-node traffic is unauthenticated by default, so network isolation is a required input, not an option.
Can our own engineers run the retirement cycle without an outside partner?
Yes, if the team has the skills and time, and many do. The work needs Kubernetes or container operations for vLLM and LiteLLM, PostgreSQL administration, Redis for shared state once more than one gateway instance runs, Prometheus-style monitoring, and evaluation design so that a replacement model is tested against your own prompts. It also needs change management that ties each version pin to a documented test result. The honest question is whether those skills are staffed for every retirement, not only the first one. Ask an outside firm to separate one-time build work from ongoing upgrades so you can decide which parts to keep in-house.
Does self-hosting the gateway and serving layers lock us in?
The licenses limit lock-in risk. vLLM is Apache-2.0 licensed, and the LiteLLM code outside its enterprise directory is MIT licensed, so you can run, modify and move those components without a license fee. The lock-in risk sits elsewhere: in the models you choose, in the evaluation datasets you build, in the configuration that encodes routing and budgets, and in any enterprise-only features you adopt. Keep configuration in version control, keep evaluation sets portable, and check each model's own license before you pin it. Confirm the exact terms with the project maintainers and your legal team.
Which LiteLLM Enterprise features do we need beyond the free tier?
The free tier covers the gateway, virtual keys, budgets, routing and logging. From v1.76.0, SSO in the admin UI is free for up to 5 users, and an Enterprise license is needed beyond that. Enterprise adds SCIM provisioning, JWT authentication with identity provider tokens, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations and team-level guardrails. If you have more than five administrators who need SSO, or an audit requirement, price the license into your plan. Confirm current terms directly with LiteLLM, since the enterprise documentation governs.
What should a quote for this work include?
A quote should follow from a written scope and list each deliverable. Ask for the following: the deployment inventory and assumptions; the environments covered; the GPU, network and identity prerequisites you must supply; the acceptance tests and evaluation method for each model replacement; the backup, upgrade and documentation deliverables; the handover plan; and what is one-time versus recurring. Recurring items usually include upgrades, monitoring and on-call support, which DeployOpen offers as retainers. Ask for the exclusions in writing, including any third-party license costs such as an enterprise key. DeployOpen does not publish a price list, so the quote is the only fee document.
How DeployOpen plans and runs version-pinned model serving
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including the vLLM serving layer and gateway that keep model versions under your control. The work starts with a written scope that defines which deployments, models and applications move, and ends with a handover your own team can operate.
How we scope
- Inventory every Azure OpenAI deployment, model version, API version and application that calls it before any work starts.
- Agree in writing which open-weight models, serving hosts and gateway configuration are in scope, with deliverables and timelines.
- Define the revalidation test set on your own prompts so each pinned model version has documented acceptance criteria.
How we build
- Deploy vLLM in your cloud account, data center or air-gapped network with its OpenAI-compatible endpoints and pinned release tags.
- Place a reverse proxy in front of vLLM that allowlists only user-facing endpoints, since the --api-key option alone does not cover every path.
- Configure a LiteLLM gateway with model aliases, virtual keys and routing so applications keep stable names while versions change under your control.
How we hand over
- Document the pinned model and vLLM versions, the upgrade procedure and the evidence produced by each revalidation run.
- Run a joint test cycle with your engineers on your own prompts, then hand over runbooks, configuration and monitoring dashboards.
- Agree in writing how version changes, rollbacks and retirement notices are tracked, so no deployment changes without a recorded decision.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover, covering new vLLM and LiteLLM releases.
- Schedule revalidation when a new open-weight model or vLLM release is proposed, and test it against the pinned acceptance criteria.
- Prepare systems and records for compliance assessments; independent auditors and certification bodies issue any audit reports.
