When the OpenAI API fits and when self-hosted vLLM fits
Most teams reach this decision through a trigger: a renewal, a residency or retention requirement, a deprecation notice from OpenAI, or a per-token bill that keeps climbing as usage grows. The OpenAI API fits teams that accept hosted processing and want no inference infrastructure to run. You get current hosted models, no GPUs to size, and no one paged at night when serving degrades. vLLM fits a different set of constraints: prompts that must stay inside your own environment, model versions you pin and test on your schedule, and an endpoint whose access rules, logging and network path your security team controls. Both paths expose an OpenAI-style interface, so the choice is about who runs the service, not how applications call it.
Start by writing down your data rules. Confirm with your legal and privacy teams whether prompts and completions may be processed by a third party at all, and if so under which contractual and retention terms. Then test model quality on your own tasks, using a fixed evaluation set rather than public benchmarks, because an open-weight model that is weaker on general questions may still be adequate for your extraction or classification work. Next, estimate traffic volume and expected GPU utilization. Steady, high utilization favors owning capacity; spiky or low volume usually favors the API. The cost of vLLM is not a license fee, since vLLM is Apache-2.0 with no paid edition. It is GPUs or other accelerators, the surrounding infrastructure, implementation, and the operators who keep it running.
Finally, decide who will be on call. A self-hosted endpoint makes your team responsible for GPU failures, scheduling, upgrades to the fast-moving vLLM releases, and the reverse proxy that limits which endpoints users can reach, since the vLLM --api-key option does not cover every route and the vLLM security documentation recommends an allowlist in front of it. If no team can own that rotation with clear escalation, the hosted API remains the safer operating choice even when the data rules favor self-hosting. Write these answers down before comparing prices, because they determine which costs and risks you are actually taking on.
Which option fits which scenario
This table matches common operating conditions to the option that usually fits better, with the condition that drives each choice. Read each row against your own constraints, and treat it as a starting point because frontier hosted models may still outperform open-weight models on some tasks, so test both on your own prompts before deciding.
| Scenario | Better fit | Driving condition |
|---|---|---|
| Strict prompt-retention rules | Self-hosted vLLM | Prompts never leave your environment, so no third-party retention terms need to be reviewed or negotiated. |
| Residency in a specific region | Self-hosted vLLM | You choose the region or data center for GPUs, but confirm that every logging and monitoring component also stays there. |
| Need to pin a model version | Self-hosted vLLM | Weights you store and serve do not change unless you change them, although you own the revalidation work for upgrades. |
| Low or bursty traffic | Hosted OpenAI API | Idle GPU capacity still costs money, so per-token billing usually suits sporadic demand better. |
| Steady high-volume traffic | Self-hosted vLLM | High, predictable utilization spreads GPU and operations cost across many requests, so confirm your own utilization numbers. |
| Air-gapped networks | Self-hosted vLLM | The hosted API requires internet access, while vLLM can run inside an isolated network once weights and images are transferred in. |
| Teams without GPU or SRE capacity | Hosted OpenAI API | Someone must size, patch, monitor and staff on-call for GPU serving, which is a continuing responsibility, not a one-time setup. |
| Tasks needing the strongest frontier reasoning | Hosted OpenAI API, tested first | Open-weight models may trail on some tasks, so run an evaluation on your own prompts before choosing. |
Where prompts are processed and who controls the endpoint
With the hosted OpenAI API, prompts are processed in the provider's infrastructure under terms the provider sets. With vLLM, the processing location, the retention settings and the endpoint's access controls are decisions you make and must document. The cards below cover where prompts travel, what is retained, how the endpoint is protected and who answers for each part.
Where prompts are processed
Hosted OpenAI requests are processed in the provider's infrastructure, so your residency position depends on its regional and contractual options. A vLLM server runs in your cloud account, your data center or an air-gapped network, and prompts never leave that boundary unless your own design sends them out. Confirm the exact network path for every application that calls the endpoint, including logging, tracing and any gateway that forwards requests to an external model.
Retention you control
With the hosted API, retention is set by the provider's policy and by any zero data retention approval you hold, so you verify it against contract terms. With vLLM, prompts are retained only where you configure logs, traces or metrics to capture them. Check what your inference server, reverse proxy, gateway and observability stack write to disk, and set retention periods for each store in writing.
Endpoint authentication and exposure
The --api-key option protects the /v1 routes but not /metrics, /health, /invocations, /pause, /resume or /abort_requests, and the vLLM security docs warn against relying on it alone. Place a reverse proxy in front that allowlists only the routes meant for users, and deny the rest by default. Test each unprotected route from outside the trusted network before go-live.
Multi-node traffic isolation
The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Any cluster that spans nodes for tensor or pipeline parallelism needs those nodes placed on an isolated network that clients cannot reach. Ask your platform team to confirm the isolation design, the Ray or multiprocessing ports and who can open firewall rules on that segment.
Compliance evidence and audit
DeployOpen can prepare systems, configuration and evidence for compliance assessments, including access logs, endpoint configuration and retention settings. It does not issue audit reports, attestations or certifications, and independent auditors or certification bodies make those determinations. Ask your assessor early which evidence they will request for the inference endpoint so it is collected from day one.
Your obligations as operator
When you run the model yourself, you take on patching the serving stack, rotating keys, monitoring the endpoint, applying access reviews and answering incidents that the provider handled before. Name an owner for the vLLM release cycle, since minor releases arrive about every two weeks, and define who is on call when the endpoint fails. Write these duties into your runbook before you cut over production traffic.
How a request flows through a self-hosted vLLM endpoint
Applications that already use the OpenAI client library keep their code and change only the base URL, so the request path becomes application, reverse proxy, vLLM server and then the GPU workers. The vLLM server, started with vllm serve, exposes OpenAI-compatible routes such as /v1/chat/completions and /v1/embeddings, which is why most client code moves without rewrites. The point to settle first is that vLLM does not give you the controls on its own: its --api-key option covers the /v1 routes but not /metrics, /health or /pause, and the vLLM security documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Authentication, request logging and rate limiting therefore belong at the proxy or in a gateway in front of it, and your security team should own that layer explicitly.
Map each control to a component before go-live. The reverse proxy enforces which paths are reachable and can terminate TLS and log access, while vLLM's Prometheus metrics at /metrics give you request counts, token throughput, latency histograms and KV cache use for capacity alerts. Keep the metrics and admin-style endpoints off the user network. Inside the cluster, multi-node runs need their own protection, since the vLLM security docs state that communication between nodes is insecure by default and must be isolated on a dedicated network. Confirm the exact flags and endpoint list for the vLLM version you run, because the list of routes and options changes between releases.
A gateway such as LiteLLM can sit between the proxy and vLLM, or in front of both, when you need per-team keys, budgets, TPM and RPM limits, fallbacks to another model and spend tracking. LiteLLM's virtual keys require a PostgreSQL database, and its enterprise documentation describes SSO, audit logs and IP allowlisting as paid features beyond the free tier, so confirm which of those your organization needs before choosing the layer. Adding a gateway adds a component to run, back up and upgrade, so decide whether the controls justify it or whether the proxy and vLLM settings are enough for a single team.
What drives the cost of each side
The hosted API bills you per token and adds contract terms for residency and retention. Self-hosted vLLM replaces that bill with infrastructure, people and upkeep, most of which recurs. Each card names who is paid, whether the cost is one-time or recurring, and what moves it up or down.
Per-token API usage
You pay OpenAI for each input and output token, so the bill follows traffic volume, prompt length and output length. It is the only line that scales directly with use, and it recurs every billing cycle. Check how your prompts are sized, whether you reuse long system prompts, and which models your applications call by default. Ask any quote for the per-token rates that apply to your models and for how they change if you move to a different model tier.
Enterprise contract and residency terms
Data residency, retention limits and enterprise controls are usually negotiated in a separate agreement with OpenAI rather than bought as a self-serve option. These terms recur for as long as the contract runs and can add cost on top of token prices. Ask OpenAI in writing which endpoints each arrangement covers, what approval it requires and what changes if a model is deprecated during the term.
GPU or accelerator capacity
vLLM has no license fee, so the largest self-hosted line is the accelerators that run the model. You can rent them by the hour, reserve them for a term or buy them outright, and each option changes whether the cost recurs or is paid up front. Utilization decides the result: idle GPUs cost the same as busy ones. Ask for the GPU model, memory per card, the number of cards needed for your target model and a utilization assumption.
Networking, storage and redundancy
Model weights, container images, logs and cache layers need storage, and multi-node runs need fast links between nodes. Spare capacity for failover and for model rollouts also adds recurring cost. Check how much storage each model version consumes, whether you keep a second serving pool for failover, and whether your cloud or data center charges for egress or cross-zone traffic.
Implementation and hardening
This is one-time work to deploy vLLM on Kubernetes or bare metal, place a reverse proxy in front of it that allowlists only the endpoints users need, set up authentication and secure the network between nodes. The vLLM security documentation says node-to-node traffic is unauthenticated and unencrypted by default, so isolating those nodes is part of this work. Ask any quote to list each hardening item, the environment it targets and the acceptance tests it will pass.
Monitoring, evaluation and upgrades
Metrics collection, dashboards and alerting are set up once, but they need upkeep. vLLM ships new minor releases about every two weeks: v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026. Each release means reviewing changes, retesting outputs against your prompts and scheduling a rollout, so this work recurs all year. Ask for a release policy that defines which versions you pin and how long you support them.
Engineers who operate it
The largest recurring cost is usually the people who keep inference running, cover on-call rotations, handle incidents and maintain model evaluation suites. Their time is paid whether traffic is high or low. Ask who will be on call for the serving stack, what response times the team commits to in writing and whether a retainer covers after-hours support or only business hours.
What a move from the OpenAI API to vLLM involves
The sequence turns an undocumented set of API calls into a self-hosted vLLM endpoint that has been tested against your own prompts, hardened, and phased in with the hosted API kept as a fallback until the cutover is proven.
Inventory every application and prompt that calls the API
Engineering leads list each service, script and third-party tool that sends requests to the OpenAI API, including the models, endpoints (chat completions, responses, embeddings, audio), token sizes, and any features the code relies on such as function calling, structured output or streaming. Owners of each application confirm the list. The output is a register that shows which workloads are simple to move and which depend on OpenAI-specific behavior. Ask each owner which data classes their prompts contain, because that determines which environment may host them.
Evaluate open-weight candidates on your own tasks
Engineers shortlist open-weight models that vLLM can serve and run them against a sample of real, de-identified prompts from the inventory, scored by the people who own each use case. The evaluation covers answer quality, format reliability, refusal behavior and latency. Model licenses are reviewed by legal before any model is approved for production. The output is a signed-off model choice per workload, with recorded scores and the exact model revision used, so the choice can be reproduced later.
Size and stand up the vLLM cluster
Platform engineers size GPU or other accelerator capacity against the model, context lengths and concurrency measured in the evaluation, then deploy vLLM in your own cloud account or data center using native manifests, Helm or one of the Kubernetes options vLLM documents. Multi-node runs need tensor and pipeline parallelism planned in advance. The output is a running endpoint with Prometheus metrics exposed, a load test report and a capacity note that states the traffic level the cluster was tested at.
Harden the endpoint and isolate the network
Security engineers place a reverse proxy in front of vLLM that allowlists only the endpoints users need, because the vLLM --api-key option does not cover every path, including /metrics, /health and administrative routes. Node-to-node traffic in a multi-node deployment is insecure by default, so those nodes sit on an isolated network. Authentication, TLS and logging are added at the proxy. The output is a test report showing that unlisted paths are blocked from user networks and that inter-node traffic is reachable only from cluster members.
Validate outputs against current prompts
Application owners rerun the inventoried prompts against the hardened endpoint and compare results with the outputs from the hosted API. Prompts are revised where an open-weight model needs different phrasing, and structured-output or tool-call parsers are tested on real responses. Any use case that fails the agreed threshold stays on the hosted API until fixed. The output is a sign-off record per application, listing the prompt version, model revision and the reviewer who accepted the results.
Route a small share of traffic to vLLM
Teams place a gateway, such as a self-hosted LiteLLM proxy, between applications and both backends so that routing is a configuration change rather than a code change. They start with low-risk workloads and send a small percentage of traffic to vLLM, watching error rates, latency, token counts and output quality on the dashboards built from vLLM's metrics. The hosted API stays configured as a fallback. The output is a documented traffic split and a rollback procedure that has been tested once. Moving the detailed cutover steps to a separate migration guide keeps this page focused on the decision.
Retire the hosted path and document operations
Once each workload has held its agreed quality and reliability level, the traffic share moves to vLLM, and the hosted API is removed from production routing but kept in a documented, tested fallback state for as long as the business requires it. Operations owners write runbooks for upgrades, model rollbacks, GPU failures and on-call escalation, and confirm who is paid to respond to incidents, whether internal staff or a support retainer. The output is a decommissioning record, an updated architecture diagram and a named on-call rota.
Questions buyers ask before choosing vLLM over the OpenAI API
What does it cost to run vLLM, and what drives that cost?
vLLM itself has no license fee, so the cost comes from accelerator capacity, the surrounding infrastructure, implementation and the people who operate it. GPU or other accelerator capacity is usually the largest recurring line, and it depends on the model size, the precision you serve at, the context lengths your users send and how many requests you must handle at peak. Ask any quote to separate one-time items (design, integration, hardening, initial load testing) from recurring items (compute, storage, monitoring, on-call coverage and upgrades). Compare those against the per-token spend you pay today, using your real traffic profile rather than a list price.
How long does a vLLM implementation take, and what does it depend on?
The timeline depends on the scope you agree in writing, not on a standard duration. The main variables are how many models you serve, whether you run a single node or a multi-node cluster with tensor or pipeline parallelism, whether you need a Kubernetes deployment with its own autoscaling, and how much of the endpoint hardening, identity integration and logging you require. Procurement and GPU lead times often set the critical path more than software setup does. Ask for a fixed-scope plan with milestones and the assumptions behind each one.
What does the work need from our own team?
Plan for people who can own the infrastructure after handover. That usually means platform or infrastructure engineers who can operate GPU nodes and Kubernetes, someone who owns the reverse proxy and authentication in front of the endpoint, and an engineer who understands model evaluation and prompt behavior. vLLM's security documentation says node-to-node communication in a multi-node deployment is insecure by default and must sit on an isolated network, so network and security staff need to be involved early. Ask your implementation partner to name the roles each phase requires and which ones your team must staff.
Can our team run vLLM without outside help?
Yes, if you have the skills described above and time to build them. vLLM is open source under Apache-2.0, its documentation covers serving, parallelism, metrics and Kubernetes deployment, and you can start with a single node to learn the operating model. The gap usually appears in production hardening: placing the endpoint behind a reverse proxy that allowlists only user-facing paths, since the --api-key option does not protect every route, plus capacity planning, load testing and on-call. Decide which of those you will own in-house and which you will buy as fixed-scope work or retainer support.
Is vLLM free to use, and what does it not include?
The vLLM software is licensed under Apache-2.0 and has no paid edition, so you do not pay a license fee to run it. What it does not include is everything around the engine: the GPUs or other accelerators, the cloud or data center, the reverse proxy and identity layer, monitoring and alerting, backup of configuration and model artifacts, model weight licensing review, and the staff who operate it. Confirm with your counsel that the specific open-weight model you choose allows your intended use, because the model license is separate from the serving engine's license.
How does model quality compare with the OpenAI API, and how do we test it?
Quality depends on the model you choose and the task, so no general comparison should decide the question for you. Build a test set from your own prompts, expected outputs and failure cases, then run the same set against the hosted model and each candidate on vLLM. Score results with the criteria your business uses, such as accuracy, format compliance, refusal behavior and latency at your expected concurrency. Keep the OpenAI model as a baseline during the trial, and rerun the suite whenever you change the model, the prompt template or the serving configuration.
How do we keep up with vLLM's fast release cadence?
vLLM ships new minor releases about every two weeks, for example v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, according to its release history. Do not upgrade on release day. Instead, set a rule that you pin a tested version, track release notes, rerun your evaluation suite and performance tests on a candidate version in a staging environment, and then promote it on a schedule you control. Ask any support provider how they handle version pinning, regression testing and rollback before you sign.
How DeployOpen helps evaluate, implement and harden a vLLM endpoint
DeployOpen is a Chainscore Labs company that plans, implements, hardens, backs up, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a vLLM endpoint, that means working through the decision with your security, platform and application teams, then building the serving stack, gateway and controls in your environment.
How we scope
- Engagements are fixed-scope, with scope and deliverables agreed in writing before work starts.
- Scope maps which applications call the OpenAI API, which endpoints they use, and which data classes they send.
- Scope sets the target environment for GPUs, network zones, identity provider and logging destinations.
- Scope names the evaluation criteria for model choice, output quality and residency, agreed with your owners.
How we build
- We deploy vLLM with its OpenAI-compatible server in your cluster, with GPU capacity sized to the agreed workload.
- We place a reverse proxy in front of vLLM that allowlists only the endpoints meant for users, not admin or metrics paths.
- We wire authentication, rate limits, logging and metrics into your existing monitoring, and test multi-node traffic isolation.
- We run security testing of the endpoint and its gateway before it carries production traffic.
How we hand over
- Handover includes architecture diagrams, configuration records, runbooks for restarts, scaling and model swaps.
- We document the controls and evidence that support compliance readiness, without issuing audit reports or certifications.
- Teams receive a rollback path to the hosted API during cutover, agreed in the written scope.
- Knowledge transfer covers version upgrade steps, since vLLM publishes minor releases roughly every two weeks.
How we support
- Ongoing support, upgrades and continuous security testing are available as retainers.
- Retainers can cover on-call coverage terms, which you should confirm and define in the written scope.
- Support scope covers vLLM version upgrades, security patches, model changes and capacity reviews.
- Quotes follow from the written scope, and DeployOpen does not publish a price list.
