Why teams already on Custom Model Import ask about self-hosting vLLM
The question is whether running vLLM in your own infrastructure gives you more control at an operating cost you can accept. Many teams reach it after importing open-weight model weights into Amazon Bedrock Custom Model Import and then finding that the managed path limits them in specific ways: billing accrues per minute in Custom Model Units, a model that scales to zero pays a cold start before it answers the first request, and the platform decides when a model's serving stack, quantization and runtime change. The short answer is that vLLM is open source under Apache-2.0 and has no paid edition. The trade is therefore not license cost against no license cost. It is infrastructure and operating work that you own against the convenience Bedrock absorbs for you.
Before comparing, pull the facts from your own account. Export the Custom Model Import billing and invocation patterns for the last quarter, and note how many hours each imported model is idle versus busy. Record the exact model artifact, tokenizer and quantization you imported, because those are what you would need to reproduce on vLLM. Confirm with AWS which Custom Model Import regions and instance types your account uses, and whether any compliance requirement depends on the managed service itself. A self-hosted deployment keeps the model on your compute, but it also makes you responsible for GPU capacity planning, patching the inference stack, and keeping a warm replica where a cold start would breach your latency target. The sections that follow show where those responsibilities fall, and what each one costs.
What the self-hosting decision requires you to own
Bedrock Custom Model Import hides the serving layer, capacity planning and most of the access controls behind an AWS account. Serving open-weight models on vLLM hands those decisions back to your team, from accelerator capacity to the network between nodes. The cards below list what to decide and verify before you compare costs or cut over.
GPU and accelerator capacity
Your team now chooses the accelerator type, the number of devices and how many replicas stay running at all times. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, as well as on several CPU and Apple silicon platforms, so the hardware choice affects which model sizes fit. Under Custom Model Import, AWS sets this capacity for you and bills per minute. Ask your infrastructure team to size for peak concurrency, not average load, and to confirm the cold start and warm-up behavior of the chosen hardware.
The serving engine and its releases
vLLM is Apache-2.0 licensed with no paid edition, so the software itself carries no license fee. Your team owns upgrades, and the project ships new minor releases about every two weeks. Each upgrade can change scheduling, quantization support and API behavior, so pin a version, run your evaluation set against each candidate and keep a rollback path. Bedrock Custom Model Import does not expose this choice to you.
OpenAI-compatible API surface
The vLLM server exposes /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, so applications written for the OpenAI API can point at it with a base URL change. Confirm which endpoints your current Bedrock callers use, because Bedrock's own request formats differ. Check tool calling, streaming and token usage fields against your application tests before you assume parity.
Endpoint authentication and exposure
The --api-key option protects only the /v1, /v2, /inference and /cohere paths. It does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the vLLM docs warn against relying on it alone. Place a reverse proxy in front that allowlists only the endpoints your users need, and block the control paths from application networks. Bedrock uses IAM for this, so your access model must be rebuilt.
Metrics and observability
vLLM serves Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. That endpoint is not protected by --api-key, so restrict it to your monitoring network. Confirm that your dashboards and alerts track queue depth and KV cache pressure, since those signals explain most latency problems that Bedrock's own metrics would otherwise show.
Multi-node networking
A model too large for one node runs across nodes with pipeline parallelism, using Ray or vLLM's multiprocessing backend. The vLLM security docs state that communication between nodes is insecure by default, with no authorization or encryption. Place cluster nodes on an isolated network segment, and add encryption and access controls where that segment cannot be trusted. Ask your architects to document this design before the first production deployment.
How Custom Model Import billing and scaling differ from a warm vLLM deployment
Custom Model Import charges for the capacity your imported model uses, metered per minute in Custom Model Units, on top of the usual AWS charges for data transfer, logging and storage of the model artifacts. A warm vLLM deployment flips that structure: you pay for GPU or other accelerator instances whether or not requests arrive, plus storage, networking, monitoring and the engineering hours that keep the service running. Neither model is cheaper in general. The answer depends on how many hours per day the model is actually busy.
- Recurring Bedrock charges: Per-minute capacity billing continues for as long as the imported model is active and available, so idle periods still produce cost unless the model is scaled down.
- Recurring self-hosting charges: Reserved or on-demand accelerator hours run continuously for a warm replica, and they recur for every replica you keep.
- One-time self-hosting work: Model conversion, serving configuration, load testing, gateway integration and hardening are one-time efforts, though they may be repeated at each model upgrade.
- Cost drivers: Utilization, the number of replicas needed to hold latency, the accelerator generation and the share of your team's time that operations consumes.
- Quote request: Ask AWS to confirm current per-minute rates for your region and model size in writing, and ask your implementation partner to price the one-time work and the recurring operating hours separately.
Before comparing totals, confirm the current Custom Model Units rate with AWS, because pricing changes and varies by region.
Custom Model Import can release capacity when a model receives no traffic, which removes idle charges but introduces a delay when the next request arrives. That delay, the cold start, depends on the model size, the weights loaded and how the service provisions capacity on demand. Neither AWS documentation nor this page provides a fixed duration, so measure the delay for your own model and request pattern before deciding. A warm vLLM deployment avoids the cold start by keeping weights resident in accelerator memory, at the cost of paying for that idle capacity.
- Scale-to-zero behavior: Confirm with AWS how long the model remains loaded after the last request and what triggers a scale-down.
- Cold start measurement: Time the first request after an idle period, across several model sizes and at your peak concurrency, and record the p95 value rather than an average.
- Warm pool option: A minimum replica count on vLLM removes cold starts for user-facing paths, while batch jobs may tolerate them.
- Health and readiness: vLLM exposes a /health endpoint, so probes can keep traffic away from a replica that has not finished loading.
- Request routing: Decide whether a gateway queues requests during a cold start or returns an error that the application must retry.
Record the cold start distribution and the traffic gaps in your logs before you choose a scaling policy for production.
Capacity planning answers how many replicas, of what size, you need to meet peak demand within your latency target. With Custom Model Import, AWS selects and manages the underlying capacity, so your planning focuses on quotas, per-minute throughput limits and the account and regional limits that AWS confirms for your model. With vLLM, you size accelerators, memory for the KV cache, parallelism across GPUs and the number of replicas, and you verify those choices with load tests. vLLM supports tensor parallelism within a node and pipeline parallelism across nodes, and multi-node runs need an isolated network, so the design work is significant.
- Throughput measurement: Replay real prompt and completion lengths against a candidate configuration and record tokens per second, time to first token and queue depth.
- KV cache headroom: Watch the KV cache metrics vLLM exposes at /metrics, because cache pressure is the usual limit on concurrency.
- Headroom for failover: Keep enough spare capacity to absorb the loss of one replica or node without breaching your latency target.
- Upgrade capacity: Plan a second replica or a blue-green window for each vLLM minor release, which ships about every two weeks.
- Quota confirmation: Ask AWS to confirm the account and regional quotas that apply to your imported model, and the process for raising them.
Run a load test on your own traffic profile, and confirm all quota and rate figures with AWS and your hardware vendor before committing to either option.
Which option fits which operating model
This table compares four ways to run open-weight models for a team that currently uses Bedrock Custom Model Import. Read each row against your own constraints on billing, scale to zero, version control and quantization. The main takeaway is that moving to vLLM shifts control and operating work to your team, while the LiteLLM layer adds a single endpoint with a license boundary you need to plan around.
| Option | How it meets the constraint | Who controls versions and quantization | Trade-offs |
|---|---|---|---|
| Bedrock Custom Model Import | Imported weights run on AWS-managed capacity, billed per minute of Custom Model Unit use, and can scale down when idle | AWS controls the serving runtime and its upgrades, and you control the imported weights and the format you prepare | Cold starts after idle periods, billing tied to AWS-defined units, and model support limited to what the service accepts |
| vLLM on EC2 or EKS | You size GPU node groups, keep replicas warm, and expose an OpenAI-compatible endpoint inside your AWS accounts | Your team pins the vLLM release, chooses the quantization scheme and decides when to upgrade | You own GPU capacity planning, autoscaling, node security and upgrades, and idle GPU time is paid for unless you scale down |
| vLLM on your own hardware | Servers in your data center or air-gapped network run vLLM with no cloud metering | Your team controls every version, quantization choice and rollout window, including offline weight transfer | Hardware purchase, power, cooling and spare parts are yours, and multi-node runs need an isolated network because inter-node traffic is unauthenticated by default |
| vLLM behind LiteLLM | LiteLLM gives applications one OpenAI-compatible endpoint with virtual keys, budgets, rate limits, fallbacks and logging | Your team controls the gateway configuration and model routing, while vLLM versions and quantization stay with your serving team | The MIT core covers the gateway, but Enterprise SSO, SAML and audit logs sit in the enterprise directory under a separate license, so confirm these needs before relying on them |
Who controls model versions, quantization and upgrades
With Custom Model Import, your team supplies the model weights, but AWS controls the serving runtime that loads and executes them. You do not choose the inference engine release, its kernels or the defaults it applies, and any change to that runtime happens on AWS's schedule. Self-hosting reverses this split. With vLLM, your team pins three things explicitly: the model revision (the exact weights and tokenizer files), the quantization format (for example, the scheme used to produce the weights you load) and the vLLM release that serves them. Each pin is a decision you own and a record you must keep, including who approved it and why.
Upgrades become a recurring task rather than a one-time event. The vLLM project ships new minor releases about every two weeks, for example v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, according to its GitHub releases page. You are not obliged to follow each one, but a team that stays on an old release will eventually need to move, and a jump across several releases is harder to test than a series of small ones. Plan a standing upgrade cycle that reruns your regression prompts, checks the /metrics output for latency and KV cache changes, and confirms the /v1/chat/completions behavior your applications depend on before a new release reaches production.
Quantization deserves its own gate. A quantized model can produce answers that differ from the full-precision version in ways that a generic benchmark will not reveal, so any quantization choice must be validated against your own evaluation set: the prompts, expected answers and failure cases from your actual workloads. Treat a quantization change as a model change, with the same sign-off as a new model revision. Ask your infrastructure and evaluation owners to confirm which quantized formats your chosen vLLM version supports on your accelerators before you commit to a format, because support depends on the hardware and release in use.
How to move from Custom Model Import to a vLLM deployment you control
This sequence takes a team from its current Custom Model Import setup to a vLLM deployment it sizes, secures, monitors and cuts over to on its own schedule, with a checkpoint at every stage.
Inventory imported models and traffic
Your platform or cloud team lists every model imported through Custom Model Import, the weights and tokenizer files each one uses, the applications and service accounts that call it, and the request volume, prompt lengths and latency targets each caller depends on. Confirm with application owners which endpoints they call and which OpenAI-style request fields they send, because vLLM's compatibility covers the common chat, completions, responses and embeddings routes but each caller should be checked against them. The step is done when you have a written list of models, callers and peak load that the sizing work can use.
Size GPU capacity on a test workload
Load the same weights into a test environment on the accelerator type you are considering, using the vLLM installation guidance for your hardware (NVIDIA CUDA, AMD ROCm and Intel XPU are supported GPU families). Replay a representative sample of production prompts and measure throughput, time to first token and memory headroom for the KV cache. Ask your cloud provider or hardware vendor to confirm instance availability, reservation terms and the cost of the capacity you select. The step is done when you can state the GPU count and instance type that meet your latency targets at peak load.
Deploy vLLM behind an allowlisting proxy
Start the server with vllm serve on the sized hardware, and place a reverse proxy in front of it that forwards only the endpoints your users need, such as the OpenAI-compatible chat, completions, responses and embeddings routes. The vLLM security documentation notes that the --api-key option does not cover every route, including /invocations, /pause, /resume, /abort_requests, /metrics and /health, so those paths must be blocked at the proxy rather than assumed protected. Your security engineers verify the allowlist by sending requests to each blocked path from a user network and confirming they fail. The step is done when the allowlist test is recorded.
Add API keys and network isolation
Enable --api-key on the server and issue keys through your existing secrets process, so that each consuming service has its own credential that can be rotated. Multi-node deployments need extra work, because the vLLM security documentation states that node-to-node communication is insecure by default, without authorization or encryption, and must be protected by an isolated network. Your network team confirms that inference nodes accept traffic only from the gateway and from the monitoring network. The step is done when an unauthenticated request is rejected and the network rules are reviewed by someone outside the implementation team.
Wire /metrics into Prometheus
Point your existing Prometheus scraper at the server's /metrics endpoint from the monitoring network only, so the proxy does not expose it to users. Confirm the metric families you intend to alert on, such as request counts, token throughput, latency histograms and KV cache usage, and set histogram buckets if your latency targets need finer resolution. Build a dashboard and alert rules that match the targets from the sizing test. The step is done when a test load produces visible series on the dashboard and an alert fires on a deliberately induced threshold breach.
Run side-by-side evaluations
Send the same prompts from your evaluation set to the imported model on Bedrock and to the vLLM deployment, using identical sampling settings where the two interfaces allow it. Your application owners and, where required, your risk or compliance reviewers score outputs for accuracy, format, refusal behavior and any safety criteria you already track. Record any regressions caused by differences in tokenizer handling, default generation parameters or quantization settings, and decide with the owners whether each one is acceptable. The step is done when a signed-off comparison report exists for every model in the inventory.
Cut over and retain a rollback path
Route a small share of production traffic to vLLM through the gateway, widen it in steps while watching the dashboards and error rates, and keep Custom Model Import reachable until the new path has run through your normal operating cycle without incident. Confirm with your AWS account team the billing and access terms that apply to the imported models during this window, since those terms come from your contract and not from this page. Record the decommissioning date for the Bedrock resources and the owner who approves it. The step is done when all callers use vLLM, the rollback has been tested, and the Bedrock resources are removed under a documented change.
Questions buyers ask before choosing vLLM over Custom Model Import
What does running vLLM cost compared with Custom Model Import?
The software is free, so the cost is the infrastructure and the people who run it. vLLM has no paid edition, so your spend comes from GPU or other accelerator capacity, the surrounding infrastructure (networking, storage, load balancing, monitoring), the implementation work, and the engineers who operate the cluster. Custom Model Import bills per minute in Custom Model Units while the model is active, so its cost tracks how long your model runs. A self-hosted cluster costs money whenever it is running, even when idle, so utilization is the main variable. Ask any implementation partner for a quote that separates one-time work (design, deployment, hardening, load testing) from recurring work (hosting, upgrades, on-call support, model evaluation).
How long does a migration from Custom Model Import to vLLM take?
The timeline depends on how many models you serve, how many applications call them, and how much of the surrounding stack must be rebuilt. The main drivers are the hardware lead time, how much the team already knows about Kubernetes or GPU nodes, the number of endpoints that need authentication and a reverse proxy, the monitoring and alerting you require, and the load testing needed to size capacity. Migrating the client code is usually simpler because vLLM exposes OpenAI-compatible endpoints such as /v1/chat/completions, but you still need to compare outputs on your own prompts before cutting over. Ask for a written timeline tied to a defined scope, with a named cutover and rollback plan.
What does our team need to provide for a self-hosted vLLM deployment?
You need a platform that can host GPU workloads and people who will own it after handover. Typical requirements are:
- GPU or accelerator capacity in your cloud account or data center, matched to the model size you plan to serve
- A Kubernetes or VM platform, plus a decision on single-node or multi-node serving, since multi-node runs need an isolated network
- Identity and access controls for the API, plus a reverse proxy that allows only user-facing endpoints
- Monitoring for the Prometheus metrics that vLLM exposes at
/metrics - An owner for model versions, quantization choices and upgrades
Confirm these items with your provider before the quote is final.
Can our own engineers run vLLM without an outside partner?
Yes, if your team has experience with GPU nodes, container platforms and model serving, because vLLM is open source and needs no license. The harder part is ongoing operation: tracking the roughly two-week minor release cadence, testing each release against your models, patching the platform, and keeping security settings intact. vLLM's security documentation says node-to-node traffic in multi-node deployments is insecure by default and must be isolated, and that the --api-key option does not cover every endpoint. Teams without this skill set often buy fixed-scope implementation first, then take on the run work themselves or through a retainer.
Does keeping vLLM instances warm cost more than scale to zero?
Yes, keeping instances warm costs more because you pay for idle accelerators, but it avoids the cold-start delay that scale-to-zero creates. Custom Model Import avoids idle charges by scaling down, but the first request after a scale-down waits for the model to load. A vLLM deployment gives you the choice: keep a minimum replica count for latency-sensitive traffic and scale down batch or internal workloads. The cost of warm capacity is the largest recurring line item, so ask the provider to model idle hours against peak hours and to state the minimum replica count, the autoscaling rules and the expected load-test results.
Who controls model versions, quantization and upgrades with vLLM?
You do. With Custom Model Import, the service layer determines how your imported model runs, and your control is limited to the model artifacts you import. With vLLM, you choose the model weights, the quantization format, the vLLM release and the container image, and you decide when to upgrade. That control brings responsibility: each vLLM release can change behavior, so pin versions, keep a staging environment and rerun your evaluation set before promoting a new release. Ask your provider to document the upgrade procedure, rollback steps and the evidence kept for each change.
Do we keep control of data and logs when serving models ourselves?
With a self-hosted vLLM deployment, prompts, completions and logs stay on infrastructure you control, which helps where data residency or retention rules apply. You still need to configure logging so that prompts are not written to logs by default, restrict access to the metrics and health endpoints, and protect traffic between nodes. Custom Model Import keeps the model inside your AWS account, but the Bedrock runtime layer remains AWS-operated. Confirm with your compliance and security teams which controls your organization needs, and ask the implementation partner to document how each one is met in your environment. DeployOpen prepares controls and evidence for compliance assessments, but independent auditors issue any attestation.
What should a quote for this work include?
A useful quote lists the scope, the deliverables and the assumptions in writing before work starts. Ask for these items:
- The models, sizes and quantization formats in scope, and the GPU capacity assumed for each
- The deployment method (Kubernetes manifests, Helm or another documented option) and the high-availability design
- The security controls included, such as the reverse proxy allowlist and network isolation
- Monitoring, load testing and evaluation work, and what is handed over
- Support, upgrade and retainer terms, shown separately from one-time work
DeployOpen agrees scope, deliverables and timelines in writing before work begins and does not publish a price list, so the quote follows from that written scope.
Evidence auditors and internal reviewers ask for
Once you serve open-weight models yourself, the evidence trail moves from the Bedrock console to your own records. Auditors and internal reviewers will ask what was running, who changed it, who could reach it and what the logs show, so these cards cover the records to build before go-live.
Pinned model and vLLM versions
Record the exact model repository revision, weight checksums, quantization format and the vLLM version that served each production endpoint. Under Custom Model Import you rely on AWS to show which model build was deployed, but on vLLM the image tag, model revision and serving flags are yours to document. Check that each production deployment can be traced back to one recorded combination, and that the model license terms were reviewed and filed alongside it.
Change records for upgrades
vLLM ships new minor releases about every two weeks, so an unrecorded upgrade is easy to miss. Keep a change record for each vLLM, model, quantization or configuration change that states the reason, the tested behavior, the approver and the rollback target. Confirm that a rollback to the previous image and model revision has been rehearsed, not just assumed.
Endpoint authentication configuration
The vLLM --api-key option covers only some routes, so the evidence must show what is actually reachable. Keep the reverse proxy configuration that allowlists only the user-facing endpoints, the key rotation procedure and the list of routes left unprotected by the key. Test each excluded route from outside the network and file the results, including the admin routes for pause, resume and abort.
Network isolation for multi-node traffic
The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default. Keep diagrams and firewall or security group rules showing that inter-node traffic runs on an isolated network, plus test results proving that nodes are unreachable from user subnets. Bedrock hides this layer from you, so this record becomes your own responsibility.
Metrics and log retention
vLLM exposes Prometheus metrics at /metrics, covering request counts, token throughput, latency and KV cache use, but it does not keep history on its own. Document the scrape interval, the retention period in your monitoring store and who can read those metrics, since the endpoint itself is not covered by the API key. Confirm with your compliance lead how long request logs and gateway logs must be kept, and whether prompts are stored at all.
Access reviews for operators
Whoever can deploy, scale or reconfigure the GPU cluster can change what the model does and what data it sees. Keep a current list of people and service accounts with cluster and model deployment rights, the review date and the removal record for leavers. Bedrock limits this to IAM policies you already audit, so a self-hosted cluster needs its own review cycle.
What a fixed-scope migration covers
DeployOpen plans, implements, hardens, documents and supports self-hosted platforms inside your own infrastructure, and most engagements of this kind are fixed-scope: the scope, deliverables and timelines are agreed in writing before work starts. For a move from Bedrock Custom Model Import to a vLLM deployment, the written scope should name the cloud account or data center that will host the GPU nodes, the model weights and their source, the serving configuration, the gateway or reverse proxy in front of the endpoints, and the monitoring and evidence the team will hand over. A written scope is the document that turns a vague intention to self-host into work you can review, price and hold a supplier to.
Hardening is where a self-hosted vLLM deployment differs most from a managed import. The vLLM security documentation says node-to-node traffic in a multi-node deployment is insecure by default, and that the --api-key option does not cover endpoints such as /metrics or /health, so a reverse proxy that allowlists only user-facing endpoints is expected. Ask for these items to be named in the scope: the network placement of cluster nodes, the proxy configuration, the handling of the Prometheus metrics endpoint, and the restoration steps if a GPU node or model volume fails. Ongoing support, version upgrades and continuous testing can be added as a retainer once the fixed-scope build is delivered, which matters because vLLM ships minor releases about every two weeks, so someone must own that upgrade cycle after go-live.
How DeployOpen deploys vLLM under your constraints
DeployOpen plans, builds, hardens and documents self-hosted vLLM deployments inside your own cloud accounts, data centers or air-gapped networks. For teams weighing Custom Model Import against a deployment they size and control, we start from your traffic pattern, your version and quantization requirements, and the controls your assessors expect.
How we scope
- Engagements are fixed-scope, with the model list, hardware target, networks and deliverables agreed in writing before work starts.
- We record current Custom Model Import usage, request volumes and latency expectations so sizing reflects your real load, not assumptions.
- The written scope names who owns model versions, quantization changes and upgrade approval after handover.
How we build
- We deploy vLLM on your Kubernetes or VM platform with a reverse proxy that allowlists only the /v1 endpoints your applications call.
- We configure Prometheus scraping of the /metrics endpoint and set alerts on KV cache use, queueing and latency for your warm capacity.
- We test endpoint authentication, multi-node network isolation and access paths, and document the results for your security team.
How we hand over
- You receive architecture diagrams, runbooks for scaling and restarts, and a model version register tied to each deployed weight file.
- Upgrade procedures cover vLLM minor releases, which ship about every two weeks, with a staging test plan for each change.
- Training for your platform team covers operations, cost monitoring and the steps to roll back a model or serving configuration.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- We prepare systems, controls and evidence for compliance assessments, while independent auditors issue any attestations.
- Quotes follow from the written scope. Ask us to itemize GPU hosting, implementation, testing and retainer effort separately.
