Amazon Bedrock vs self-hosted vLLM: which fits your workload
Bedrock fits teams that want many hosted models with little infrastructure to run, and vLLM fits teams that need to pin specific open-weight models and control where prompts and completions go. The choice usually surfaces during a Bedrock renewal, a model retirement notice, a retention requirement or a data residency review. Bedrock gives you a managed catalog, AWS-native IAM and networking, and AWS handles capacity and patching of the serving layer. Those are real operating benefits if your security team already governs access through AWS accounts and policies, and if the models you need are in the catalog.
vLLM, the open-source engine licensed under Apache-2.0, fits when the team wants to choose the exact model build and keep it for as long as it meets requirements. You control the endpoint, the logs and the retention behavior, and you can run the server inside your own VPC, on EC2, on EKS or on hardware you own. The cost moves from per-token charges to GPU capacity, the surrounding infrastructure and the people who run it. vLLM's security documentation also makes clear that the team owns network isolation, endpoint allowlisting and authentication, so those controls must be designed and tested before go-live. Confirm current model support and hardware requirements with the vLLM project and your own benchmarks, since they change between releases.
The decision turns on four questions: which models you must use and whether open weights can replace them, what retention and residency rules apply to each workload, how steady and how large the traffic is, and whether your team can operate GPU clusters, upgrades and security hardening over time. If the answers point to a steady, high volume of open-weight workloads with strict retention limits, self-hosting deserves a serious evaluation. If you need proprietary models or bursty traffic with no in-house platform capacity, Bedrock often remains the simpler option.
Which option fits which scenario
This table maps common Bedrock decision points to the option that usually fits better and the condition that would reverse the choice. Read each row against your own workload, because the answer depends on which models you need, where data may go and who will run the GPUs.
| Scenario | Better fit and reason | Where the other option wins |
|---|---|---|
| Every model must sit inside a zero-retention policy | Self-hosted vLLM, because prompts and completions stay in your accounts and no provider review window applies to your traffic | Bedrock wins when the models you need are available under a retention mode your policy accepts, and you can enforce it with service control policies |
| A specific open-weight model version must stay fixed for months | Self-hosted vLLM, because you pin the weights and decide when to upgrade rather than following a provider lifecycle | Bedrock wins when a provider-managed version meets your needs and you accept its lifecycle dates, since Bedrock handles the upgrades |
| Bursty traffic with no steady baseline | Bedrock, because on-demand token billing and provider-managed capacity avoid idle GPUs between peaks | Self-hosted vLLM wins once sustained utilization is high enough that reserved GPU capacity costs less than on-demand tokens, which you should model with your own traffic data |
| The product needs proprietary models that Bedrock offers | Bedrock, because proprietary models are only available through the provider's API, not as weights you can host | Self-hosted vLLM wins only when an open-weight model meets quality targets on your own evaluation set |
| Air-gapped or on-premises requirement | Self-hosted vLLM, because Bedrock is reached only through AWS endpoints, even with PrivateLink | Bedrock wins only if your network can reach AWS endpoints and your policy allows it, so confirm connectivity and policy first |
| No GPU operations skills on the team | Bedrock, because AWS operates the inference fleet and you manage requests and quotas | Self-hosted vLLM wins when you can staff or contract GPU, Kubernetes and inference operations, and that cost is included in your comparison |
| Strict data residency inside your own cloud account | Self-hosted vLLM on your accounts, so inference and logs stay where your controls already apply | Bedrock wins when the regions and model availability you need are offered and your residency rules accept AWS-managed processing |
Capabilities side by side
This table compares how Amazon Bedrock and a self-hosted vLLM deployment handle the capabilities that matter in an architecture review. Where a row says a capability depends on an add-on or your own engineering, confirm it with AWS or in your own environment before assuming parity. Read the cells as the starting point for that check, not as a verdict.
| Capability | Amazon Bedrock | Self-hosted vLLM |
|---|---|---|
| Model catalog and selection | Hosted catalog of models from Amazon and third-party providers, with availability and model IDs set by AWS and the provider | Any open-weight model you download and load; you choose the version, quantization and when to upgrade |
| API surface | Bedrock's own API; applications written for it need changes to call an OpenAI-compatible endpoint | OpenAI-compatible endpoints including /v1/chat/completions and /v1/responses, so OpenAI-format clients can point at it |
| Authentication and access control | IAM policies and AWS account controls; confirm the exact model-level permissions in your AWS setup | The --api-key option covers /v1 paths only; a reverse proxy with an allowlist and your own identity layer are needed |
| Metrics and observability | Monitoring through AWS services such as CloudWatch; confirm which invocation metrics and logs your account enables | Prometheus-format metrics at /metrics covering requests, token throughput, latency and KV cache use; you build dashboards and alerts |
| Scaling across GPUs and nodes | Capacity managed by AWS; you select throughput options such as on-demand or provisioned capacity | Tensor parallelism within a node and pipeline parallelism across nodes; you size, place and secure the GPU fleet |
| Guardrails | Bedrock Guardrails as a managed feature; coverage per model should be confirmed in AWS documentation | No built-in equivalent; NeMo Guardrails or Guardrails AI must be integrated and tested by your team |
| Licensing and support | Commercial service billed by AWS; support follows your AWS support plan | vLLM is Apache-2.0 with no paid edition; support comes from the community, your staff or a contracted implementation partner |
Who controls model versions and prompt retention
The core difference is who decides when a model changes. Bedrock offers a managed catalog, and AWS and the model providers update, replace and retire models on their own lifecycles, so the model behind a given ID can change under your application. A self-hosted vLLM deployment keeps the weights and vLLM release you pin until your team chooses to upgrade. Before you commit, list every Bedrock model ID your workloads call, then check the lifecycle status of each one in the AWS documentation for that model and region. Record the dates in your change calendar so a forced migration does not arrive as a surprise.
Retention is the second question, and the answer depends on the model and the region. Bedrock's data handling behavior for prompts and completions varies by model, so confirm the retention terms for each model you use in the current AWS documentation, and ask your AWS account team to put them in writing for your contract. On a self-hosted stack, prompts and completions persist only where you configure them. Your gateway logs, tracing backend and vLLM logs are your decisions, so set retention periods and redaction rules for each store and verify them in your own environment.
Self-hosting also moves upgrade work to you. vLLM ships new minor releases about every two weeks: v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026. Pinning a release is only useful if you also run a test cycle before you move. Replay a fixed set of your own prompts against the candidate release, compare outputs and latency with the pinned version, check the /metrics endpoint for regressions, and only then promote the image. Decide in advance how long you will stay on a release and who signs off on each upgrade.
Data location, hosting models and compliance implications
Bedrock and a self-hosted vLLM cluster put your prompts, logs and access controls in different places, and those differences decide what an auditor will ask for. The cards below cover data flow, hosting choices, identity, network isolation and compliance evidence, so you can check each one against your own regulatory scope.
Where data flows in each option
With Bedrock, prompts and completions travel to AWS-managed endpoints in the Region you call, and retention depends on the model and account settings. With vLLM, requests stay inside the network you build, because the server runs in your accounts, data centers or air-gapped network. Map every hop, including logs, metrics and any gateway, before you compare residency claims. Confirm the exact retention behavior with AWS for each Bedrock model you use.
Hosting choices on the vLLM side
vLLM can run on EC2 GPU instances, on EKS with native manifests or Helm, through the vLLM production-stack, on KServe or on your own hardware. Each choice changes who patches the nodes, who owns the GPU scheduling and where logs are kept. vLLM supports NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, so confirm your accelerator and driver support before you size the cluster.
Identity and access controls
Bedrock access is governed by IAM roles, policies and AWS account boundaries. A self-hosted vLLM server does not bring IAM with it, so you must place your own SSO and key management in front of it, typically through a gateway that issues per-team keys and records who called which model. Decide which identity system is the source of truth and how service accounts are rotated. Confirm that the gateway you choose can enforce these rules before you rely on it.
Network isolation for vLLM
The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, so those nodes must sit on an isolated network. The --api-key option does not cover every endpoint, including /metrics and /health, and the docs recommend a reverse proxy that allowlists only the endpoints meant for users. Treat this as a design requirement in your network review, not a later hardening task.
Compliance evidence to collect
Bedrock gives you AWS-issued compliance documentation for its services, while a self-hosted stack makes your team responsible for producing the evidence: configuration baselines, access reviews, patch records, network diagrams and logs. Write down which controls each component supports and where the evidence lives. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports or certifications. Independent auditors and certification bodies decide whether your controls meet the standard.
What Bedrock absorbs and what a self-hosted cluster hands back
Bedrock handles the fleet behind the API. AWS selects, provisions and maintains the accelerators that serve each model, and you pay for usage or reserved capacity without choosing instance types or placing model replicas. A self-hosted vLLM deployment gives you that work back. Your team chooses the GPU or accelerator type, sizes the nodes, and decides how a model is split across them. vLLM scales a model with tensor parallelism inside a node and pipeline parallelism across nodes, and it uses Ray or its own multiprocessing backend for multi-node runs.
- GPU provisioning: Your team sources capacity on EC2, in Kubernetes node pools or on your own hardware, and handles quota, drivers and replacement of failed devices.
- Parallelism choices: Engineers decide how tensor and pipeline parallelism map onto the nodes, which depends on the model size and the interconnect between machines.
- Multi-node communication: The vLLM security docs state that communication between nodes is insecure by default, so nodes must sit on an isolated network.
- Cluster orchestration: The vLLM docs list native manifests, Helm, the production-stack, KServe, KubeRay and other Kubernetes options, each with its own operating model.
- Skills needed: GPU operations, Kubernetes, distributed systems and network design.
Confirm with your platform team whether your accelerator type and interconnect are supported by the vLLM release you plan to run before you size the cluster.
Managed capacity on Bedrock hides warm-up behavior from the caller, and the terms you buy determine how much headroom you have during peaks. On a self-hosted cluster, the model stays loaded only while your nodes keep it loaded. Keeping capacity warm means paying for GPUs that sit idle between requests, and scaling to zero means accepting a model load when traffic returns. Both choices are yours to make and to measure.
- Warm capacity: You size replicas for peak demand and keep them running, which costs money whether or not requests arrive.
- Scale-up timing: Adding a replica requires starting a new vLLM process and loading weights, so autoscaling policy and load testing matter more than they do on a managed endpoint.
- Batching and concurrency limits: You tune request concurrency and memory settings against your own prompts and output lengths, since those settings determine what one GPU can serve.
- Load testing: Your team must produce throughput and latency figures for your models and hardware, because no published figures should replace your own tests.
- Skills needed: Performance engineering, capacity planning and an understanding of how inference memory is used.
Ask the vendor or the vLLM maintainers for guidance on your model and hardware, then validate every figure with a load test on your own workload.
Bedrock is reached through AWS-managed endpoints with IAM authorization, and AWS patches the service plane. With vLLM, the serving process, the host operating system, the container images and the network path are yours to secure and keep current. The vLLM --api-key option is a useful first control, but the security docs say it protects only the endpoints under /v1, /v2, /inference and /cohere. It does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the docs warn against relying on it alone.
- Reverse proxy: Place a reverse proxy in front of the server that allowlists only the endpoints meant for users, and block the administrative and metrics paths from application networks.
- Authentication layer: Pair the proxy or a gateway such as LiteLLM with identity controls, so keys and quotas are tied to teams and applications.
- Patching: Your team tracks vLLM releases, which ship new minor versions about every two weeks, and tests each one before rollout.
- Node isolation: Multi-node traffic stays on a private network, and access to the cluster follows your existing network policy.
- Skills needed: Application security, network and proxy configuration, and a patch process for GPU hosts and container images.
Confirm with your security team which endpoints must be reachable, and test that every unintended path returns a denial before go-live.
Bedrock reports usage and invocation metrics through AWS monitoring tools, which are available without installing anything on your side. A vLLM server exposes its own telemetry at /metrics in Prometheus format, covering request counts, token throughput, latency histograms and KV cache use, with configurable histogram buckets. The data is useful, but you must collect it, store it, alert on it and connect it to cost and incident processes yourself.
- Metrics collection: Your team deploys a Prometheus-compatible scraper and decides retention, dashboards and alert rules.
- Restricted scrape path: The /metrics path is not covered by the API key, so the scraper needs network access that application traffic does not have.
- Cost attribution: vLLM reports tokens and requests, but mapping them to teams or products requires a gateway or your own tagging, such as LiteLLM spend tracking.
- Tracing and logs: Request-level logging and prompt-level traces are separate work, and you decide what to store, since prompts may contain personal data.
- Skills needed: Prometheus and Grafana operations, alerting design and data-handling policy for logs.
Confirm which of these metrics your version of vLLM exposes, and check whether your retention rules allow prompt content in logs before you enable request logging.
What each option costs to own and what drives it
Bedrock bills by usage and commitment, while a self-hosted vLLM stack bills you for hardware, the surrounding services and the people who run them. The cards below separate one-time work from recurring spend and name the drivers that move each line, so you can compare quotes on the same basis.
Bedrock token and commitment pricing
Bedrock charges by pricing mode, which can include on-demand tokens, Flex, Priority, batch, Provisioned Throughput commitments and Custom Model Import billing. Each mode suits a different traffic pattern, so the mix you choose matters as much as the rate. Pull current prices from AWS for each model and region you use, and model your own token volumes against them. Check whether commitment terms run beyond your planned cutover date.
GPU and accelerator capacity
This is the largest recurring line for self-hosted vLLM. Cost depends on the accelerator type, how many GPUs the model needs through tensor and pipeline parallelism, and whether you buy hardware, reserve cloud instances or run on demand. Idle capacity is paid for whether or not requests arrive, so utilization and traffic steadiness drive the effective cost per token more than any list rate. Ask vendors and cloud providers for pricing per GPU-hour and for reservation terms.
Gateway and database layer
A shared gateway handles keys, budgets, rate limits, routing and logging, and it needs PostgreSQL for virtual keys and, once more than one proxy instance runs, Redis 7.0 or newer. LiteLLM is the open-source option here, and its Enterprise license covers SSO beyond the free user limit and audit logs. Costs are the infrastructure to run these services, the backup and failover design, and the engineering hours to keep them patched. Ask whether your quote includes high availability for both data stores.
Monitoring and logging
Self-hosted inference exposes Prometheus metrics at /metrics, covering request counts, token throughput, latency and KV cache use. Storage, retention and dashboards for those metrics, plus any tracing backend you add, become your cost. Bedrock's CloudWatch and logging charges are replaced by your own storage and alerting. Confirm how long you must keep prompt and completion logs for audit, since that setting drives storage size and cost.
One-time implementation and migration
Design, hardening, integration, load testing, guardrail replacement and evaluation of outputs against your own prompts are one-time costs. They also include cutting over application code from the Bedrock API to an OpenAI-compatible endpoint. Scope these in writing with a fixed deliverable list, and ask the provider to name the acceptance tests and the cutover plan. Estimates should separate the work you do in-house from the work you buy.
People who operate the cluster
Bedrock absorbs patching, capacity management, regional availability and model hosting. A self-hosted stack returns those tasks to your team, including vLLM upgrades (new minor releases about every two weeks in the supplied release history), GPU driver changes and on-call response. Staffing is usually the largest hidden recurring cost. Ask whether a support retainer covers upgrades, security patches and out-of-hours incidents, and what response times it commits to.
What a move from Bedrock to self-hosted vLLM involves
The sequence turns Bedrock calls, model choices and guardrail policies into a tested, self-operated inference stack, and each step closes with an output you can check before the next one starts. Timelines depend on scope and are agreed in writing before work begins.
Inventory Bedrock models and API calls
List every application, service account and batch job that calls Bedrock, the model IDs it uses, the APIs it calls (Converse, InvokeModel, batch inference or Custom Model Import), and the Bedrock features each one depends on, such as Guardrails, Knowledge Bases or Agents. Your platform team and application owners do this together, because application owners know which prompts and output formats must not change. The step is done when every caller maps to a model, an API surface and a data classification in one inventory that is signed off by the owners.
Choose replacement open-weight models
For each Bedrock model in scope, shortlist open-weight models that match the task, license terms and language coverage, and confirm each one runs on vLLM with the accelerators you plan to use. Check the vLLM supported-models list and the model's own license, not only its benchmark scores. Size GPU capacity from expected concurrency, context lengths and latency targets, and ask the implementation partner to document the assumptions. The output is a model list with licenses, hardware targets and a capacity sheet that you can challenge.
Stand up vLLM behind an authenticated gateway
Deploy vLLM on your chosen hardware, on EC2, EKS or your own servers, and place a gateway such as LiteLLM in front of it so applications call one OpenAI-compatible endpoint. vLLM's own --api-key option does not cover every endpoint, so put a reverse proxy in front that allowlists only the paths meant for users, and isolate multi-node traffic on a private network. Configure virtual keys, budgets and rate limits in the gateway, store the master key in a secret manager, and enable Prometheus metrics. The step is done when an unauthenticated request is refused and an authorized request is logged with its key and model.
Route a test workload and compare outputs
Send a copy of production-shaped traffic, using your own prompts and anonymized or synthetic data where production data cannot leave the boundary, to the candidate models and to Bedrock in parallel. Score answers against task-specific criteria with human reviewers and automated evaluations, and record latency, throughput and error rates at realistic concurrency. Include cold start behavior, because a self-hosted deployment is warm only if you keep it warm. The output is a written comparison that shows where each candidate passes, where it fails, and which applications need prompt changes before cutover.
Replace Bedrock Guardrails with an open-source approach
Map each Bedrock Guardrails control, including content filters, denied topics, word filters, sensitive information filters and any contextual grounding checks, to a self-hosted equivalent such as NeMo Guardrails rails or Guardrails AI validators. Some policies map directly, and others need custom code or no equivalent at all, so record which controls are replaced, which are reimplemented and which are accepted as gaps by the risk owner. Test the new controls with the same red-team prompts and PII samples used against Bedrock. The output is a control mapping with test results and sign-off from security and compliance.
Run a staged cutover
Move one application at a time behind a gateway route that can switch between Bedrock and vLLM by configuration, starting with lower-risk workloads and widening traffic in steps. Keep a rollback path to the Bedrock model for each route until the owner accepts the results, and watch error rates, latency, token usage and guardrail hits on dashboards. Confirm that logs and traces stay inside your environment as designed. The step is done when every application runs on the self-hosted route, with a signed cutover record for each one.
Retire Bedrock access on schedule
Remove application permissions, IAM roles and service endpoints that call Bedrock, then disable model access where your agreements allow. Check the terms of any Provisioned Throughput commitment, Custom Model Import arrangement or other contract, and the data retention obligations that apply to prompts and completions already sent to Bedrock, before you deprovision anything. Document the final state, including retained logs and who approved the change. The output is an access review that shows no active Bedrock calls and a dated record of the decommissioning decision.
Questions buyers ask before choosing vLLM or Bedrock
What does a move from Bedrock to self-hosted vLLM cost, and what drives it?
The cost is a mix of one-time work and recurring spend, and the split depends on your workload. vLLM itself has no license fee, so the recurring cost is GPU or accelerator capacity, the surrounding infrastructure (storage, network, a gateway, monitoring) and the people who operate it. One-time costs include planning, model evaluation, implementation and hardening. The main drivers are model size, peak concurrency, required availability and how many environments you run. Ask any quote to separate hardware and cloud charges, implementation fees and ongoing support so you can compare them with your current Bedrock invoices, including any Provisioned Throughput commitments.
How long does it take to move workloads off Bedrock?
Duration depends on how many applications call Bedrock, how many models you must replace, and how much evaluation your outputs require. A single chat use case with one model is far simpler than a portfolio of agents using several Bedrock models with guardrails attached. Other drivers include GPU procurement lead times, network approvals, identity integration and the number of prompts you must test before cutting over. Because Provisioned Throughput terms and model lifecycle dates in your account set a deadline, check those first. Ask for a written scope with deliverables and milestones before you commit to a timeline.
What does our team need to provide for a self-hosted vLLM deployment?
Plan to provide the following before implementation starts:
- GPU or accelerator capacity in a cloud account, data center or on-premises hardware that supports vLLM's CUDA, ROCm or Intel XPU backends
- An identity provider for administrator and user access to gateway and monitoring tools
- Network approvals for private subnets, ingress, and any outbound paths you need
- Staff time from platform engineering, security and the application teams that will change their API calls
Confirm these details with your cloud, security and procurement owners early, because approvals often take longer than the technical build.
Can our internal team run vLLM and the gateway without outside help?
Yes, a capable internal platform team can run vLLM, but the decision depends on skills and capacity rather than on the software. Running it in production means GPU driver and scheduler management, multi-node networking, Kubernetes operations, metrics, a PostgreSQL database and Redis for a multi-instance LiteLLM gateway, and upgrade planning. vLLM's security docs say node-to-node traffic is unauthenticated and unencrypted by default, so isolated networking is required. Be honest about on-call coverage and key-person risk. DeployOpen can plan, implement, harden, document and support these platforms through fixed-scope engagements or retainers, so you can choose which parts to keep in-house.
How are vLLM upgrades and model version changes handled?
You control versions, which is the main difference from Bedrock. vLLM ships new minor releases about every two weeks, for example v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, so upgrades need a regular testing cycle rather than an emergency response. You also decide when to change a model's weights, quantization or serving settings. Set a process that re-runs your evaluation prompts before each change, keeps rollback images ready and pins model revisions. Confirm with your team who approves changes and how often you will schedule them.
Does vLLM support the same API calls as Bedrock?
Not directly. vLLM's server exposes OpenAI-compatible endpoints, including /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, so applications written against the OpenAI request format can usually point at it. Bedrock uses its own runtime API and model IDs, so those calls must change. A LiteLLM gateway in front of vLLM can present one OpenAI-compatible endpoint and handle routing, keys and budgets. Check each feature you use, such as tool calling, structured output, streaming behavior and multimodal input, against the vLLM version you plan to run, because support varies by model.
Can we avoid Bedrock's data retention rules completely by self-hosting?
Self-hosting can remove Bedrock's retention terms for the workloads you move, because prompts and completions stay in your environment rather than going to a provider's service. However, it does not remove your own retention obligations. Logs from vLLM, the gateway and tracing tools, along with any prompt caches and backups, become your records to manage. Confirm your retention periods, log redaction and access controls with your compliance and legal teams. DeployOpen prepares systems, controls and evidence for compliance assessments, but auditors and certification bodies issue any audit report or attestation.
Which security settings must we configure before exposing a vLLM endpoint?
The vLLM --api-key option protects only some routes. According to the vLLM security docs, it covers /v1, /v2, /inference and /cohere, but not /metrics, /health, /pause, /resume, /abort_requests or /invocations. The docs warn against relying on it alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Also keep metrics internal, isolate multi-node traffic, and store LiteLLM's master key in a secret manager. Ask your security team to test these paths before go-live, and include the result in your change records.
How DeployOpen helps you evaluate, implement or migrate
DeployOpen is a Chainscore Labs company that plans, implements, hardens and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Amazon Bedrock vs self-hosted vLLM, we start with a written view of which models, data flows and operating duties you would take on, so the comparison rests on your environment rather than on general claims.
How we scope
- We agree a written scope covering the models, prompts and workloads to be evaluated or moved off Bedrock
- We map Bedrock retention modes, model versions and Provisioned Throughput terms to your own requirements
- We document which Bedrock guardrails, endpoints and IAM controls need open-source replacements in the scope
How we build
- We size a vLLM deployment on your chosen GPU, EKS or on-premises hardware with the parallelism plan agreed in scope
- We place an OpenAI-compatible gateway in front of vLLM with authentication, budgets and a reverse proxy allowlist
- We test output quality against your own prompts and record throughput, latency and cold start measurements
How we hand over
- We deliver runbooks for model upgrades, vLLM version changes, scaling, monitoring and incident response
- We document the security configuration, including node isolation, endpoint exposure and secret handling
- We provide cutover plans and evidence packs that your internal teams and independent auditors can review
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover
- Retainers can cover tracking of vLLM release changes and planned upgrades in your environment
- Our work prepares systems and evidence for compliance assessments, which independent auditors complete
