Migrate from Amazon Bedrock to self-hosted models on vLLM

Bedrock Guardrails content filters screen prompts and responses for categories such as hate, insults, sexual content, violence and misconduct, with configurable strength. In NeMo Guardrails, the closest equivalent is a combination of input rails that screen user messages before the model sees them and output rails that check the model's answer before the user receives it. The library includes self-check input and output moderation and NVIDIA content safety models, so you can choose between a prompt-based self-check that uses your own model and a dedicated classifier.

• Input rails: Block or redirect harmful user messages before they reach vLLM, which limits token spend on requests you will refuse anyway.
• Output rails: Check completions before return, which covers cases where a benign prompt produces harmful text.
• Model choice: Confirm which content safety model is licensed for your use and whether it runs fully inside your network.
• Thresholds: Bedrock exposes filter strength settings; NeMo Guardrails expects you to tune prompts or classifier thresholds yourself and record the values.
• Refusal text: Write standard refusal messages in the configuration so behavior stays consistent across applications.

Confirm with your security team which categories are in scope and which thresholds they accepted before cutover.

Bedrock denied topics let you describe a subject in natural language and block conversations about it. NeMo Guardrails handles this with dialog rails, which use Colang flows to define user intents, bot responses and the conditions under which a conversation is redirected. Topic safety models are also available in the library for classifying whether a message falls in a restricted area. Dialog rails give you more control over multi-turn conversations than a single filter, but they also need more authoring work.

• Dialog rails: Define the denied topic as a user intent and map it to a fixed refusal or handoff flow.
• Topic safety models: Use a classifier when the topic is hard to describe with example phrases.
• Multi-turn checks: Test whether a user can reach the denied topic gradually across several turns.
• Colang version: Colang 1.0 is the default, so decide early whether you will author flows in 1.0 or 2.x.
• Ownership: Assign a named owner for each topic definition, since changes to scope affect what users can ask.

Review each denied topic with the business owner and keep the agreed definitions in version control.

Bedrock word filters block listed terms and PII masking detects and redacts entities such as names, addresses and account numbers. In NeMo Guardrails, word lists and pattern checks are usually implemented as input or output rails backed by Python actions, which run custom code during the flow. PII detection can use an open-source detector wired in as an action, or a rule set you write yourself. Execution rails can also check tool calls and retrieved data before an action runs, which matters when a model can send data to another system.

• Word filters: Implement as input or output rails with a maintained term list stored in the repository.
• PII masking: Run detection on both prompts and completions, and log redaction events without storing the raw values.
• Execution rails: Apply checks to tool inputs and outputs so redacted values do not leave the boundary through a function call.
• Retrieval rails: If you use retrieval, screen chunks before they enter the prompt.
• Coverage gaps: Confirm which PII types your detector supports, since an unlisted identifier will pass through unchanged.

Run a test set of real-format but synthetic identifiers, and document every identifier type that is not covered.

NeMo Guardrails v0.24.1 was released on 16 September 2026, and the library supports Python 3.10 through 3.13. The configuration uses a config.yml file, optional Python actions and Colang files. Colang 1.0 is the default, but NVIDIA documents a Colang 1.0 to 2.x migration using the nemoguardrails convert tool, and says converted configurations still need manual review. You can call the library from Python or run a guardrails server with a Dockerfile. NVIDIA also offers a NeMo Guardrails microservice, a production container deployed on Kubernetes with Helm, built on the same configuration model.

• Pin the version: Record the exact library version and pin it in your build, since the release line moves quickly.
• Choose Colang version: Keep Colang 1.0 if your flows already work, or convert to 2.x and review every converted flow by hand.
• Server or library: Use the guardrails server for shared, language-neutral access, or the Python API when one application owns the rails.
• Microservice: The Helm-deployed microservice fits Kubernetes platforms, but confirm its license and support terms with NVIDIA before you adopt it.
• Placement: Run the guardrails layer next to vLLM and the LiteLLM gateway, inside the same network boundary.

Confirm the supported Python and Kubernetes versions in your environment before you build the image.

Guardrails change behavior in two directions: they can block attacks that should be blocked, and they can block ordinary requests that should be answered. Bedrock Guardrails gave you a managed set of policies with its own test tooling, so you cannot assume the replacement matches it. Build a test suite that includes adversarial prompts such as jailbreak and prompt injection attempts, harmful requests in each category, and benign prompts that look similar to restricted content. Run the same suite against Bedrock and the replacement and compare outcomes side by side.

• Adversarial set: Include jailbreak, prompt injection and encoding tricks that target each rail.
• Benign set: Include normal business requests that use sensitive words, so you can measure false positives.
• PII set: Use synthetic identifiers in every format your applications produce.
• Multi-turn set: Test denied topics across conversations, not only single prompts.
• Sign-off: Record results, the reviewer and the accepted residual risk before traffic moves.

Confirm the acceptance criteria with your security and compliance leads, and keep the test suite in version control so it runs again after every configuration change.

Teams paying per token on demand have the most flexibility, because no contract ties the cutover to a date. The main work is application-side: find every Bedrock call, record the model IDs in use, and decide which open-weight replacement serves each workload. Because billing is metered, you can run old and new paths side by side and shift traffic gradually without penalty.

• Inventory calls first: search code, prompt stores and scheduled jobs for bedrock-runtime calls and model IDs, including batch and evaluation jobs that teams forget.
• Shift by workload: move low-risk summarization or classification traffic before customer-facing chat, so you learn the operating model on less sensitive paths.
• Run in parallel: route a share of requests to vLLM behind a LiteLLM endpoint and compare outputs on your own prompts before increasing that share.
• Plan for capacity, not contracts: GPU capacity is the new constraint, so confirm procurement lead times with your hardware or cloud provider early.
• Keep the fallback: retain Bedrock credentials until the replacement has passed your acceptance tests for each workload.

Before starting, check which of your models are Legacy or approaching end of life in your Bedrock account, since that can force a migration on a date you did not choose.

Teams with Provisioned Throughput commitments are bound by the term they signed, so the cutover date is set by the contract rather than by engineering readiness. The sequencing question becomes whether you can have the replacement validated before the commitment ends, and whether you should avoid renewing it while the migration is underway. Read the commitment terms with your account team and confirm exactly what happens at expiry, including any notice periods, before you set milestones.

• Find the end date: record each commitment's term end and whether it renews automatically, and confirm the notice requirements in writing.
• Work backward from expiry: schedule evaluation, load testing and the gateway cutover so the final acceptance gate falls before the term ends.
• Do not stack capacity: avoid buying a second commitment for the same workload, since that locks in cost while the self-hosted cluster is still being built.
• Budget for overlap: running dedicated GPU capacity while the commitment still serves traffic is a cost to model, so ask your team what overlap period the plan includes.
• Scale the new cluster in steps: size vLLM for the peak you actually measured, then add replicas once metrics confirm the headroom.

Confirm with your account team whether any capacity can be reduced or moved before the term ends, because the answer changes the overlap cost.

Teams that already import their own open-weight models into Bedrock have a head start on the model choice, because the weights and their evaluation history already exist. The shift is in who operates the serving layer. Import billing and scaling behavior are handled by Bedrock, while a vLLM deployment makes you responsible for instance sizing, warm capacity, quantization choices and upgrades. Your existing evaluation results are the best starting baseline for the new deployment.

• Reuse the weights: confirm you hold the model files and license terms in your own storage, so the same artifacts can load into vLLM without another download from Bedrock.
• Set warm capacity deliberately: where Bedrock scales to zero, decide whether your deployment keeps replicas warm, since that trade-off moves cost and latency.
• Pin versions: record the vLLM release and model revision in your configuration, and track the vLLM release cadence so upgrades are planned rather than forced.
• Test quantization: compare any quantized variant against the full-precision baseline on your own prompts before promoting it.
• Move the endpoint: point the existing application at the LiteLLM endpoint so the model switch does not require application code changes.

Verify that your license for each imported model allows self-hosting outside Bedrock before you copy any weights.

Regulated teams with strict data residency requirements often start with a compliance question rather than a model question. The first step is to document where prompts, completions, logs and traces go today, including any retention mode set on the Bedrock side, and to decide which workloads must stay inside your boundary. Self-hosting moves the inference path into your own infrastructure, but it also makes your team responsible for access control, logging and evidence, which your assessors will examine.

• Map data flows: list every system that receives prompts or completions, including observability tools and gateways, and confirm which ones leave your boundary.
• Choose the boundary first: decide whether the vLLM cluster, LiteLLM and any guardrail service run in your VPC, data center or disconnected network, then size the rest of the plan around that choice.
• Authenticate the endpoint: vLLM's --api-key option does not protect every route, so place a reverse proxy in front that exposes only the endpoints users need.
• Isolate multi-node traffic: the vLLM security documentation treats inter-node traffic as insecure by default, so place nodes on an isolated network.
• Prepare evidence early: collect configuration, access logs and change records from the start, since Bedrock's retention terms no longer apply to your own stack.

Ask your compliance lead and independent assessor which controls they expect to see before the first production request, because preparing that evidence takes longer than the technical cutover.