Which Bedrock models keep prompts for AWS review, and what that means for your controls
Some Bedrock models require the aws_review mode, which keeps prompts and completions for up to 30 days for possible AWS review, so a zero-retention policy cannot cover every model in the catalog. Self-hosted vLLM is the option that keeps every model you run inside your own retention scope. For a regulated team, the decision is therefore not whether Bedrock can be made compliant, but which workloads can stay on Bedrock models that meet your retention rule and which need a model you host yourself.
Start with an inventory. List every Bedrock model ID your applications call, then confirm in the current AWS documentation which of them run under aws_review and which support zero retention. Apply a service control policy only where AWS documents the condition keys that enforce the zero-retention setting, and test that a denied call fails in a non-production account before you rely on it. Expect that a blocked model is unavailable to the team until it is mapped to an approved replacement, so record the owner and the use case for each one. Ask your compliance lead to sign off on that mapping in writing.
Self-hosting moves the retention question into your own systems. vLLM serves OpenAI-compatible endpoints, so applications can point at it, but your access logs, the Prometheus metrics at /metrics and any tracing tool in the path may each store prompts or identifiers. Set retention on each of them. The vLLM security documentation also says that --api-key does not protect /metrics or /health, and that inter-node traffic in multi-node deployments is unauthenticated and unencrypted by default, so place a reverse proxy in front and isolate the nodes before you go live.
What retention constraints require from the platform
A retention mode decides where prompts and completions can go, and regulators care about the same question from several angles. The cards below set out the control obligations that follow, so that a platform choice can be tested against each one before a model is approved.
Scope of ePHI under HIPAA
The HIPAA Security Rule applies to covered entities and their business associates for all electronic protected health information they create, receive, maintain or transmit. A prompt that contains patient details is ePHI, even if it is only a summarization request. Confirm whether your AWS provider relationship is covered by a business associate contract that flows the Security Rule terms to any subcontractor handling that data, and whether the model provider's retention mode is part of that chain. HIPAA has no certification, so ask for evidence you can show an assessor.
Cardholder data environment boundaries
PCI DSS covers any system that stores, processes or transmits cardholder data, and also systems that could affect the security of that environment. If an LLM receives card numbers or authentication data, or sits in a path that handles them, the model service may fall inside the cardholder data environment. Ask whether the chosen service must be assessed as part of your scope, and whether a Qualified Security Assessor or a Self-Assessment Questionnaire will need to describe it. Self-hosting keeps that boundary under your own network controls.
Processing location and data return
DORA Article 30(2) expects an ICT services contract to state where services are provided and data processed, to give advance notice of changes, and to cover return of data on insolvency or termination. For a bank or payment firm, a retention mode that names a region or an AWS review process is a fact to record in the register of information. Check that your contract with the model provider states processing locations, and that you could move your prompts, logs and evaluation data out if the arrangement ended.
Security of processing under GDPR
GDPR Article 32 expects controls matched to the risk of the processing, including access control and strong authentication. The CNIL fined Hopital Prive de la Loire EUR 500,000 on 3 September 2026 after finding that external access to patient records lacked VPN and multi-factor authentication and ignored the care team's need to know. The lesson for LLM traffic is that prompt logs and gateway keys deserve the same access control as the source records. Check who can read stored prompts and who holds the keys that call the model.
Evidence that retention controls work
A retention setting is only useful if you can show it is enforced. Ask the provider for the exact retention mode each model uses, how it is set through organization policy, and how a change would be detected. On a self-hosted stack, the evidence is your own configuration: the serving endpoints, the logging settings and the storage lifecycle rules. Keep records of which prompts were stored, for how long and where, so that an auditor or a regulator's question can be answered from logs rather than from memory.
How Bedrock retention modes differ and how zero retention is enforced with SCPs
Bedrock's data handling for some foundation models depends on a retention mode, and the one that matters most here is aws_review. Under that mode, prompts and completions may be kept for up to 30 days so AWS can review them for possible misuse. The 30-day window is the figure to plan around. Before you rely on any mode, confirm the current list of retention modes, which models each one applies to, and the exact behavior of each mode in the current AWS documentation for your account and Region, because the list changes over time and this page does not restate it.
If your policy requires zero retention, you can use an AWS Organizations service control policy to stop invocation of models that would fall under review. The approach is to write a deny statement on the Bedrock invocation actions, scoped with a condition key that identifies the model or the retention mode, and attach it to the organizational units that host regulated workloads. Confirm the exact action names and condition keys in the AWS service authorization reference and test the policy in a sandbox account first, because a mistyped key can either block nothing or block approved models. SCPs also do not apply to the management account, so keep regulated workloads out of it.
Treat the SCP as a control you must evidence, not only configure. Record the policy version, the organizational units it covers, test results showing that a denied model call fails, and the change log for the policy. Auditors reviewing a regulated program will ask how the control is enforced and how you know it works, and a documented test with a dated result answers that question better than the policy text alone. If a model your workload needs is denied under zero retention, you have three choices: change the model, accept a documented exception, or run that model yourself. The last option is the only one that keeps every model under your own retention terms.
Which self-hosted options keep every model in your retention scope
This table compares vLLM for inference with LiteLLM, Langfuse, Opik, Portkey hybrid and Arize Phoenix for logging and tracing. Read each row against your own question: where prompts and completions are processed and stored, what license limits apply, and which part of the stack the option covers.
| Option | Where prompts and traces stay | License and enterprise limits | Trade-off to confirm |
|---|---|---|---|
| vLLM (inference engine) | Inference runs on your GPUs or cloud accounts, so prompts and completions do not leave your network when the endpoint is isolated | Apache-2.0 with no paid edition; cost is accelerator capacity, infrastructure and staff | Serving only, so it does not log traces; the security docs require a reverse proxy and network isolation for multi-node traffic |
| LiteLLM AI Gateway (gateway) | Self-hosted server in your infrastructure that routes requests to your chosen model endpoints | MIT except the enterprise directory; SSO, SAML, audit logs and multi-team management need LiteLLM Enterprise | Confirm which features your team needs fall under the enterprise license before you rely on them |
| Langfuse (tracing) | Traces, prompts and completions are stored in your PostgreSQL, ClickHouse and object storage | Open core: code outside the ee directories is MIT; some advanced features need an enterprise license key | Requires operating several data stores; confirm the enterprise features you need are covered by a key |
| Opik (tracing and evaluation) | Full platform, including backend, UI and tracing, runs in your own environment | Apache-2.0; the README states the full platform can be self-hosted without a commercial license | Confirm the deployment footprint and retention settings suit your data classification |
| Portkey hybrid (gateway and logging) | Gateway runs in your VPC, but the control plane stays in Portkey's VPC | Gateway code is MIT; prompt management, advanced logging, access control and PII redaction are enterprise features | Gateway calls out over HTTPS to control.portkey.ai, so confirm that egress path meets your retention and residency rules |
| Arize Phoenix (tracing and evaluation) | Runs in your own infrastructure as a self-hosted tool | Elastic License 2.0, not OSI-approved; forbids offering the software to third parties as a hosted or managed service | Confirm your use fits the license terms with your legal team before a managed offering or client-facing service is built on it |
How to deploy vLLM behind a gateway under a zero-retention rule
This sequence takes open-weight models out of any third-party retention window by serving them on your own compute, behind an isolated endpoint, a filtered reverse proxy, a gateway that logs to your own tracing store, and monitoring you control. The result is a deployment where prompts and completions stay inside your network and you decide how long they are kept.
Choose GPU capacity inside your boundary
Decide where the accelerators will run: a cloud account you control, a private data center or an air-gapped network. Confirm with your hardware or cloud team which GPU types the target model needs, and check that the vLLM installation docs list support for your accelerator, since vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, among other targets. The step is done when the account, subnet and accelerator reservation are documented and approved by the data owner, with no path to an external inference provider.
Run vLLM on an isolated network
Start the model with the vllm serve command on hosts placed in a subnet with no internet egress for inference traffic. If you span several nodes, remember that the vLLM security docs state inter-node communication is insecure by default, so the nodes must sit on an isolated network with no authorization or encryption between them. Pin the exact vLLM version in your deployment manifest. The step is done when the server answers a test request from an approved client and cannot reach any external host.
Place a reverse proxy with an endpoint allowlist
Put a reverse proxy in front of the vLLM server and allow only the user-facing routes your applications need, such as the OpenAI-compatible chat and completions paths. The vLLM --api-key option does not protect /metrics, /health, /pause, /resume, /abort_requests or /invocations, and the docs warn against relying on it alone, so the proxy must block those paths for general clients. Keep metrics reachable only from your monitoring network. The step is done when a request to a blocked path is refused and an allowed path succeeds with a valid key.
Route applications through a LiteLLM gateway
Deploy the self-hosted LiteLLM AI Gateway as the single OpenAI-compatible endpoint your applications call, and register the vLLM server as a backend model. Issue virtual keys per team, set budgets and rate limits, and configure logging to your own Langfuse or Opik instance instead of any hosted tracing service. Confirm that request and response bodies are either logged only to stores you run or excluded from logging where your retention rule forbids storage. The step is done when a test trace appears in your tracing store and no outbound call leaves your network.
Configure Prometheus to scrape vLLM metrics
Add the vLLM API server's /metrics endpoint as a scrape target in your Prometheus configuration, using the monitoring network path you opened in the proxy step. The metrics cover request counts, token throughput, latency histograms and KV cache use, and histogram buckets can be configured. Build dashboards and alerts in your existing Grafana or equivalent stack. The step is done when the target shows as healthy in Prometheus and a throughput panel updates after a test load.
Pin versions and test each upgrade
vLLM ships new minor releases about every two weeks, for example v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, so an unpinned deployment changes under you quickly. Pin the vLLM image, the LiteLLM version and the model weights by hash, and write a change process that runs your evaluation prompts against a candidate version in a staging environment before promotion. Confirm with your security team how advisories and release notes are reviewed. The step is done when the pinned versions and the upgrade runbook are approved and a staged upgrade has been run once end to end.
Document the retention and evidence trail
Record where prompts and completions are processed, which stores hold logs, how long each store retains data and who can delete it, mapped to your data retention policy. Keep the network diagrams, proxy allowlist, version pins and access logs as evidence for internal reviewers and, where relevant, for an independent auditor, who alone decides conformance. The step is done when the data flow diagram, retention settings and evidence list are signed off by the data owner and the security lead.
When Bedrock still fits and when self-hosting is the only route
Bedrock still fits when every model on your approved list is eligible for zero retention and an organization-level control enforces that setting. Check this model by model, because a single approved model that requires aws_review keeps its prompts and completions for up to 30 days for possible AWS review, which puts that workload outside a zero-retention position. Enforce the outcome with service control policies so that a developer cannot invoke a model outside the list, and confirm the current retention eligibility of each model in AWS documentation before sign-off, since model lists change. If the list passes, keeping Bedrock is a sound choice, and the remaining work is governance: a written model allowlist, a review step for each new model, and dated evidence of the SCP configuration for your assessor.
Self-hosting becomes the only route to keep every model in scope when a required model needs aws_review, or when a control owner will not accept a 30-day review window for the data class in question. Two regimes make the case stronger. Under DORA, Article 30(2) requires every ICT services contract to state where services are provided and data processed, and Article 28(8) requires documented, tested exit strategies for services that support critical or important functions. Open-weight models on vLLM in your own account give you a processing location you can name and weights you can keep, which makes an exit plan testable. Under HIPAA, a business associate contract under 45 CFR 164.314(a)(2)(i) must require compliance with the Security Rule and flow the same terms down to subcontractors that handle ePHI, so the contract terms still need to be in place wherever the model runs.
Self-hosting does not remove third parties from the picture. Your cloud provider, a managed Kubernetes service, a hosting partner or a monitoring vendor may still touch ePHI or regulated data, and each one needs written contract terms and an incident reporting duty. Ask each provider in writing whether it will sign a business associate agreement and where its logs, snapshots and backups are stored. The costs also change shape: Bedrock bills per token, while a self-hosted cluster carries GPU capacity, operations and upgrades, and vLLM ships new minor releases about every two weeks. Many organizations split the estate, keeping zero-retention models on Bedrock and moving aws_review or regulated workloads to self-hosted models.
What auditors and regulators ask for when retention is in scope
What data flow diagram do auditors expect for prompts and logs?
Auditors expect a diagram that shows every place a prompt, completion, trace or log is created, stored, copied and deleted, with the account, region and retention setting for each. Start at the application, then follow the request through the gateway, the inference endpoint, the observability store and any backup or export bucket. Label which components sit inside your cloud account and which sit with a vendor, because that boundary decides which contracts and controls apply. Keep the diagram tied to a version date so changes after an assessment are visible. DeployOpen can prepare this diagram as part of a fixed-scope engagement, but the accuracy depends on your environment, which you must confirm.
Which SCP text and change records prove zero retention is enforced?
Reviewers look for the exact service control policy text, the organizational unit it attaches to, and a change history showing who approved each edit and when. They also check that the policy was tested, for example by attempting a call to a model that requires retention and confirming it is denied. Keep the policy in version control, record each deployment, and retain the evidence from periodic checks that the policy is still attached. Confirm with AWS documentation which retention behavior each Bedrock model requires before you write the deny conditions, since the rules depend on the model.
How should the model inventory map to retention modes?
The inventory should list every model in use, its provider, its hosting location, its retention mode and the business workloads that call it. Auditors use it to confirm that no workload reaches a model whose retention mode conflicts with the data classification. Add a column for the replacement or fallback model so the inventory also shows how a workload continues if a model is restricted. Update the inventory whenever a model ID changes, because a silent version change can change the retention mode.
What access control and MFA records do assessors request?
Assessors request the list of identities that can call the model endpoints, read the logs and change the policies, along with the approval records for each grant. They expect multi-factor authentication for administrative and remote access, and they check that the configuration and the identity provider settings match the written standard. Keep quarterly access reviews, removal records for leavers and evidence that break-glass accounts are monitored. The FTC Safeguards Rule, for example, requires multi-factor authentication for any individual accessing an information system unless the Qualified Individual approves an equivalent control in writing (16 CFR 314.4(c)(5)).
What does a DORA Article 28(3) register entry need to contain?
DORA Article 28(3) requires a register of information covering all contractual arrangements for ICT services, so every AI service provider, including a hosted model provider, needs an entry. Each entry should identify the provider, the service, the function it supports, the locations where services are provided and data is processed, and whether the function is critical or important. Article 30(2) also requires the contract to state where services are provided and data processed, with advance notice of changes, and to cover data return on insolvency or termination. Check each contract against these points, because the register is only as accurate as the contracts behind it. DORA has applied since 17 January 2025 to the financial entities it covers.
What exit strategy does Article 28(8) require for AI services?
Article 28(8) requires documented and tested exit strategies and transition plans for ICT services that support critical or important functions. For a model provider, the exit plan should name the replacement model or self-hosted endpoint, the gateway change needed to route traffic, the data that must be deleted or returned, and the test that proves the replacement answers as expected. Record when the plan was last tested and what it found. Teams often leave this step until a renewal forces a decision, which leaves little time to test. Confirm with your legal and compliance leads which functions count as critical or important in your organization.
Can a certification or audit report prove HIPAA compliance for retention?
No. HHS states that no Security Rule standard requires certification and that private certifications do not prevent HHS from later finding a violation (HHS FAQ 2003). The HHS Office for Civil Rights enforces the rule through complaints, compliance reviews and audits. DeployOpen prepares systems, controls and evidence for assessments, but it does not issue audit reports, attestations or certifications, and independent auditors and certification bodies issue their own opinions. Where a business associate contract applies, 45 CFR 164.314(a)(2)(i) requires the contract to address Security Rule compliance and incident reporting, so the contract is part of the evidence set.
Who issues the attestation a regulator or card scheme will accept?
An independent party issues it, not DeployOpen or the platform vendor. For PCI DSS, the payment brands decide who must validate, and the assessment is documented as a Report on Compliance by a Qualified Security Assessor or a Self-Assessment Questionnaire, each with an Attestation of Compliance. The PCI SSC does not enforce compliance itself. For HITRUST, a licensed assessor and the HITRUST process produce the result, and the certification lasts one year for e1 and i1 and two years for r2. Ask your auditor or assessor early which evidence format they accept, because the same retention controls can be documented in different ways.
How DeployOpen implements retention-controlled model hosting
DeployOpen plans, implements, hardens and documents self-hosted open-source platforms inside the client's own infrastructure, including cloud accounts, data centers and air-gapped networks. For teams whose Amazon Bedrock data retention modes conflict with their data controls, we scope a self-hosted model hosting stack that keeps prompts and completions under the client's own retention rules.
How we scope
- We map every model in use against its Bedrock retention mode and flag any that require aws_review.
- We confirm which workloads carry regulated data and which controls, such as SCP enforcement, must hold.
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we build
- We deploy open-weight models on vLLM within your own cloud account, data center or air-gapped network.
- We place an OpenAI-compatible gateway in front of vLLM and restrict endpoints to those users need.
- We configure logging and metrics so prompt and completion retention is set by your policy, not a vendor default.
How we hand over
- We document architecture, retention settings, access paths and the evidence an auditor or assessor would request.
- We test the deployment against your written retention and access requirements before handover.
- We deliver runbooks so your team can operate, back up and upgrade the stack without us.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- We track vLLM releases and plan upgrades in a maintenance window agreed with your team.
- We prepare systems and evidence for compliance assessments, while independent auditors issue any audit reports or attestations.
