Azure OpenAI abuse monitoring and when self-hosting is the cleaner answer
If your prompts can contain patient records, cardholder data or customer files, you need to know whether flagged content is stored for human review and who can see it. Azure OpenAI's abuse monitoring can retain prompts and completions that it flags for review by Microsoft personnel, unless your subscription has been approved for modified abuse monitoring. That approval changes who may review the data and under what conditions, but it does not remove the review path. Confirm with your Microsoft account team which subscriptions are approved, which regions and deployments the approval covers, and how long flagged content is kept. Record those answers in your data flow documentation. This is a constraint-driven decision about where review can happen, not an assessment of the product's overall quality.
Self-hosted open-weight models served on vLLM remove the third-party review path. Prompts go to an endpoint in your own network, and no vendor retains them for abuse review. Your own logs and retention rules still apply, so you decide what is stored. vLLM exposes OpenAI-compatible routes such as /v1/chat/completions, so many applications only need a new base URL. Be aware that vLLM's --api-key option does not cover every route, including /metrics and /health. The vLLM security documentation recommends a reverse proxy that allowlists only the endpoints users should reach, and that proxy belongs in your change scope.
Before choosing, map the regulations that apply to the data. Under HIPAA, a business associate contract must require the vendor to follow the Security Rule and report security incidents, so ask Microsoft and any other provider for that language in writing. Under DORA, Article 30(2) requires an ICT services contract to state where data is processed and how data is returned on termination. HHS does not issue HIPAA certifications, so do not accept a certificate as proof of compliance. Self-hosting shifts these obligations to your own controls and to your assessor, which is the trade-off to weigh against the vendor's review and retention terms.
What regulated content requires from an LLM endpoint
Prompts and completions from regulated work often contain patient records, account details or cardholder data, so the endpoint that handles them becomes part of your compliance scope. The cards below set out the obligations that touch that data and the questions each one raises for a hosted or self-hosted model.
HIPAA business associate terms
If your workloads send electronic protected health information to a model endpoint, the provider may be a business associate under 45 CFR 164.314(a)(2)(i). That contract must require Security Rule compliance, flow the same terms to subcontractors that handle ePHI, and require reporting of security incidents. Ask your provider whether it will sign such terms and how it handles any flagged content it retains for review.
HIPAA technical safeguards for model access
The technical safeguards in 45 CFR 164.312 require unique user identification, an emergency access procedure, automatic logoff and encryption decisions. For an LLM endpoint, this means every calling service and person needs a distinct identity, logs must show who sent which prompt, and transport and storage encryption must be documented. Check which of these controls you configure yourself and which the provider operates.
Access control on patient data
France's CNIL fined Hopital Prive de la Loire EUR 500,000 under GDPR Articles 32 and 34 on 3 September 2026. The regulator found external access lacked VPN and multi-factor authentication and that access ignored the care team's need to know. For a model endpoint, apply the same test: restrict which services and staff can send prompts, require strong authentication for remote access, and limit each key to the data it needs.
Stating where data is processed
DORA Article 30(2) requires every ICT services contract for a financial entity to state where services are provided and data processed, to give advance notice of changes, and to cover return of data on insolvency or termination. Applied to an LLM endpoint, you need the processing region for prompts, logs and any retained review copies, a contractual notice period for moves, and a tested way to recover your records and configuration if the provider exits.
Cardholder data in prompt scope
PCI DSS applies to any entity that stores, processes or transmits cardholder data, or that could affect the security of the cardholder data environment. If a support assistant or document tool passes card numbers or sensitive authentication data to a model, that endpoint and its logs may fall in scope. Confirm whether your assessor will treat the model service as in scope and whether you can keep card data out of prompts through upstream redaction.
What Azure OpenAI stores, where and for whom
Under default settings, Azure OpenAI screens prompts and completions automatically to detect content that breaches Microsoft's usage rules. Content that is flagged can be kept so that Microsoft personnel may review it. The purpose of that review is to judge misuse, not to serve your application, so the stored material sits outside the request path your users see. Treat this as a data handling decision that the default configuration makes on your behalf, and record it in your data inventory.
- Trigger: Only content that the automated classifiers flag is kept for human review under default abuse monitoring.
- Who reviews: Reviewers are Microsoft personnel. Your engineers do not select them or control their access.
- Retention and location: Confirm the retention period, storage region and reviewer locations in the current Microsoft documentation for your deployment type. Do not assume them.
- Scope: Confirm which deployment types, including Global and regional options, fall under default monitoring, since the terms may differ between them.
- Regulated content: Health, payment and personal data inside prompts can enter this review path if the content is flagged.
Check with your account team in writing which records are flagged and kept for your subscription and deployment type, and file the answer with your compliance records.
Customers can apply for modified abuse monitoring. If Microsoft approves the application, the flagged-content review that applies by default changes for the approved use. Microsoft sets the eligibility criteria, the scope of the change and what remains stored, and it may update them. For that reason, treat the approval as a written commitment to verify, and do not rely on a general statement about it.
- Coverage: Confirm which subscriptions, resources and deployments the approval names, and whether it reaches existing deployments.
- Remaining storage: Ask in writing which data paths still apply after approval, including whether any flagged content is still kept and for how long.
- Timing: Ask when the approval takes effect and whether it covers data already stored before the approval date.
- Evidence: Keep the approval confirmation with your records. An auditor will ask for it, so store it where your compliance team can produce it.
- Changes: Ask under what conditions Microsoft can withdraw or change the approval and how you would be told.
Check the written scope and eligibility criteria with your Microsoft account team, then compare them with the current Azure OpenAI data privacy documentation before your compliance team relies on them.
Microsoft may offer zero data retention for some configurations. It is a separate option from modified abuse monitoring, and its availability depends on your contract, eligibility, models and regions. Ask what it stops storing before you design a system around it, because the answer determines which of your compliance controls still need to cover Azure OpenAI.
- Availability: Confirm whether zero data retention is offered to your organization, and for which models, regions and deployment types.
- Eligibility: Ask what approval or contract change is required and whether it is a step separate from modified abuse monitoring.
- Coverage: Ask which records it covers, such as prompts, completions, stored responses, fine-tuning files and abuse monitoring data, and which it leaves in place.
- Features: Confirm which Azure OpenAI features behave differently or stop working, since features that store state may depend on retained data.
- Your own logs: Your application may still log prompts and completions. Zero data retention on the Microsoft side does not change those records, so include them in your data inventory.
Check the written scope and the list of affected features with your account team, then test a sample request in a non-production subscription to confirm what your team can observe.
Which open-source options keep prompts under your control
This table compares the open-source components that would serve models, route requests and store traces in your own infrastructure, instead of sending prompts to a hosted service. Read each row against your own data flows, because a tracing tool that captures prompts creates a second copy that needs the same controls as the model itself. The main takeaway is that the model server and the observability stack each need their own storage and access decisions.
| Option | How it keeps prompts and completions in your infrastructure | Trade-offs | What to confirm |
|---|---|---|---|
| vLLM serving open-weight models | The OpenAI-compatible server runs on GPUs you operate, so prompts and completions are processed inside your network and are not sent to a model provider's abuse review queue. | You own model choice, version pinning, GPU capacity and on-call work. Inference logging is your decision, and default settings do not decide retention for you. | Confirm which request and response fields your server writes to logs, and whether any vendor support channel can receive transcripts. |
| vLLM API exposure | The --api-key option protects only the /v1, /v2, /inference and /cohere paths, so other endpoints remain reachable unless you filter them. | A reverse proxy that allowlists user-facing endpoints adds a component to build, patch and monitor. | Confirm that /metrics, /health and control endpoints are blocked from untrusted networks in your deployment. |
| LiteLLM as the gateway | A self-hosted server gives applications one OpenAI-compatible endpoint, with virtual keys, budgets and logging that you configure and store yourself. | Logging to Langfuse, Datadog or OpenTelemetry sends prompt content to those systems. SSO, SAML and audit logs sit in the enterprise directory under a separate license. | Confirm which callbacks are enabled, what each one stores, and whether the enterprise license covers the audit and SSO controls you need. |
| Langfuse for traces | A self-hosted deployment on Kubernetes stores traces in PostgreSQL, ClickHouse and S3 or blob storage that you control, with no vendor review of content. | Advanced features need an enterprise license key. Trace storage becomes a second copy of prompts that needs retention rules and access control. | Confirm which features your use case needs fall under the enterprise license, and set a retention period for traces that contain prompts. |
| Opik for traces and evaluations | Apache-2.0 licensed, and the full platform can be self-hosted without a commercial license, so traces stay in your database and object storage. | Smaller ecosystem and fewer enterprise integrations than some alternatives. Evaluation datasets built from production prompts also need protection. | Confirm the storage backends, authentication options and backup approach in the current documentation before you rely on them for regulated data. |
| Arize Phoenix for tracing | Self-hosted use keeps OpenTelemetry traces with your own storage, and it can be run in your environment. | Elastic License 2.0 is not OSI-approved and forbids offering Phoenix to third parties as a hosted or managed service. Legal review is needed if you run it for other organizations. | Ask counsel whether your internal use and any service provider arrangement fall within the license terms. |
| Log redaction and storage location | Tracing tools can capture full prompts and completions, so they can hold the same patient, customer or cardholder data that abuse monitoring would review. | Built-in PII redaction varies by product. Portkey lists PII redaction as an enterprise feature outside its open-source gateway, and you may need your own filter. | Confirm which fields are masked before storage, which regions hold backups, and who can read raw traces. |
How to deploy vLLM behind a gateway under this constraint
This sequence moves inference off a shared abuse-review path and onto endpoints you control, with network isolation, a narrow public surface, gateway-enforced keys, monitoring and traceable model versions. The output is a deployment where every prompt and completion stays inside your boundary and each change is recorded.
Isolate the serving nodes on a private network
The platform or network team places the vLLM hosts and any multi-node workers in a private subnet with no public ingress. The vLLM security documentation states that node-to-node traffic in a multi-node deployment is insecure by default, with no authorization or encryption, so the isolation is a requirement, not an option. Done means a network test from outside the subnet cannot reach the vLLM ports, and the worker nodes can reach each other only on the ports the cluster needs.
Expose only user-facing paths through a reverse proxy
The security team configures a reverse proxy in front of the vLLM API server that allowlists only the endpoints applications need, such as the chat completions and embeddings paths. The vLLM --api-key option does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the documentation warns against relying on the key alone. Done means requests to those unprotected paths return a denial from the proxy, and an allowed path succeeds only with a valid key.
Route applications through LiteLLM with virtual keys
The platform team runs the self-hosted LiteLLM AI Gateway as the single OpenAI-compatible entry point for applications, and registers the vLLM endpoints as upstream models. Teams receive virtual keys with budgets and rate limits, so no application holds the vLLM credential directly. Done means each application key is issued and revoked in the gateway, and a test call through the gateway reaches the vLLM backend and is logged with the key identity. Confirm which LiteLLM features your deployment needs and whether they fall under the enterprise directory license.
Enable Prometheus scraping for serving and gateway metrics
The operations team points a Prometheus server at the vLLM /metrics endpoint, which is served on the API server and covers request counts, token throughput, latency histograms and KV cache use, and at the gateway's Prometheus output. Because /metrics is blocked at the proxy for users, the scrape should come from the monitoring network only. Done means dashboards show live throughput and KV cache use, and an alert fires when the test endpoint stops returning metrics.
Point tracing at self-hosted Langfuse or Opik with redaction
The engineering team sends gateway traces to a self-hosted Langfuse or Opik instance inside the same boundary, so traces are not stored by a third-party review process. Both platforms can run on infrastructure you control, but Langfuse depends on PostgreSQL, ClickHouse, Redis or Valkey, and S3 or blob storage, and some Langfuse features need an enterprise key. Configure redaction of prompt and completion fields before storage, and set retention. Done means a test prompt containing a dummy identifier appears in the trace store in redacted form only.
Pin model weights and the vLLM release
The platform team records the exact open-weight model revision, its checksum, the vLLM version and the container image digest in a versioned deployment manifest. vLLM ships new minor releases about every two weeks, so an unpinned upgrade can change behavior without a review. Upgrades should go through a staging environment and a replay of your own evaluation set before promotion. Done means the manifest identifies the running release, and a staging change requires a recorded approval before it reaches production.
Record the deployment as compliance evidence
The compliance lead collects the network diagrams, proxy allowlist, key issuance records, scrape configuration, redaction settings, retention policy and the pinned release manifest into one evidence folder. These records support your own assessments, but they do not constitute an audit report or certification, which an independent assessor or certification body issues. Done means each control in your framework has a named artifact and an owner, and an internal reviewer can trace a sample request from the gateway log to the pinned model version.
What modified abuse monitoring changes and what it does not
Modified abuse monitoring changes one part of the processing chain: the path that flagged prompts and completions take to human review. It does not remove Azure from that chain. Requests still run on Azure infrastructure, Azure still operates the service, and your data still passes through Microsoft systems. The approval changes who reviews flagged content, not whether Azure processes your data. Treat the approval as one changed control inside a larger design, not as a finding that the workload is cleared for regulated content. Confirm in writing which resources, subscriptions and models the approval covers, the date it took effect, and whether Microsoft can change or withdraw it.
Where Azure handles electronic protected health information for a covered entity or business associate, the contract is the first control to check. Under 45 CFR 164.314, a business associate contract must require the business associate to comply with the Security Rule, flow the same terms down to subcontractors that handle ePHI, and report security incidents, including breaches of unsecured PHI. The abuse monitoring approval is a separate matter. It does not stand in for that contract, and it does not satisfy any HIPAA standard, because HHS states that no Security Rule standard requires certification. Ask for the signed agreement that covers your Azure OpenAI resources, and confirm which subcontractors and incident reporting terms apply to the human review path.
Turn both documents into entries in your risk register. For each deployment, record the approval status, the review path for flagged content, the subprocessors involved, the incident notice terms and the processing location. If the workload falls under DORA, Article 30(2) requires every ICT services contract to state where services are provided and data processed, and to give advance notice of changes, so compare those terms with the approval. Set a date to re-check both, because approval scope and contract terms can change. If the risk owner cannot accept the remaining exposure for regulated content, a self-hosted model served on vLLM in your own environment is the option to evaluate next.
Evidence auditors and regulators ask for
Whether you keep Azure OpenAI or serve open-weight models yourself, the same evidence decides the audit. Auditors and regulators want to see where prompts and completions flow, who is bound by contract to protect them, who can reach them, and what gets logged and kept. Each card below names one artifact and what to check in it.
Data flow diagrams with trust boundaries
Map every path a prompt and completion takes, from the application to the gateway, the model server, the tracing store and any review or storage tier. Mark each trust boundary and state which party operates each component. For Azure OpenAI, the diagram must show where Microsoft may store flagged content under abuse monitoring. For a self-hosted vLLM stack, it must show that the model endpoint sits on an isolated network, since the vLLM security docs state that inter-node traffic is unauthenticated by default.
Processor and business associate agreements
Collect the contract that defines who may handle prompts and patient or cardholder data, and where it may be processed. For HIPAA, a business associate contract must require Security Rule compliance, flow the same terms to subcontractors handling ePHI, and require reporting of security incidents under 45 CFR 164.314. Under DORA, Article 30(2) requires every ICT services contract to state service locations, data processing locations and data return on termination. Ask counsel to confirm these terms with each vendor, including the gateway and tracing vendors.
Access control and MFA records
Keep exported lists of who holds administrative and data access to the gateway, the model host, the trace database and the object store, along with the MFA enrollment and enforcement evidence for each. The FTC Safeguards Rule requires multi-factor authentication for any individual accessing an information system unless the Qualified Individual approves an equivalent control in writing. Check that the vLLM --api-key option is not your only control, because it does not protect /metrics, /health or /invocations.
Logging and tracing configuration with retention
Document what the tracing and gateway layers record, such as prompts, completions, user identifiers and token counts, and how long each store keeps them. Abuse monitoring retention on the Azure side is a contract question you must confirm with Microsoft. For self-hosted stacks, record the settings in configuration files and the retention jobs that enforce them. Confirm whether logging of prompt bodies can be turned off or redacted before data reaches the trace store, and whether the vendor's enterprise features needed for redaction are licensed.
Model and version inventory
Maintain a register listing each model, its exact weights version, the serving engine release, the deployment environment and the date each version went live or was retired. Regulators reviewing an AI-assisted clinical or financial workflow will ask which model produced a given output. A pinned vLLM release, such as v0.31.0 released on October 5, 2026, together with its weight hash, gives you a fixed reference point, while hosted model versions change on the vendor's schedule.
Vendor and residency statements under DORA
For EU financial entities, DORA Article 28(3) requires a register of all ICT service contracts, and Article 30(2) requires each contract to name processing locations and give advance notice of changes. Build one register row per model, gateway and tracing vendor, with residency statements taken from the contract rather than from marketing pages. The DORA register is a documentation duty, not a certification, so the evidence you produce is your own record and not an authority-issued attestation.
Questions buyers ask before choosing self-hosted models
Can we switch from Azure OpenAI to self-hosted models without rewriting our applications?
In most cases, the application code changes are limited, because vLLM's server, started with vllm serve, exposes OpenAI-compatible endpoints including /v1/chat/completions, /v1/responses and /v1/embeddings. Applications written for the OpenAI API can point their base URL at your server. The work that remains is usually outside the call itself: prompt behavior, output format checks, tool-calling behavior, rate limits and error handling differ between models and need to be tested on your own prompts before cutover. A gateway such as LiteLLM can hold the endpoint map so that routing changes do not touch every service. Confirm the exact endpoints and parameters your applications use against the vLLM documentation before you plan the switch.
What drives the cost of self-hosting, and is there a vLLM license fee?
vLLM is Apache-2.0 licensed and has no paid edition, so there is no license fee to budget. The recurring cost comes from GPU or other accelerator capacity, the surrounding infrastructure (storage, networking, a gateway, and a tracing and monitoring stack such as Langfuse or Opik), and the people who build and operate it. Utilization is the main driver: idle accelerators cost the same as busy ones. Implementation effort is mostly one-time, while capacity, patching, model upgrades and on-call are recurring. When you ask for a quote, request each component as a separate line item, state the expected request volume and peak concurrency, and ask which costs are fixed to a scope and which recur under a retainer.
What does our internal team need to provide during a self-hosting project?
Your team needs to own decisions that a vendor cannot make for you. These include which workloads move first, which data classes may be processed on which hardware, who approves model versions, and who is accountable for the security controls around the endpoint. Engineers need access to the cluster or servers, identity and network changes, and a way to run evaluations on real prompts. Compliance and privacy staff need to review where prompts and completions are stored, retained and logged. Plan for an on-call rota that covers GPU node failures, model load errors and capacity spikes. The exact staffing depends on your scope, so ask for a written list of internal roles before the work starts.
How long does migrating from Azure OpenAI to self-hosted models take?
The timeline depends on how many deployments you have, how many applications call them, how much evaluation your prompts need, and whether hardware must be procured or is already available. Inventorying deployments and API versions is usually the first step, followed by standing up a test environment, choosing replacement models and running side-by-side evaluations before any production traffic moves. Provisioned workloads have their own contract and retirement dates, which can set the deadline. No duration is stated here because no researched figure supports one. Ask for a scope with named milestones and a written schedule, and ask which steps depend on hardware delivery or on your change approval process.
Is a Microsoft approval for modified abuse monitoring enough for HIPAA or other regulated content?
An approval changes how Azure OpenAI handles flagged content, but it does not by itself make an organization compliant with HIPAA or any other framework. Under 45 CFR 164.314(a)(2)(i), a business associate contract must require the vendor to comply with the Security Rule, flow the same terms to subcontractors that handle ePHI, and report security incidents to the covered entity. Your team still has to confirm the approval scope, what is retained and where, and whether the contract terms meet these requirements. HIPAA has no certification, so no approval or private attestation proves compliance on its own. Serving models in your own environment keeps prompts and completions under your control, but it also moves the safeguards to you. Confirm these points with your legal and compliance teams and with the vendor in writing.
What do HIPAA civil penalties look like for a security failure?
For penalties assessed on or after 28 January 2026, 45 CFR 102.3 sets an inflation-adjusted civil money penalty range from $145 per violation to $2,190,294 per violation, with a calendar-year cap of $2,190,294. The actual amount depends on the level of culpability and the facts of the case, and penalties are assessed by HHS's Office for Civil Rights through complaints, compliance reviews and audits. The per-violation structure means that a single control failure affecting many records or many days can be counted in several ways. Penalties are one exposure among several. Breach notification, contract remedies and reputational harm also apply. Have counsel confirm how these ranges apply to your data flows.
Does self-hosting vLLM handle compliance, security and access control on its own?
No. vLLM provides the inference engine, not a complete security layer. The vLLM security documentation notes that its --api-key option does not protect every endpoint, including /invocations, /metrics and /health, and recommends a reverse proxy that allowlists only the endpoints meant for users. Multi-node communication is insecure by default and must be isolated on a separate network. You will need to design authentication, authorization, logging, network segmentation, encryption and monitoring, and document how they map to the frameworks that apply to you. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any audit report or attestation.
Primary sources to read before deciding
Start with the vLLM security documentation, because the exposure decisions you make there determine what a self-hosted endpoint reveals. The --api-key option protects the /v1, /v2, /inference and /cohere routes, but it does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the documentation warns against relying on it alone. The recommended pattern is a reverse proxy that allowlists only the endpoints your users need, so control routes and metrics stay on an internal network. Multi-node deployments add a second exposure: traffic between nodes is insecure by default, with no authorization or encryption, so place those nodes on an isolated network. Because these details change between releases, confirm them against the exact vLLM version you pin before you write the architecture into your risk assessment. Read the vLLM security documentation.
Next, read Appendix A to Subpart C of Part 164 alongside 45 CFR 164.314 in the eCFR. Appendix A lists 18 Security Rule standards across administrative, physical and technical safeguards, with 36 implementation specifications, of which 14 are required and 22 are addressable. Addressable specifications still require an assessment, a documented decision and, where you decline one, an equivalent measure. Section 164.314 matters once a third party touches electronic protected health information, since a business associate contract must require Security Rule compliance, flow the same terms down to subcontractors that handle ePHI and report security incidents to the covered entity. Self-hosting moves several of these safeguards onto your own team, so assign a named owner to each specification. Whether HIPAA applies to your workload is a question for counsel. Read Appendix A to Subpart C of Part 164.
Finally, read the HHS FAQ on certification before accepting any vendor claim of HIPAA certification. HHS states that no Security Rule standard requires an entity to certify compliance and that private certifications do not prevent HHS from later finding a violation, so your evidence should come from your own risk analysis and documented safeguards. For the open-source stack, read the LiteLLM and Langfuse self-hosting documentation directly: LiteLLM's enterprise directory carries its own license, Langfuse keeps some advanced features behind an enterprise license key, and Langfuse needs PostgreSQL, ClickHouse, Redis or Valkey and S3 or blob storage inside your environment. Last, confirm Microsoft's current abuse monitoring terms, retention periods and modified abuse monitoring approval process directly with Microsoft, in writing. Those terms change, and this page does not restate them as current. Read the HHS FAQ on certifying compliance.
How DeployOpen deploys vLLM under regulated data constraints
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For abuse monitoring questions, that means scoping where prompts and completions are processed, deploying open-weight models on vLLM behind your own controls, and handing over evidence your assessors can review.
How we scope
- Map which workloads send prompts or completions to Azure OpenAI and which data classes they carry
- Record where each flagged-content path would store data today and who would have access to it
- Agree the written scope, deliverables and timelines before any build work starts
- Define the endpoints, regions and network boundaries the self-hosted vLLM service will use
How we build
- Deploy vLLM in your environment with access limited to the endpoints users need, behind a reverse proxy
- Place cluster nodes on an isolated network, since inter-node traffic is unauthenticated by default
- Configure logging, metrics and retention so you control what is kept, for how long and where
- Test the deployment against the security controls and data handling rules agreed in the scope
How we hand over
- Document the architecture, data flows, retention settings and access model for your own records
- Provide runbooks for model updates, pinned versions and incident handling
- Deliver evidence packs for your assessors, noting that DeployOpen prepares evidence but does not issue audit reports or certifications
- Train your operators on day-to-day running, monitoring and upgrade procedures
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines confirmed in writing
- Ongoing support, upgrades and continuous security testing are available as a retainer
- Quotes follow from the written scope, so ask for each cost component to be itemized
- Revisit controls when your regulator, contract or model inventory changes
