LLM guardrails for healthcare

Why LLM guardrails for healthcare need a different build

A clinical or operations team that wants an assistant touching protected health information should expect a guardrail layer built from NeMo Guardrails or Guardrails AI, enforced at an AI gateway, with detection models running on premises. The gateway applies input and output checks to every request, the guard libraries define what counts as a violation, and the classifier models that detect identifiers and out-of-scope clinical advice run inside the organization's own infrastructure so that the text being inspected never goes to a third-party moderation API. This is the same pattern used for other regulated assistants, but healthcare adds requirements on where the data travels and what must be recorded.

The HIPAA Security Rule (45 CFR Part 164 Subpart C) applies to covered entities and their business associates for all electronic protected health information they create, receive, maintain or transmit (45 CFR 164.302). That scope reaches beyond the chat screen. Patient identifiers can appear in the prompt, in retrieved chunks from a knowledge base, in model responses and in traces, prompt logs and evaluation datasets. Each of those stores becomes part of the ePHI inventory, so the guardrail design should decide early which of them keep raw text, which keep only hashed or redacted text, and how long each is retained. Confirm with your privacy office and counsel whether the assistant vendor, the model provider and any observability tool are business associates under 45 CFR 164.314 before any PHI enters a prompt.

HIPAA has no certification. HHS states that no Security Rule standard requires an entity to certify compliance, and the HHS Office for Civil Rights enforces the rule through complaints, compliance reviews and audits. The evidence you will be asked for is therefore the documented set of controls: the policy configuration and its version history, the test results showing that redaction and scope rules work on representative data, the decision logs for blocked and redacted requests, and the access records for who can change the rules. Ask your implementation partner to design these artifacts before go-live, because retrofitting logging into a running assistant is far harder than building it in.

Which HIPAA and related obligations shape the guardrail design

This table maps each obligation to a concrete guardrail design choice, what the assessor will look for, and where teams usually fall short. Read it as a design checklist: each row names the build decision it drives, and the main takeaway is that the guard layer itself becomes part of the audit trail for any assistant that touches PHI.

Obligation	What it asks for	Guardrail design choice	Where teams usually fall short
Unique user identification (45 CFR 164.312(a)(2)(i), required)	Each user of ePHI systems has a unique name or number so activity can be tied to a person	Pass the authenticated user identity from the chat front end through the AI gateway into every guard decision and log record	Guards log a shared service account, so blocked or redacted prompts cannot be traced to a clinician
Automatic logoff (45 CFR 164.312(a)(2)(iii), addressable)	Electronic sessions end after a period of inactivity	Set session expiry in the assistant and gateway, and make sure guard context and cached conversation state are cleared on logoff	Session tokens or conversation memory persist in the gateway or observability store after the user leaves
Encryption and decryption (45 CFR 164.312(a)(2)(iv)) and transmission security (45 CFR 164.312(e))	ePHI is protected at rest and in transit, with encryption assessed as addressable	Keep classifier models, guard inputs, guard outputs and logs inside the covered environment with encrypted links between guard services	Guard servers or validator packages call external endpoints, or traces store raw prompts unencrypted
Audit controls (45 CFR 164.312(b))	Mechanisms record and examine activity in systems that contain or use ePHI	Log each input and output decision with timestamp, user, policy version, rail that fired and whether text was redacted, without storing the full PHI payload where avoidable	Logs capture the rejection but not the policy version, so a reviewer cannot reconstruct why a response was allowed
Business associate contract terms (45 CFR 164.314(a)(2)(i))	Business associates comply with the Security Rule, flow terms to subcontractors handling ePHI, and report security incidents, including breaches of unsecured PHI	Confirm which vendor, model host or validator provider touches ePHI, and route guard incident alerts to the covered entity's reporting process	Hosted moderation APIs or third-party guard services process prompts without a contract that covers them
Proposed Security Rule revision (90 FR 898, not final as of October 2026)	HHS proposed stronger requirements, including on cybersecurity and asset inventory, with final action projected for July 2027 in the Unified Agenda	Design for the proposed direction now: maintain an inventory of every guard component, model file and version, and document configuration changes	Teams treat the current rule as final and have no inventory of which classifier or policy version ran on a given date
CNIL fine under GDPR Articles 32 and 34 (3 September 2026, EUR 500,000)	Security of processing, and notification of individuals after a breach, with access limited to what each team needs to know	Enforce need-to-know policy scopes per clinical role in the guard layer, and require multi-factor authentication for any external access to the assistant or its logs	Remote access to the assistant or log viewer bypasses the controls applied to the EHR, and access rules ignore the care team's actual need
NIS2 Annex I health sector (Directive (EU) 2022/2555), where EU operations apply	Covered healthcare providers and certain medical device and pharmaceutical manufacturers meet security and incident reporting duties	Include guard alerts and blocked-data events in the incident response process, with on-premises classifiers so no PHI leaves the EU boundary	Guard telemetry goes to an SaaS dashboard outside the scope of the incident response plan

Which open-source guardrail and gateway platforms fit clinical data

Healthcare teams usually combine a guardrail framework, a gateway that enforces it, and a trace store that records decisions. The options below differ in license terms, deployment model and how much policy work they leave to your team. Confirm each item against your own PHI handling before you commit.

NeMo Guardrails for policy rails

NeMo Guardrails is Apache-2.0 licensed and defines input, dialog, retrieval, execution and output rails in Colang, with Colang 1.0 as the default. It fits scope control, such as keeping an assistant away from diagnosis or dosing advice, and it ships self-check and jailbreak detection components. Its microservice option runs on Kubernetes with Helm, which helps keep classifier models inside your environment. Before go-live, confirm which built-in detection models you can run fully on premises and whether Colang 2.x migration work is needed for any existing configuration.

Guardrails AI for PHI validators

Guardrails AI is Apache-2.0 licensed and builds guards from validators installed from Guardrails Hub, such as regex matching for medical record numbers, dates of birth or member IDs. Validators can be chained, so a redaction step can run before a toxicity or scope check. It can run as a standalone REST service started with the guardrails start command, which lets applications call it through the OpenAI SDK. Confirm that every Hub validator you use runs without outbound calls, and note that guardrails configure asks whether to enable anonymous metrics reporting.

LiteLLM for enforcement and logging

LiteLLM is an AI gateway that sits between applications and model providers, and it supports guardrails and logging at the gateway layer. Placing guard calls there means every request to a model passes the same checks regardless of which application sent it. It is MIT licensed except for its enterprise directory, so confirm which features your deployment needs fall outside that directory. Check where request logs are stored, how long they are retained, and whether prompt content is masked before it reaches the log store.

Apache APISIX as an AI gateway

Apache APISIX is a cloud-native API gateway that also provides AI gateway functions, and it is Apache-2.0 licensed. It suits organizations that already run APISIX for authentication, rate limiting and partner access, because guard routing can then follow existing access policy. Confirm which AI-specific plugins are available in your chosen release and whether they can call a self-hosted guardrail service. Test how the gateway handles failures in the guard service, since a failed check must fail closed for PHI routes.

Langfuse for decision traces

Langfuse is an open source LLM observability platform licensed under MIT, except its ee directories. It can store traces of prompts, guard decisions and model outputs in your own database, which supports the HIPAA audit trail you need to reconstruct what the assistant saw and what it blocked. Restrict who can view traces, because raw prompts can contain PHI even after redaction. Confirm which ee features you need and whether your retention schedule matches your audit obligations.

Phoenix licensing limits

Arize Phoenix is an LLM tracing and evaluation tool licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. Running it inside your own network for your own staff is generally the use the license permits, but a vendor that hosts Phoenix for you may not be allowed to. Ask legal counsel to review the license text against your deployment model, especially if a managed service partner will operate the tracing stack for you.

How input and output rails redact PHI and block out-of-scope advice

Input rails run before any model call. A NeMo Guardrails configuration, defined in a config.yml file with optional Python actions and Colang files, can route each user prompt through detection for patient identifiers such as names, record numbers, dates of birth and contact details. Matches can be masked before the prompt reaches the model, and the decision is written to a log record. Output rails apply the same kind of detection to the model's answer, which matters because a model can reproduce identifiers from a retrieved chart, a pasted note or its own context. Retrieval rails on RAG chunks, if the assistant uses one, cover the third path. Confirm with your security and clinical informatics leads which identifier types must be masked and which can pass, because that list drives the policy you write.

Dialog rails hold the assistant to approved clinical scope. Colang flows define which topics the assistant may answer, such as scheduling, policy questions or general education content, and which it must decline or hand to a clinician, such as individual dosing or diagnosis. Colang 1.0 is the default, and Colang 2.0 is also supported. NVIDIA's migration guide says a nemoguardrails convert run between versions still needs manual review, so plan that review as a named task with a clinical reviewer signing off on converted flows. Build a fixed test set of in-scope and out-of-scope questions and rerun it after every policy change.

Detection is probabilistic, so measure it on your own records. No identifier detector catches every case, and a detector tuned to be strict will flag legitimate clinical text, such as a medication name that resembles a surname or a lab value that looks like a date, which blocks a valid question. Vendor samples will not show your note templates, abbreviations or EHR export format. Use de-identified or synthetic records that match your templates, record both missed identifiers and false positives, and set thresholds with clinicians who can judge the blocked text. Ask any implementation partner to deliver this measurement as part of acceptance, not as an afterthought.

Deployment patterns for hospitals, payers and business associates

In a data center the covered entity operates, the guardrail stack usually runs on hardware the organization owns or leases directly. NeMo Guardrails or Guardrails AI sits behind the AI gateway, and the detection models for patient identifiers, scope checks and output screening load onto the same cluster as the language model. Because no third party hosts the prompts, the main contract question is with the vendors that supply software and support, not with a cloud host. The HIPAA Security Rule still applies to every copy of electronic protected health information the stack creates, including guard decision logs.

Guard model placement: Run classifier and redaction models on internal GPU or CPU nodes, with the model files stored in an internal registry so the cluster does not pull them from the internet at inference time.
Log storage: Write blocked-request and redaction records to an internal log store with access control and retention settings that match your records policy, and keep raw prompts out of general application logs.
BAA scope: Where a vendor or support firm can see ePHI during installation, upgrades or troubleshooting, a business associate contract under 45 CFR 164.314 must be in place before that access begins.
Hosting provider checks: Confirm with your facilities and network teams who has physical and administrative access to the racks and backup media.
Testing environment: Keep test prompts that contain real patient data inside the same controls as production.

Confirm with each software vendor whether any telemetry or license check sends data outside your network, and document the answer for your security risk analysis.

How a healthcare guardrail rollout moves from pilot to audit-ready

This sequence takes a healthcare assistant from a bounded pilot to a production guardrail layer with validated redaction, scope checks, decision logs and monitoring, with each phase producing evidence the privacy officer and security team can review.

Map PHI data flows and approved assistant scope
The organization, with DeployOpen, documents where protected health information enters the assistant, which systems it touches (EHR extracts, document stores, ticketing, chat history), which users and roles can prompt it, and which clinical and administrative tasks the assistant is approved to support. The output is a data flow diagram and a written scope statement signed off by the privacy officer and a clinical owner. Done means every PHI path has a named owner and every out-of-scope task is listed as a refusal category.
Select and configure rails and the AI gateway
The team chooses the guardrail framework (NeMo Guardrails, Guardrails AI or both), decides which input and output rails apply, and places enforcement at the self-hosted AI gateway so every request passes through the same controls. Configuration is kept in version control with a named approver for each policy change. Done means a test environment routes all traffic through the gateway, the rail configuration builds from source, and the guard models needed for detection are running on premises.
Build a de-identified test corpus and red-team set
The team assembles prompts and documents that mimic real clinical and administrative requests using synthetic or de-identified records, plus adversarial prompts for injection, requests for clinical advice outside scope and attempts to extract identifiers. The corpus is stored in a controlled repository, because it must not itself become a PHI store. Done means each risk category in the scope statement has labeled test cases with expected outcomes agreed by the privacy and security teams.
Validate redaction and scope checks with reviewers
Clinical staff and privacy staff review a sample of guard decisions, including missed identifiers, over-redacted clinical content and refusals that should have been answered. The team records false positives and false negatives by category, tunes thresholds or validators, and reruns the test set after each change. Done means reviewers have signed off on the results, residual misses are documented with compensating controls, and the privacy officer accepts the known limits of redaction.
Enable audit logging and retention
Each guard decision is logged with a timestamp, user or service identity, rail triggered, action taken (allow, redact, block or refuse) and a policy version identifier, while raw PHI is excluded or minimized in log content. The team sets retention periods with the privacy officer and confirms that logs are stored inside the organization's infrastructure with access restricted to named roles. Done means a sample of blocked and redacted requests can be reconstructed from logs without exposing unnecessary PHI, and the retention rule is written down. Confirm applicable retention obligations with counsel, since HIPAA does not set a single log retention period for this use.
Cut over with monitoring and change control
The assistant moves to a limited production group with dashboards for block and redaction rates, latency added by guard checks, and unusual refusal patterns. Policy changes go through the same review and test rerun used in validation, and incidents involving PHI follow the business associate and incident response procedures agreed with the covered entity. Done means monitoring alerts reach a named on-call owner, a rollback path has been exercised, and the security team holds a current evidence package covering scope, test results, configuration history and logs.

Evidence auditors and privacy officers ask for, and which component produces it

This table maps each piece of evidence that HIPAA assessors, covered entity privacy officers and business associate reviewers typically request to the component that generates it. Read it as a checklist to agree with your compliance lead before the guardrail design is frozen. DeployOpen prepares this evidence, but independent auditors and certification bodies issue any audit report, since HIPAA has no certification.

Evidence	What it shows	Producing component and what to confirm
Rail configuration version history	Which input, output and dialog rules were active on a given date, and who approved each change	NeMo Guardrails configuration files (config.yml, Colang) kept in version control with signed approvals. Confirm your change process links each commit to a ticket.
Logs of blocked requests	Prompts and responses stopped by a rail, with the rule that fired and the timestamp	Guardrail decision logs written by NeMo Guardrails or Guardrails AI validators, forwarded to your central log store. Confirm logs record the rule ID without storing raw PHI.
Logs of redacted requests	Proof that identifiers were masked before a model or log saw them, and how many were masked	Redaction events from Guardrails AI validators or NeMo Guardrails actions, with LiteLLM request logs showing the redacted payload that reached the model.
Access records for the guard and gateway	Who administered policies, who called the gateway and which service accounts were used	LiteLLM admin and key audit logs, plus identity provider and host logs for the guard service. Confirm unique user identification for every human administrator.
Model trace records	Input, output and cost of each model call, linked to the request ID used by the guard	Langfuse traces or Phoenix spans, stored in your own database. Confirm trace content is minimized or masked before storage to limit ePHI copies.
Redaction and scope test results	Repeatable results showing detection rates and out-of-scope refusals on a test set built from synthetic records	Test harness outputs from your red team or DeployOpen testing engagement. Confirm the test set, pass thresholds and retest dates are documented.
Business associate and subcontractor flow-down records	Signed contracts requiring Security Rule compliance, subcontractor flow-down and incident reporting	Contract file held by legal and procurement, not by a software component. Confirm each hosting or support subcontractor that touches ePHI is listed, as 45 CFR 164.314 requires.
Security incident reports	What happened, what was affected, containment steps and notifications made	Incident tickets linked to alert records from the gateway and log stores. Confirm your incident procedure defines when a blocked-request pattern becomes a reportable event.

What do healthcare teams ask before building LLM guardrails?

What does it cost to build healthcare LLM guardrails, and what drives the effort?

Cost depends mostly on the PHI data sources the assistant touches, the number of rails you need, how far validation must reach, and how much clinical review time your team can give. The software itself is open source, so the recurring spend is usually compute for any self-hosted classifier models, engineering time to write and tune policies, and ongoing red-team testing. Your costs are one-time for initial policy design and integration, then recurring for tuning, model updates and re-testing after each change. Ask any implementation partner for a fixed-scope quote that lists each rail, each data source, each validation set and the clinical reviewers named by role. DeployOpen does not publish a price list, and a quote follows from that written scope.

How long does a healthcare guardrail rollout take?

The timeline is set by how many PHI sources and clinical scope rules must be defined, how long clinical owners take to approve policies, and how much testing against representative data is needed before go-live. Teams that already run an AI gateway and a logging pipeline usually move faster than teams starting both from scratch. Ask for a written plan with phases for requirements, policy design, integration with the gateway, testing on de-identified or approved data, and a controlled pilot. Confirm with the partner how many review cycles are included, because clinical sign-off is often the longest step.

What must our internal team supply for the project?

Your team has to supply decisions that only your organization can make. Privacy and compliance staff define which identifiers count as PHI in your context and which disclosures are permitted. Clinical owners define the approved scope for the assistant and approve what it may and may not say. Security staff own network placement, identity for administrators, key management, log storage and access reviews. Engineering provides access to the model endpoints, the AI gateway and test environments. Name one accountable owner for each area before the work starts, because unclear ownership is a common reason policy changes stall after launch.

Can a hospital build and run healthcare guardrails on its own?

A capable internal team can build and run guardrails alone, but it needs engineers who can write rail configurations, test classifiers and maintain policies, plus clinical and privacy reviewers who stay engaged after launch. Open-source libraries such as NeMo Guardrails and Guardrails AI are available under permissive licenses, so the software is not the barrier. The harder work is building a validation set, documenting decisions and keeping controls current as models and policies change. Many organizations split the work: internal staff own policy decisions and operations, while an implementation partner handles integration, testing and documentation under a written scope. Decide which side owns each task before you compare options.

Does HIPAA certify our LLM guardrail setup?

No. HHS states that no Security Rule standard requires an entity to certify compliance, and private certifications do not prevent HHS from later finding a violation. HHS Office for Civil Rights enforces the rule through complaints, compliance reviews and audits. What you can show an auditor is evidence: documented risk analysis, configured safeguards for access and audit controls, logs of guard decisions, and business associate contracts that cover any vendor handling ePHI. Be careful with claims that a product or partner makes you HIPAA compliant. Ask partners to describe their work as readiness and evidence preparation, and confirm the business associate terms in writing.

How do guardrail logs support HIPAA audits, and how long should we keep them?

Guardrail logs record which input or output rule fired, the decision taken, the policy version and the time, which gives you evidence of audit controls and of how safeguards behaved. The HIPAA Security Rule requires audit controls under 45 CFR 164.312, but the rule does not set a single retention period for these records, so confirm the retention you need with your privacy and legal teams. Keep logs free of raw PHI where possible, store them with access controls, and avoid sending them to an uncontrolled observability tool. If the assistant is also a high-risk AI system under the EU AI Act, Article 19(1) requires providers to keep automatically generated logs for at least six months unless other law provides otherwise. Confirm the applicable rule for your jurisdiction.

How do HIPAA penalties relate to gaps in our guardrail controls?

Penalties attach to violations of the rule, and a missing or weak safeguard can be one of the facts that supports a violation finding. For penalties assessed on or after 28 January 2026, the inflation-adjusted civil money penalty runs from $145 to $2,190,294 per violation, with a calendar-year cap of $2,190,294, under 45 CFR 102.3 as published at 91 FR 3666. The amount assessed depends on factors HHS considers, including the culpability level and the response to the issue. Guardrails reduce exposure by limiting PHI disclosure, but they do not replace risk analysis, access control, encryption and workforce training. Confirm with counsel how your guardrail records fit into your overall risk management program.

What to ask an implementation partner before signing

Start with the test data. Ask how the partner builds and tunes detectors for PHI without copying real patient records into engineering environments, whether synthetic or de-identified clinical text is used, who approves that data set, and where it is stored and deleted. Ask where every detection model runs, including the NER or classifier models behind redaction and any topic or safety classifiers, and confirm that no inference call leaves your network. Then ask how model files and validator packages reach the environment, who performs the offline update, how each update is checked before promotion, and what happens to detection quality if an update is delayed.

Next, ask how policy changes are versioned and approved. A clinical scope rule, such as which questions an assistant may answer about medication dosing, should live in version control with a named clinical owner, a privacy or compliance reviewer and a documented rollback path. Ask how false positives on clinical text are measured: which labeled sample of notes, discharge summaries or portal messages is used, who labels it, what precision and recall are reported per entity type, and whether the partner will rerun that set after every policy change. A partner that cannot show these numbers on your data before go-live is describing intent, not accuracy.

Finally, ask how logs are protected and retained. Logs of blocked and redacted requests can themselves contain PHI, so confirm the encryption, access roles, retention period and deletion process for each log store, and who is allowed to read raw prompts. Ask how the partner handles business associate terms if it touches ePHI during testing or support, and whether it will sign a business associate agreement that flows the Security Rule obligations down to its own subcontractors under 45 CFR 164.314. Treat any claim of HIPAA certification as a red flag, since the HHS FAQ on certification states that no Security Rule standard requires certification. Ask for evidence instead: the test plan, the accuracy report and the access review records.

How DeployOpen scopes and supports healthcare guardrails

DeployOpen plans, implements, hardens, documents and supports self-hosted guardrails inside the health organization's own cloud accounts, data centers or air-gapped networks. Work starts from a written, fixed scope that names the assistants, data flows and decisions the controls must cover.

How we scope

Inventory every assistant, model endpoint and data flow that could carry PHI into prompts, retrieved content or logs
Agree which identifiers, clinical topics and out-of-scope requests the input and output rails must handle
Write the scope, deliverables and timeline in advance, with a quote following from that written scope

How we build

Configure NeMo Guardrails or Guardrails AI with versioned policies, and enforce them at the AI gateway in front of the models
Run guardrail classifier models on premises so patient text is never sent to a hosted moderation service
Test each rail against a red-team set of PHI leakage, clinical advice and injection cases before any release

How we hand over

Deliver configuration, runbooks and a record of test results for each rail, mapped to the controls your assessors ask about
Set up decision logging for blocked and redacted requests, with retention settings agreed with your compliance team
Train your engineers and clinical governance owners on how to change policies and review guard decisions

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after handover
Retest guard behavior when models, policy packs or guardrail library versions change
Confirm with your legal and compliance advisers which evidence your HIPAA and any other obligations require, since DeployOpen does not issue audit reports or certifications