Why LLM guardrails for the public sector need their own boundary
An agency that deploys a citizen-facing or internal assistant has to answer for every output it produces, including the ones it did not expect. That makes the guard layer part of the system of record for the assistant, not an add-on. The guardrails themselves must run inside the agency boundary. If the classifier models, policy files, gateway and log store sit with a hosted moderation service, the agency has sent prompts, and sometimes sensitive terms, outside the boundary that its authorization and records obligations describe. Confirm with your security office which boundary applies to each component, including the model host, the AI gateway, the guard service and the observability store.
Public sector assistants carry three duties that generic guardrail setups miss. First, scope refusals: the assistant should decline requests outside the services it is authorized to answer, and the refusal text should point people to the right channel. Second, sensitive term blocking: agencies maintain lists of classified markers, law enforcement terms and case-related identifiers, and these lists must be versioned and kept off external services. Third, accessibility: the ADA Title II web rule applies WCAG 2.1 Level A and AA to state and local government web content, including content delivered through contracts, with compliance dates that depend on the population an entity serves (28 CFR 35.200). Guard rejections shown to citizens must therefore be readable by assistive technology. The short answer for most teams is NeMo Guardrails or Guardrails AI enforced at a self-hosted AI gateway, with models, prompts and logs stored in-boundary. Each option needs its own check against your policy requirements before you choose.
If the assistant is a high-risk AI system under the EU AI Act, Article 12(1) requires technical support for automatic event logging over the system's lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law sets a longer period. Confirm whether your use case is high risk before you design the log schema. DeployOpen prepares systems, controls and evidence for assessments, such as guard configuration records, blocked-request logs and test results, but it does not issue audit reports, attestations or certifications. Ask your assessor early which evidence they expect and where it must be stored.
Which public sector rules shape the guardrail build
This table maps the federal, criminal justice, defense, accessibility and EU rules most likely to apply to a government assistant to the decisions they force on guard placement, logging, encryption and model hosting. Read each row against your agency's authorities and data classification, because the same rule can point in different directions depending on whether the assistant touches criminal justice information, controlled unclassified information or public web content.
| Requirement | What it asks for | Impact on the guardrail build |
|---|---|---|
| NIST SP 800-53 Rev. 5 baselines | SP 800-53B selects 149 controls and enhancements at low, 287 at moderate and 370 at high. Release 5.2.0 of 27 August 2025 is current. | Confirm the impact level first. Guard decision logs, policy files and model endpoints inherit the baseline of the data they handle, so a moderate system needs the same control evidence for the guard layer as for the app. |
| FISMA and Inspector General assessment | Inspectors General independently assess agency system security and report to OMB under FISMA. | Expect an outside reviewer to test whether guards actually run. Keep guard configuration under change control and retain evidence of each blocked or allowed decision for sampling. |
| FedRAMP scope under M-24-15 | Covers cloud products that process federal information on behalf of an agency. Single-agency systems on cloud infrastructure, not offered as a shared service, are out of scope. | Hosting the guard models and gateway in a cloud account may or may not trigger FedRAMP. Confirm the scope determination with your authorizing official before choosing a cloud region or managed inference service. |
| CJIS Security Policy v6.1 encryption and scanning | For criminal justice information outside physically secure locations, FIPS 140-3 validated modules or AES with 256-bit keys, and vulnerability scanning at least monthly. | Guard decision records, blocked prompts and quarantined text may contain CJI. Encrypt them at rest and in transit with validated modules and include the guard host in monthly scanning. |
| 32 CFR 2002.14(h)(2) for CUI on non-federal systems | Agencies apply NIST SP 800-171 when setting protections for controlled unclassified information on non-federal systems, unless the category's authority prescribes other safeguards. | If a contractor operates the guardrail stack, its classifier models, logs and policy repository fall under these requirements. Keep CUI-bearing prompts and outputs inside the boundary and confirm the marking rules for guard logs. |
| ADA Title II WCAG 2.1 A and AA dates | State and local government web content and mobile apps, including contracted ones, must meet WCAG 2.1 Level A and AA from 26 April 2027 for entities serving 50,000 or more people, and from 26 April 2028 for smaller entities and special districts. | Refusal and block messages are user-facing content. Design them for screen readers and keyboard use, and test them, because a guard that blocks a request with an inaccessible message still creates a compliance defect. |
| NIS2 scope and national security exclusion | NIS2 covers central government entities regardless of size and lists public administration as highly critical. It does not apply to entities working in national security, public security, defense or law enforcement. | Determine whether the assistant falls inside NIS2 before scoping incident logging for guard bypasses. Assistants used for law enforcement or defense are excluded, so their guard controls must be justified under other national rules. |
| Public records and retention | Agency retention schedules and records laws govern how long prompts, blocked requests and guard decisions are kept. | Guard logs can become records. Set retention for blocked-request logs separately from general chat logs and confirm with records officers before choosing storage locations. |
Where the guard models and logs can run in each agency setting
For a federal or state agency, the first question is whether the cloud service that hosts any part of the assistant is in FedRAMP scope. FedRAMP covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency. A system used only for one agency's operations on cloud infrastructure, and not offered as a shared service, falls outside that scope. Prompts, retrieved documents, guard decisions and conversation logs usually contain agency information, so the guard model, policy store and log sink should be treated as in-boundary by default until the agency's authorizing official confirms otherwise. Controls are drawn from the NIST SP 800-53 catalog, and the baseline selected for the system determines how many controls apply.
- Guard models and classifiers: Run self-hosted NeMo Guardrails or Guardrails AI validators inside the agency's authorization boundary, so input and output text is not sent to an external moderation API.
- Decision and prompt logs: Store blocked requests and guard verdicts in agency-controlled storage with retention set by records rules, not by a vendor's default.
- Cloud hosting: Confirm whether any cloud component is a shared service, since that moves it into FedRAMP scope and requires an authorized provider.
- Policy and configuration files: Keep Colang or validator configurations under change control, because a policy edit changes what the agency's assistant allows.
Confirm with the authorizing official which NIST SP 800-53 baseline applies, whether each cloud component is in FedRAMP scope, and where log retention obligations are recorded.
When an assistant may encounter criminal justice information (CJI), the FBI CJIS Security Policy sets minimum requirements that apply to the systems and people that touch it. Version 6.1 took effect on June 26, 2026. Any component that processes CJI, including the guard model, its prompt logs and the gateway in front of it, falls inside the CJIS audit perimeter unless the agency's CJIS system officer approves a different design. Encryption requirements are strict: CJI in transit and at rest outside physically secure locations must use FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys, and vulnerability scanning must occur at least monthly.
- Encryption of guard logs: Verify that prompt and decision logs use approved modules and 256-bit keys, not only the transport layer.
- Guard model hosting: Run classifier models on agency-controlled hardware or in a CJIS-compliant environment, so CJI is never sent to a commercial moderation endpoint.
- Personnel and access: Restrict who can read blocked prompts, since blocked prompts often contain the very CJI the guard was meant to stop.
- Scanning cadence: Include the guard stack in the monthly vulnerability scanning scope and in patching schedules.
- Cloud providers: Confirm the provider's CJIS posture in writing before any component that touches CJI is hosted there.
Confirm with the CJIS system officer which components fall inside the audit perimeter, which cryptographic modules are approved, and whether the Requirements Companion Document changes any design decision.
Contractors that handle controlled unclassified information (CUI) on behalf of federal agencies face requirements that follow the data into their own systems. Under 32 CFR 2002.14(h)(2), agencies use NIST SP 800-171 when setting requirements for CUI on non-federal systems, unless the CUI category's authority prescribes other safeguards. A defense contractor seeking CMMC Level 2 may store or process CUI in a cloud service only if that service is FedRAMP Authorized at the Moderate baseline or higher, or meets equivalent requirements, under 32 CFR 170.17(c)(5). The contractor's own boundary therefore decides where the guard model, its logs and its gateway can run.
- Contractor-owned guard stack: Host guard models and gateways inside the contractor's assessed boundary when the assistant touches CUI.
- Cloud components: Check whether each cloud service that stores or processes CUI is FedRAMP Authorized at Moderate or higher, or meets equivalent requirements.
- Non-cloud providers: Confirm which external providers are assessed within the contractor's own scope, because that determines whether they must be included in the boundary.
- Log handling: Treat guard logs that quote CUI as CUI, with the same access and storage rules.
- Subcontracted models: Identify any model endpoint or validator package sourced from outside the boundary and confirm how it is updated.
Confirm with the contracting officer or the program's security lead which contract clauses apply, and with counsel how CUI in logged prompts should be classified.
NIS2 covers central government entities regardless of size and lists public administration as a sector of high criticality. Member States may extend it to local public administration. NIS2 does not apply to entities working in national security, public security, defense or law enforcement, so the scope question comes first. Where NIS2 applies, the guard stack falls under the security measures the entity must take for its networks and information systems, and incident handling duties apply to the logs. For a public body in scope, guard models and logs can often run in national or agency data centers or a sovereign cloud, but the choice depends on national transposition, on the entity's risk assessment and on any other EU law that governs the data.
- Scope determination: Confirm whether the body is a central government entity, a local authority covered by national law, or an excluded security or law enforcement body.
- Guard models in national infrastructure: Run classifier models in state or agency data centers to avoid transferring citizen text to foreign moderation services.
- Incident evidence: Keep guard decision logs in a form that supports incident reporting timelines set by the national authority.
- Interoperability: Where data is exchanged with other public bodies, note the obligations under Regulation (EU) 2024/903 on interoperable public services.
- Supply chain: Document the origin of model files and validator packages, since NIS2 expects supplier risk to be managed.
Confirm with the national competent authority and the data protection officer whether NIS2 applies, how national law transposed it, and where logs may be stored.
Which open-source platforms fit public sector guardrails and why
Public sector guardrails combine a policy engine, validators, a gateway that enforces them, a model server that stays inside the agency boundary, and a trace store that records decisions. The cards below cover the open-source components that fill those roles and the checks to run before choosing each one.
NeMo Guardrails for policy rails
NeMo Guardrails is Apache-2.0 licensed and configures five rail types: input, dialog, retrieval, execution and output. It is the natural place to express out-of-scope refusals and blocked-term policies for a government assistant. Colang 1.0 is the default language, while Colang 2.x is also supported. Migrating between them with the nemoguardrails convert tool still needs manual review, so budget that effort before any version change. It runs as a Python library or as a server with a Dockerfile, and NVIDIA also offers a Kubernetes microservice option.
Guardrails AI for validators
Guardrails AI is Apache-2.0 licensed and builds guards from validators published on Guardrails Hub, such as regex matching or toxicity detection. Validators can be chained, which suits checks for classified markers, personal identifiers and reading-level rules. It also generates structured output that matches Pydantic models, useful when an agency needs responses in a fixed case-form schema. Confirm each validator's license and provenance before installation, since validator packages are installed separately and may come from different authors.
Flask service for guard calls
Guardrails AI can run as a standalone Flask REST service started with guardrails start and called through the OpenAI SDK. This keeps validation in a separate process that the application and gateway call over the agency network. Ask the implementation partner how the service is authenticated, how its configuration is versioned, and whether guardrails configure is run with anonymous metrics reporting turned off, since the quickstart prompts for that choice.
LiteLLM for gateway enforcement
LiteLLM is an AI gateway that calls more than 100 LLM APIs in OpenAI format and includes guardrails, logging, keys and budgets. It is MIT licensed except for its enterprise directory, so confirm which features your deployment depends on before relying on them. At the gateway, the agency can make every model call pass through the same input and output checks and log blocked requests in one place. Verify that the enterprise-only features you need are covered by the license terms you accept.
Apache APISIX as AI gateway
Apache APISIX is a cloud-native API gateway and AI gateway licensed under Apache-2.0. It suits agencies that already run an API gateway for citizen services and want model traffic governed by the same authentication, rate limiting and access logs. Confirm which AI routing and guard plugins your version provides, and test that they call the guard service before the model is reached. Pair it with a model server such as Ollama or vLLM so that no request leaves the boundary.
Langfuse for decision traces
Langfuse is an LLM observability and evaluation platform licensed under MIT, except for its ee directories. Recording each guard decision, the rail that fired and the blocked input category as a trace gives auditors evidence of how the assistant behaved. Store traces in the agency's own database and set retention to match records policy. Check which Langfuse features sit in the ee directories, because those files carry different license terms.
Ollama or vLLM for in-boundary models
Ollama (MIT, v0.40.5) and vLLM (Apache-2.0, v0.31.0) both serve open-weight models on agency hardware behind an endpoint applications can call. Guard classifiers, such as toxicity or jailbreak models, need to run in the same boundary, so confirm which ones your chosen rails library loads locally and how their model files are imported. Phoenix is a common tracing alternative, but its Elastic License 2.0 prohibits offering it as a hosted or managed service to third parties, so check that rule against any shared service model before choosing it.
How an agency rollout moves from scope definition to monitored operation
The sequence turns a citizen-facing or internal assistant into a guarded service with named policy owners, in-boundary guard components, tested refusals and accessibility checks, a logged evidence pack and a controlled go-live that stays under monitoring. Each phase closes with a written output that the next phase depends on.
Define scope and build the risk register
The agency names the assistant's purpose, user groups, data it may see and the systems it may call, then lists the risks to cover, such as out-of-scope requests, disclosure of classified or sensitive terms, prompt injection through uploaded documents and unsafe or inaccessible output. Each risk gets a named policy owner from the business, security and legal teams. The step closes with an approved scope statement and a risk register that records which controls address which risk and who accepts any residual risk. Where a system may handle criminal justice information or controlled unclassified information, the team confirms the applicable baseline before any design work starts.
Select guard components and import them inside the boundary
Engineers choose the guard approach for each risk, typically NeMo Guardrails rails for dialog control and input and output checks, or Guardrails AI validators for pattern and format checks, and decide which classifier models will run locally. Every package, validator, model file and container image is imported across the boundary through the agency's approved intake path, with checksums recorded and no runtime calls to external moderation services. The output is a bill of materials for the guard stack, with versions pinned and a documented owner for each component. Teams should confirm current license terms and supported Python versions for each component before approval.
Configure policies and version them
Engineers write the policy set in Colang or in validator configuration files, covering the allowed topics, refusal messages, blocked term lists and the handling of retrieved content. Policy text is stored in version control with peer review, and each change links to the risk register entry it addresses. Conversion between Colang versions requires manual review, so the team records any migration as its own change. The output is a tagged policy release that can be deployed to test and production with the same content, plus a change log that policy owners can read without reading code.
Run the accessibility and fairness test pass
Testers run representative prompts in every supported language and channel, then check that refusal messages, error states and any generated content meet the accessibility standard that applies to the agency. For assistants offered on public websites or mobile apps, the ADA Title II rule points to WCAG 2.1 Level A and AA, with the compliance date depending on the size of the population served. Fairness testers compare refusal rates and answer quality across user groups and dialects and record differences that need a policy decision. The output is a signed test report listing failures, the fixes made and the findings the policy owners accepted.
Red-team refusals and injection paths
A separate testing team, independent of the policy authors, attempts to bypass the guards with jailbreaks, instructions hidden in uploaded or indexed documents, requests for classified or sensitive terms, and out-of-scope questions designed to pull the assistant into giving advice it should not give. Each attempt is logged with the expected and actual outcome. Results feed back into the policy configuration, and every fixed failure is retested. The output is a red-team report with pass and fail results per risk, the open findings and a retest record showing that fixes held. Testing repeats after any policy or model change.
Assemble the logging and evidence pack
The team configures logging of blocked requests, rail decisions and policy versions, and confirms where those records are stored, who can read them and how long they are kept under the agency's retention rules. Where the assistant is a high-risk system under the EU AI Act, the team checks the record-keeping duties in Article 12 and the six-month minimum retention in Article 19(1). The evidence pack collects the risk register, component bill of materials, policy releases, test reports and log configuration in one controlled location. It is prepared for assessors but does not replace an independent assessment or authorization decision.
Go live in controlled stages with monitoring
The assistant is released to a limited user group first, with guard decisions and refusal rates visible to the operations team and a defined rollback path to the previous policy release. Operators review blocked requests, false positives and user complaints against the risk register, and the policy owners approve each widening of access. Monitoring covers guard health, latency and changes in refusal patterns. The step closes when the agency's service owner signs the go-live record, and the monitoring routine, review cadence and change process are documented as the operating procedure.
How guard integration points connect to the agency stack
How do guardrails integrate with agency identity and access management?
Guardrails themselves rarely authenticate users, so the identity decision sits in the application or gateway in front of them, and the guard policy should be selected based on who is asking. Guard configurations can be scoped by role, so a public citizen assistant and a staff assistant that can see internal files run different input and output rails. The agency must confirm which identity provider issues sessions, how group or role claims reach the gateway and application, and whether the guard service accepts only calls carrying trusted identity context. Test that a user without a role cannot invoke a stricter or looser policy by changing a request header.
What role does the AI gateway play in enforcing guardrails?
The AI gateway is the enforcement point that every model call passes through, so guard checks placed there cannot be skipped by an application team. Open-source options such as LiteLLM and Apache APISIX can route requests, hold keys and budgets, and call guard checks before and after inference. The agency must confirm which gateway is the single entry to model providers, that direct calls to model endpoints are blocked at the network layer, and how a guard failure is handled: fail closed for citizen-facing use, or fail open only with a documented decision. Check that blocked requests are recorded with enough detail to investigate.
How should retrieval rails work with agency document sets?
Retrieval rails inspect document chunks before they reach the model, which matters when agency manuals, policies or case files contain instructions that could hijack the assistant. The integration needs the document access controls to be applied at retrieval time, so a chunk a user cannot open is never passed to the guard or the model. The agency must confirm which document classifications exist, which index holds them, and whether retrieved text is labeled as untrusted content in the prompt. Test with sample documents that contain embedded instructions and confirm the rail flags or strips them. Coordinate with the team that owns the retrieval platform, since guard logic alone does not fix weak permissions.
How do guard logs reach the agency SIEM?
Each guard decision should produce a structured event with a timestamp, policy version, rail name, decision, a request identifier and the calling identity, while avoiding storage of raw sensitive content in the log. The logs can be shipped to the agency SIEM with the existing log forwarder, provided the agency confirms the event schema, retention period and who may read prompts and blocked outputs. For high-risk AI systems, the EU AI Act sets automatic event logging duties in Article 12 and a six-month minimum retention under Article 19 unless other law requires longer, so retention must be set with legal input. Ask the SIEM team to build detection rules for repeated blocks from one account.
How is guardrail policy versioned and changed safely?
Policies should live in version control as configuration files and Colang or validator definitions, and each change should go through review and a staged rollout before reaching production. Every guard decision should record the policy version so auditors can reconstruct which rules applied on a given date. The agency must confirm who approves policy changes, whether a change needs a security or accessibility sign-off, how a rollback is performed, and whether the test set used for regression is kept with the policy. NeMo Guardrails configurations and Guardrails AI validators both support file-based definitions that fit a standard change process.
How are guard models and validators imported into an air-gapped network?
Classifier models, validator packages and Python dependencies must be fetched outside the network, checked, and carried across by an approved transfer process, because the guard runtime cannot download them at startup. Guardrails AI validators come from Guardrails Hub and are installed as separate packages, so the agency should mirror the package index and record the exact versions. The agency must confirm its media transfer rules, how hashes and signatures are verified, who approves each import, and how model updates are scheduled. Keep a bill of materials for every imported artifact so security teams can trace what runs in production and when it was introduced.
Can the agency build these integration points with its own staff?
Yes, many agencies can, provided they have engineers who know the identity provider, the gateway, the SIEM and the document platform, and they accept ongoing ownership of policy changes. Open-source guard libraries are available for self-hosting, so the cost is mainly staff time, testing and operations rather than license fees. DeployOpen plans, implements, integrates, hardens and documents self-hosted open-source platforms inside the client's own infrastructure, and most engagements are fixed-scope with the scope agreed in writing before work starts. A quote follows from that written scope. Ask any partner which integrations they will test end to end and which remain the agency's responsibility.
What auditors and oversight bodies ask to see from guard logs
This table maps the evidence that assessors, inspectors and oversight bodies typically request for a public sector guardrail deployment to the component that produces each item. Read it as a checklist for designing logs and records before the first assessment, not as a list of certifications. DeployOpen prepares this evidence, while independent auditors and assessors give the opinion on whether it meets the applicable standard.
| Evidence | What it shows | Component that produces it |
|---|---|---|
| Versioned policy files | Which rails, topic restrictions and blocked-term lists were active on a given date, and who approved each change | Git repository holding config.yml, Colang files and validator settings, with signed release tags |
| Guard decision logs | Each input and output block, pass or redaction, with timestamp, policy version and model identifier | NeMo Guardrails server or Guardrails AI service, forwarded to central log storage through the AI gateway |
| Log retention records | Logs kept for at least six months where high-risk rules apply under Article 19(1) of the EU AI Act, with other retention set by agency records law | Log store configuration and retention policy in the agency's own infrastructure |
| Red-team test results | Which jailbreak, injection and out-of-scope prompts were tried, and which guards stopped them | Test harness and reports from DeployOpen security testing, stored with the policy version tested |
| Accessibility test records | Conformance checks of the assistant interface against WCAG 2.1 Level A and AA, as the ADA Title II rule requires | Automated scans and manual assistive technology testing run against the released interface build |
| Encryption evidence for CJI or CUI stores | FIPS 140-3 validated modules or AES with 256-bit keys for CJI at rest and in transit, as CJIS Security Policy v6.1 sets out | Key management service, disk and database encryption settings, and module certificates from the vendor |
| Monthly vulnerability scan records | Scans of the guardrail, gateway and log hosts at least monthly, with findings and closure dates | Scanner output and ticket records from the agency's vulnerability management process |
| Fairness and out-of-scope test records | Results of tests showing refusals stay consistent across demographic phrasing and that out-of-scope answers are declined | Evaluation suite run against each policy version, with results stored beside the decision logs |
What to ask an implementation partner before a public sector guardrail project
Start by asking for the partner's actual record on federal and criminal justice environments, and ask for it in writing. Ask whether they have operated systems inside a FedRAMP authorization boundary, which FedRAMP impact level those systems carried, and whether they have handled CJI under the FBI CJIS Security Policy. A credible answer names the scope of the work, the controls the partner owned, and the customer's own authorization documents, and it stays within what the partner can show. Be wary of any partner who states an authorization status for their own work. FedRAMP scope turns on whether a cloud service is offered as a shared service, so ask the partner to explain how they would determine whether your agency's guardrail deployment falls inside it. Ask who at the agency or its authorizing body confirms that conclusion.
Next, ask how every model, classifier and validator is placed inside the boundary. Request a list of every artifact the guardrail stack needs at runtime: the NeMo Guardrails or Guardrails AI package, any Guardrails Hub validators, classifier model weights, and the gateway that enforces the rails. For each one, the partner should state where it is downloaded, how it is verified, and how it is imported across a gap if the network is isolated. Ask whether any input or output check calls a hosted moderation or classification API, even as a fallback. Treat a yes, or a vague answer, as a disqualifying red flag for a public sector deployment. Also ask how the partner keeps validator packages current without online updates, and who approves each change to the boundary.
Ask how refusal policies get written and who signs them off. A capable partner will refuse to write the policy alone. They should run working sessions with the policy owners, such as the service owner, legal counsel, records officer and accessibility lead, and turn their decisions into Colang or Guardrails AI configuration under version control. Ask how the partner will test false positives on real agency language, including forms, benefit letters, acronyms and multilingual text, and how many reviewers will label the results. Ask for the migration plan for any Colang 1.0 configuration. NVIDIA's migration guide says converted configurations still need manual review, so ask who performs that review, how behavior is compared before and after conversion, and whether both versions run in parallel before cutover. Finally, ask what evidence the partner hands over at the end: configuration history, test results, blocked-request logs, a list of open risks and a runbook for the agency's own staff. DeployOpen prepares this evidence for assessors, but assessors and authorizing officials make the determinations.
What agencies ask about the cost, effort and ownership of guardrails
What drives the cost of a public sector LLM guardrail project?
The largest drivers are the number of assistants and channels covered, the number of rails and validators in scope, the classifier models that must run inside the agency boundary, and the amount of policy writing and tuning against the agency's own content. Costs fall into two groups. One-time costs include design, policy authoring, integration with the AI gateway, testing and documentation. Recurring costs include compute for guard models, logging storage, red-team retesting after model or policy changes, and staff time to review blocked requests. Ask any partner for a quote that separates these lines and states which environments are priced. Open-source licenses for NeMo Guardrails (Apache 2.0) and Guardrails AI (Apache-2.0) carry no license fee, so the quote is for labor, infrastructure and testing.
What must the agency provide before guardrail work can start?
The agency must provide a written list of the assistant's permitted topics, its refusal rules, the terms it treats as sensitive or classified, and the systems the assistant can reach. It also needs access to a sample of real or representative prompts, a named business owner who approves policy changes, and infrastructure in its own boundary: compute for guard models, a gateway, and log storage. Security and records staff should confirm how long logs of blocked requests must be retained. Without these inputs, the first phase is spent on discovery rather than configuration, so ask a partner to list each input as a dependency in the written scope.
Can our own staff run and maintain guardrails after handover?
Yes, if the team has Python skills and someone who can own policy decisions. NeMo Guardrails is configured with a config.yml file, optional Python actions and Colang files, and Guardrails AI builds guards from validators that are installed separately. Day-to-day tasks such as adding a refused topic or a sensitive term are routine. Harder tasks include migrating Colang 1.0 configurations to Colang 2.x, which NVIDIA says still needs manual review after conversion, and writing custom validators. Agencies usually keep policy ownership in-house and buy support for engineering changes and testing. Confirm the skills gap during scoping.
How often should guardrail policies and detection models be reviewed?
Review on a fixed schedule and after every trigger event. Triggers include a new assistant or data source, a model upgrade, a new version of NeMo Guardrails or Guardrails AI, a change in agency policy, and a spike in blocked or missed requests. NeMo Guardrails v0.24.1 was released on September 16, 2026, and Guardrails AI v0.11.0 on August 14, 2026, so both libraries change often enough that version pinning and a regression test suite matter. Ask the partner for the review cadence, who signs off changes, and how a rollback works. Recurring retesting is a cost line in its own right.
Does a guardrail project fall within FedRAMP scope?
It depends on where the guardrail runs and what it touches. FedRAMP covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency. Under OMB M-24-15, systems used only for a single agency's operations on cloud infrastructure and not offered as a shared service are out of scope. A guardrail that runs entirely on agency-controlled servers and never sends prompts to a third-party cloud is therefore not a FedRAMP cloud service in that sense. A hosted moderation API or a vendor-run gateway that processes federal information may be in scope. Confirm the classification with the agency's authorizing official and the FedRAMP program documentation before procurement.
Does a guardrail project fall within FBI CJIS Security Policy scope?
It falls in scope whenever the assistant processes criminal justice information (CJI), because the CJIS Security Policy sets minimum requirements for CJI wherever it is stored, transmitted or processed. Version 6.1 took effect on June 26, 2026. It requires FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys for CJI in transit and at rest outside physically secure locations, and vulnerability scanning at least monthly. Guardrail logs that contain CJI inherit those requirements. Ask the agency's CJIS Systems Officer which controls apply and whether the guard models and log stores meet them before design begins.
Which NIST control baseline applies to a guardrail system?
The baseline depends on the system's impact level and the agency's authorization path, not on the guardrail library itself. NIST SP 800-53B defines low, moderate and high baselines that select 149, 287 and 370 controls and enhancements respectively. Agencies usually map guardrail components to access control, audit and accountability, system and communications protection, and vulnerability scanning controls. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent assessors issue the findings. Ask the agency's security office which baseline the system inherits and whether a system security plan already covers the AI gateway.
How DeployOpen scopes and supports guardrails for public bodies
DeployOpen plans, implements, hardens and documents self-hosted guardrails inside the agency's own cloud accounts, data centers or air-gapped networks. For a public body, the first decision is where the guard models and policy files run and who can change them. DeployOpen scopes that boundary in writing before work starts, including which rails (input, dialog, retrieval, execution and output) run in NeMo Guardrails, which validators run in Guardrails AI, and where the enforcement point sits at the AI gateway. The written scope also names the policy owners, the change approval path for out-of-scope refusals and blocked-term lists, and the logging destination for blocked requests, so the agency's security team can review them without relying on a vendor's hosted service.
Compliance evidence is a shared deliverable. DeployOpen prepares the configuration baselines, test results, policy version records and guard decision logs that an agency's assessor or inspector general may request, mapped to the control families the agency already uses. DeployOpen does not issue audit reports, attestations or certifications; independent assessors and the authorizing bodies make those determinations. Before scoping, ask the agency's authorizing official or compliance office which catalog and baseline apply, since the assessment obligations differ by program. Confirm whether any language model or guard service is hosted by a cloud provider, because that determines whether a FedRAMP authorization question arises.
Security testing covers the guardrails themselves. DeployOpen runs application and API testing, AI red teaming against prompt injection and sensitive term bypass, and retesting after each policy change. Ongoing work such as continuous testing, dependency upgrades and policy tuning is offered as a retainer, separate from the fixed-scope implementation. Ask any partner to state which tests are one-time and which recur, who holds the test evidence, and how a change to a blocked-term list is retested before it reaches citizens. DeployOpen will confirm each of these points in the written agreement, and it does not claim government clients or certifications.
Sources and further reading
- NIST SP 800-53 Rev. 5 (CSRC) ↗
- NIST SP 800-53B, Control Baselines (CSRC) ↗
- California DOJ Information Security bulletin 2026-ISRS-001: FBI CJIS Security Policy version 6.1 ↗
- eCFR: 28 CFR Part 35 Subpart H, Web and Mobile Accessibility ↗
- NVIDIA-NeMo/Guardrails (GitHub) ↗
- AI Act Article 12 Record-keeping ↗
