Why RAG assistants need guardrails on retrieved text
An internal assistant that answers from indexed documents and web pages needs guardrails on the retrieved text itself, because anyone who can edit a page or file the index reads can place instructions in front of the model. Indirect prompt injection happens when those instructions arrive inside a chunk the assistant retrieves, not in the user's question. Input rails on the user prompt alone do not see that text, so the attack passes through them untouched. The practical answer is to place retrieval rails on each chunk before it reaches the model, output rails on the answer, and enforcement at an AI gateway, all running on the RAG engine the organization already operates.
In a direct jailbreak, the attacker types the malicious instruction into the chat box and the input rails see it. In indirect injection, the attacker plants the instruction in a wiki page, a shared drive file, a supplier PDF or a web page the crawler fetches, and the assistant reads it later on behalf of a different user who has done nothing unusual. The payload can ask the model to reveal other retrieved content, to cite a source that does not support the answer, or to call a tool with attacker-chosen arguments. Each retrieved chunk should therefore be treated as untrusted data, with the separation between instructions and content enforced by the application rather than trusted to the model.
NeMo Guardrails provides retrieval rails for RAG chunks and output rails, and it can be served as a guardrails server behind an AI gateway such as LiteLLM or Apache APISIX. Before you commit, confirm with your RAG engine vendor and your gateway team which retrieval hook each engine exposes, how chunk metadata such as source and owner reaches the rail, and where the detection models will run. Ask who owns the source inventory that decides which spaces and sites the index may read.
Requirements that shape retrieval rails for RAG
This table maps each requirement that affects a retrieval-augmented assistant to its source and to what it means for the build. Read it before choosing rails, because logging scope and data location decide where chunk checks run and what must be kept. Items marked for counsel or auditor confirmation depend on your sector and legal position.
| Requirement | What it means for the build | What to confirm with counsel or the auditor |
|---|---|---|
| EU AI Act Article 12(1): automatic event logs over the lifetime of a high-risk system | Log retrieval events: which documents and chunks were returned, the query, and the model version, so a later review can reconstruct an answer | Whether the assistant is a high-risk system at all, and which events the assessor expects to see in the logs |
| EU AI Act Article 19(1): keep logs at least six months unless other law requires longer | Set retention on retrieval and rail-decision logs to at least six months, and store them in the organization's own infrastructure with access controls | Whether sector rules, such as financial or health record keeping, require a longer period and who owns the retention decision |
| Logging blocked and modified decisions | Record each retrieval rail decision (allowed, stripped, blocked) with the chunk identifier and rule that fired, without copying sensitive text into general logs | How much of the blocked content may be stored, and whether logs containing indexed personal data fall under data protection rules |
| Location of source documents | Keep the document store and ingestion pipeline inside approved networks, so the scanning and indexing steps do not send content to external services | Which jurisdictions and data classes are permitted, and whether any third-party embedding or scanning service is acceptable |
| Location of vectors and embeddings | Host the vector database in the same boundary as the source documents, since embeddings can still carry information from the source text | Whether embeddings are treated as personal or confidential data under internal policy and applicable law |
| Access control on retrieved content | Apply document permissions at retrieval time so the assistant cannot surface chunks a user could not open directly | Whether the auditor expects access checks to be tested with sample users and documented as evidence |
| Change control on indexed sources | Record when documents are added, changed or removed from the index, so a citation can be tied to a specific version | How long source versions must be kept and whether the retention period matches the log retention period |
| Evidence of testing for indirect prompt injection | Keep test sets of poisoned documents and the results of rail tests, versioned with each rail configuration change | Which testing standard or internal control the assessor will accept, and how often testing must repeat |
Which platforms fit a self-hosted RAG guardrail stack
A RAG guardrail stack has three layers: the rail engine that inspects retrieved chunks and outputs, the gateway that enforces those rails on every model call, and the RAG engine and vector store that hold the indexed documents. The platforms below cover each layer, and each one carries a licensing or operational question to settle before the design is fixed.
NeMo Guardrails as the rail engine
Retrieval rails on RAG chunks are the core control here: they inspect retrieved text before it reaches the model, so instructions hidden in an indexed page can be flagged or dropped. The library is Apache 2.0 and also provides input, dialog, execution and output rails, which covers tool calls if the assistant has them. Confirm the Python version your platform team supports, since the library supports Python 3.10 to 3.13, and test chunk-level rails against documents from your own corpus, because injected text in real content is often subtle.
LiteLLM for gateway enforcement
A gateway is where the rails become mandatory. LiteLLM sits between the RAG application and model providers, calls more than 100 LLM APIs in OpenAI format, and includes guardrails and logging, so every request passes the same checks regardless of which application sent it. The core is MIT licensed, but the enterprise directory uses different terms. Confirm which features you need sit outside that directory, and confirm how guardrail decisions are written to logs you control.
Apache APISIX as an enterprise gateway
If your organization already runs an API gateway for other services, Apache APISIX can act as the AI gateway in front of model endpoints, with a plugin model for authentication, rate limits and request inspection. APISIX is Apache-2.0 licensed. Confirm that the AI-specific plugins you need, such as prompt inspection and token accounting, are available in the release you deploy, and confirm that rail decisions from NeMo Guardrails can be enforced at this layer rather than only logged.
RAGFlow for document-grounded retrieval
RAGFlow is an Apache-2.0 retrieval-augmented generation engine that handles document parsing, chunking and retrieval. Because it controls how chunks are built, it determines what text reaches the rails. Confirm that the chunk boundaries and metadata it produces are visible to your retrieval rails, that citations point to the original source, and that you can exclude or tag content from untrusted sources before indexing. Check also how it authenticates users and how it stores parsed text.
AnythingLLM and Dify as assistant builders
AnythingLLM is MIT licensed and suits single-team document chat, while Dify provides visual RAG pipelines and agents. Dify's license is a modified Apache 2.0 that forbids operating a multi-tenant environment from the source code without written authorization, where one tenant is one workspace. If you plan to run several business units or customers on one installation, that requires a license conversation before you build. In both tools, confirm where retrieval output can be intercepted so rails run before generation.
Qdrant as the self-hosted vector store
Qdrant is an Apache-2.0 vector search engine that keeps embeddings of indexed documents inside your infrastructure. Its default deployment listens on all network interfaces with no authentication, so access controls must be configured before any document is indexed. Qdrant supports API keys, TLS, per-collection JWT permissions and JSON audit logging of authenticated operations. Confirm that collections holding untrusted sources are separated from trusted ones and that write access to the index is limited to the ingestion pipeline.
How instructions hidden in indexed pages reach the model
The attack starts before any user types a question. An attacker places instructions inside a web page, a shared document, a wiki article or an uploaded file that the organization's ingestion pipeline later indexes. The text might be visible to readers, hidden in markup or white-on-white formatting, or embedded in metadata. When a user asks a related question, the retrieval step selects the chunk that contains the hidden text because it is semantically close to the query, and the chunk is inserted into the prompt context alongside the system instructions, the conversation history and the user's question. The model receives one stream of tokens, so it has no reliable built-in way to know which sentences were written by the application owner and which were copied from a page nobody vetted.
The trust boundary sits between retrieved content and system instructions. System prompts, tool definitions and policy text are trusted because the organization wrote and approved them. Retrieved chunks are untrusted data, even when they come from an internal repository, because the ingestion path may have accepted content from external sources, contractors or compromised accounts. Before designing controls, the team should draw this boundary on paper, list every source feeding the index and record who can write to each one. Retrieval rails, such as the retrieval rails in NeMo Guardrails, act on chunks before they are placed in the prompt, which is the only point where the application can inspect and filter this content before the model acts on it. Confirm with your vendor and your own tests exactly where a given product applies its checks in the request flow, because the answer determines what the rail can see.
How to roll out retrieval rails in phases
The sequence moves from knowing which documents and pages feed the assistant, to limiting what retrieved text can do, to proving the controls hold under adversarial tests before monitoring goes live. Each phase closes with a named artifact that the next phase depends on.
Inventory document and web sources
The platform team, with the content owners, lists every source the assistant can retrieve from: file shares, wikis, ticketing exports, public web pages and any connector that pulls external content. For each source, record who edits it, who can publish to it, and whether outside parties can influence its text. Done when the source inventory names an accountable editor for every source and flags the sources that accept input from customers, vendors or the public as untrusted.
Define allowed sources and chunk policies
Security and the content owners agree which sources may feed retrieval, which are read-only reference material, and which are excluded. They also set chunking rules, such as maximum chunk size, metadata that must travel with each chunk (source, owner, last modified), and rules for stripping markup, hidden text and embedded instructions before indexing. Done when the policy is written as a versioned document and the ingestion settings reflect it.
Build retrieval rails and content separation
Engineers add retrieval rails in NeMo Guardrails or an equivalent layer so that each retrieved chunk is checked before it reaches the model, and wrap retrieved text in a clearly delimited data block that the system prompt labels as material to quote, never as instructions. Flagged chunks are dropped or quarantined and the event is logged. Done when a test corpus containing planted instructions shows those chunks are removed or labeled, and the policy version and rail configuration are committed together.
Add citation checks and action limits
The output rails verify that every citation points to a chunk that was actually retrieved in that session, and that quoted claims appear in the cited text. Tool and action permissions are reduced for any request that involved untrusted content, so the assistant cannot send messages, call APIs or change records based on text it read in a document. Done when citation checks fail on fabricated references and an action-limit test shows that a planted instruction cannot trigger a tool call.
Run indirect prompt injection test sets
The security team builds a test set of documents and pages that carry hidden instructions, such as requests to reveal other chunks, to change the answer, or to call a tool, and runs it against each policy version. They record which attempts succeeded, which were caught by which rail, and any false positives on legitimate content. Done when a dated test report lists results per source type and the open findings have an owner and a fix or accepted risk record.
Enable logging and monitoring
Operations turns on structured logs for every retrieval decision: which chunks were retrieved, which were blocked and why, which citations failed, and which actions were withheld, linked to a request ID and policy version. Dashboards and alerts track spikes in blocked chunks from a single source, which often signals a compromised or newly edited page. Done when logs are stored in the organization's own infrastructure with the retention period set by legal and compliance, and an alert has been tested end to end.
Deployment patterns for air-gapped and connected environments
In the organization's own cloud accounts, the retrieval layer and the guard layer both run in private subnets that the team controls, while the model endpoint may be a self-hosted service or a provider reached over a private link. The main decisions are how the guard service is packaged and how storage is provisioned, because the vector store has hard requirements that managed object storage cannot meet.
- Vector store storage: Qdrant needs block-level storage with a POSIX-compatible file system and does not run on NFS or S3. Choose SSD or NVMe volumes and confirm the volume type with the vendor before sizing.
- Network exposure: A self-deployed Qdrant instance listens on all network interfaces and has no authentication by default. Bind it to a private interface, enable an admin and a read-only API key, and terminate TLS for REST and gRPC.
- Guard packaging: Run NeMo Guardrails as a server container from the provided Dockerfile behind the AI gateway, or call the Python library inside the application service if it already runs in the same boundary.
- Secrets: Keep API keys and model credentials in the cloud provider's secret store and rotate them through the normal pipeline.
- Logging: Send guard decisions to a log store in the same account and retention policy used for other AI system records.
Before go-live, confirm that the vector store, guard service and gateway accept only traffic from approved sources and that every rail decision is written to the log store.
In an on-premises data center, the team owns the hardware, the hypervisor or Kubernetes layer and the storage array, so the deployment choices are mostly about fitting the vector store to existing block storage and choosing how the guard service is operated. Qdrant's distributed mode is the main design choice when the index must survive a node failure.
- Single node or cluster: A single Qdrant node is simpler to run but has no quorum. Qdrant's distributed docs recommend at least three nodes for production so quorum survives a node failure, and a replication factor of 2 doubles storage.
- Storage fit: Confirm that the storage array presents block devices with a POSIX-compatible file system to the Qdrant nodes. Shared NFS mounts are not supported for the index.
- Kubernetes path: The community Helm chart is one production path, but it lacks enterprise features such as zero-downtime upgrades and automatic shard rebalancing. Ask the vendor which path fits your upgrade model.
- Guard service: Deploy the NeMo Guardrails server behind the internal gateway and pin the container image version so rail behavior does not change during a routine patch cycle.
- Network segmentation: Place the vector store and guard service in a zone that only the RAG application and the gateway can reach.
Confirm the storage class, the replication factor and the upgrade procedure with the platform team before choosing a topology.
In an air-gapped network, nothing can fetch models, validator packages, container images or vulnerability data from the internet at runtime. Every artifact must be built or downloaded in a connected staging zone, verified, and moved across a controlled transfer process. Guard models that normally call hosted moderation APIs must run from local files, and that changes both the build pipeline and the update cadence.
- Artifact inventory: List every item the guard stack needs: the NeMo Guardrails package, Python dependencies, the Qdrant container image, embedding and classifier model files, and the configuration folder with Colang files and Python actions.
- Transfer and verification: Record checksums for each artifact at the staging side, scan them there, and verify the checksums again after import. Keep a transfer log that the security team can review.
- Offline model files: Confirm with the vendor and model provider that each classifier and embedding model can be loaded without network calls, and that its license permits internal use.
- Update cadence: Plan batched imports for new releases and patches, since detection models and rail configurations will not update on their own.
- Offline validation: Test the full rail path, including blocked and allowed requests, on the isolated network before cutover.
Before approving an air-gapped design, ask the implementation partner to show how each artifact enters the network, who approves it and how a rollback would be performed.
Where NeMo Guardrails runs determines how many services the application depends on and how retrieval rails see the chunks. The library approach keeps the guard close to the code, while the server approach gives a single enforcement point that several assistants can share. Either choice must still place retrieval rails between the vector store and the prompt that the model receives.
- Library mode: Import the Python package into the RAG application when one team owns one assistant. Retrieval rails run in the same process, which reduces network hops but spreads configuration across application releases.
- Server mode: Run the guardrails server with a configuration folder loaded at startup. Several assistants can call one HTTP API, and configuration changes can be reviewed as a separate release.
- Framework opt-in: LangChain integration is opt-in and enabled with the NEMOGUARDRAILS_LLM_FRAMEWORK=langchain environment variable. Set it only where the application uses that framework.
- Gateway placement: Enforce the server call at the AI gateway, so applications that bypass the library still pass through the same rails.
- Latency: Measure the added time for retrieval rails on representative queries, since each chunk check adds work to the request path.
Confirm which mode your team can version and test, and document the choice with the rail configuration it depends on.
Evidence auditors ask for from RAG guardrails
This table lists the records an assessor is likely to request for a retrieval-augmented assistant, the component that produces each record and what to confirm in your own environment. Read each row as a question to answer before go-live: if the component cannot produce the record, the control is hard to evidence. DeployOpen prepares these records and controls for assessment but does not issue audit reports or certifications.
| Evidence item | Produced by | What to confirm or keep |
|---|---|---|
| Blocked and allowed retrieval decisions | NeMo Guardrails retrieval rails on chunks, logged by the guardrails server | Decision log per query showing which chunks passed, were dropped or were rewritten, with a stable request ID |
| Source provenance per chunk | Ingestion pipeline and vector store metadata (for example, Qdrant payload fields) | Document ID, source system, owner, ingestion timestamp and content hash stored with each chunk |
| Citation check results | Output rail that compares generated citations with the retrieved chunks | Record of each citation tested, the match result and the chunk it was checked against |
| Policy version history | Version control for config.yml, Colang files and Python actions | Git history or signed releases showing which policy version answered each request, with approval records |
| Injection test results | Scheduled red-team and regression suite run against the RAG pipeline | Test case set, run dates, pass and fail counts, and the model and policy versions tested |
| Vector store access logs | Qdrant JSON audit logging of authenticated API operations (since v1.17.0) | Confirm that logs are enabled, shipped off the node and retained for the period your policy requires |
| Retention of assistant logs | Logging stack in the organization's own storage (for example, Langfuse or Phoenix traces) | Ask the assessor which retention period applies; the EU AI Act Article 19(1) sets a minimum of six months for high-risk systems unless other law applies |
| Document access controls | Source system permissions synced to retrieval filters | Evidence that a user only receives chunks from documents they could open in the source system |
What do RAG guardrails cost and how long do they take?
What makes the cost of RAG guardrails go up or down?
Cost follows the scope you agree in writing, and the largest drivers are the number of document sources, the number of retrieval paths that feed the model, and how many guard checks run on each request. Classifier models need compute on your own servers or cloud accounts, so the size of the model and the request volume set the hardware bill. Engineering effort rises with the number of Colang or validator rules you need and with how many tool actions the assistant can trigger. Testing against your own indirect injection cases adds effort in proportion to the sources and workflows in scope. Ask any quote to break these items out separately so you can see which ones your choices change.
Which parts of RAG guardrails are one-time and which recur?
One-time work includes the document source inventory, the initial retrieval rails, the separation of untrusted content from instructions in the prompt design, the first round of citation checks and the baseline test suite. Recurring work includes running classifier models and the gateway that enforces them, reviewing blocked chunks, tuning rules when false positives appear, updating rails when new document sources are added, and repeating injection tests after each model or library upgrade. The NeMo Guardrails library is released under Apache 2.0, so the software itself carries no license fee, but version upgrades still need regression testing. Ask the quote to state which items are fixed and which are retainer work.
How long does it take to implement RAG guardrails?
Duration depends on the written scope, so no general figure applies. The main variables are how many document sources and retrieval paths are in scope, whether source owners can confirm what each source contains, how many workflows can trigger tool actions, and whether the team needs to build its own test cases. The NeMo Guardrails project released v0.24.1 on 16 September 2026, so the version you select at the start of the work will affect the upgrade path. Your partner should propose phases with acceptance criteria, and the schedule should be agreed in writing before work starts.
Can our own engineering team run RAG guardrails without outside help?
A capable platform team can run guardrails once it has a working configuration, a test suite and a clear owner for each rail. The NeMo Guardrails library runs through a Python API or a guardrails server with an HTTP API, and a Dockerfile is provided to build a server image, so the runtime is familiar to most engineers. The harder skills are writing rails for retrieved text, designing injection test cases and interpreting blocked-chunk logs. Ask your partner for a handover that covers configuration, rail design decisions, test maintenance and the upgrade procedure, and confirm that your staff have the Python and container skills the chosen stack requires.
What does DeployOpen need from our side to start?
DeployOpen needs named owners for each document source, such as the wiki, ticketing system, shared drives and any public web pages the assistant retrieves. Source owners should confirm which content is authoritative, who can edit it and which pages must never be treated as instructions. The engagement also needs representative test data, including sample documents, example user questions and a list of actions the assistant may trigger. Finally, it needs access to the staging environment, the vector store, the AI gateway logs and the identity provider, all within your own infrastructure. The written scope should list each of these inputs and the person responsible for providing it.
Does the cost of classifier compute depend on the platform we pick?
Yes, the compute needed for classifier models depends on the models you choose, the request volume and the latency your users accept. NeMo Guardrails includes self-check moderation, jailbreak and injection detection, and topic safety components, and each adds a model call or classifier step to the request path. The retrieval side also depends on your vector store. Qdrant is licensed under Apache-2.0 and is free to self-host, but production clusters need block-level storage and the distributed docs recommend at least three nodes for quorum, which adds infrastructure cost. Ask for a sizing estimate based on your own request profile rather than a general figure.
Should we pay for a hosted guardrail API or run open source ourselves?
Running open source inside your own infrastructure means you pay for compute, storage, engineering time and ongoing testing, while a hosted guardrail API charges per use and sends content to a third party. Your decision should weigh where your documents and prompts may be processed, how much tuning your content needs and whether your team can maintain the rails. The LLM guardrails cost page in this series covers the comparison in more depth. Ask every vendor, including DeployOpen, to price the same scope and the same test volume so the comparison is fair.
Questions to ask an implementation partner before signing
Start with the test plan, because an indirect prompt injection control is only as good as the documents used to prove it. Ask the partner how it plants hostile instructions inside test documents and web pages that the assistant will actually retrieve, such as instructions hidden in a PDF footer, an HTML comment or a table cell, and then checks whether the retrieval rails on chunks, the separation of retrieved text from system instructions and the execution rails on tool calls stop each attempt. Ask for the test cases in writing, the pass criteria, and whether the same suite runs again after every change to the NeMo Guardrails configuration or the retrieval index. A partner who cannot show a test plan for retrieved content has not tested the risk that matters most for a RAG assistant.
Next, ask how false positives are measured on your content, not on the vendor's samples. A retrieval rail that blocks legitimate policy text because it contains words like "ignore" or "instructions" will quietly degrade answers and push staff toward unmonitored tools. Ask who assembles the labeled set of clean chunks from your repositories, who approves the threshold for each rail, and how the partner reports blocked chunks by source collection. Ask where detection models run and confirm that the classifier, the NeMo Guardrails server and the vector store, Qdrant for example, sit inside your boundary, with no hosted moderation call made during retrieval. Finally, settle governance before build. Ask how policy and rail changes are versioned in source control, who signs off a change, and how a rollback is executed. Ask how citations are verified, meaning whether the output rail confirms that each cited source ID exists in the retrieved set and that the quoted span appears in that chunk. Ask who owns the logs of blocked and allowed retrievals, how long they are kept and how they are exported to your own storage, since the EU AI Act sets log retention duties for high-risk systems that the partner should map to your obligations.
