What NIST says about prompt injection and who it harms
NIST AI 100-2e2025 defines direct prompt injection as the case where a user's in-context instructions are appended to higher-trust instructions, such as the system prompt. It defines *indirect prompt injection* as instructions planted in data the system retrieves, such as a web page or a retrieval-augmented generation document, so the attacker never interacts with the application at all. The publication is explicit that current mitigations do not fully protect against either form. For any system you run or buy, treat prompt injection as a standing risk to design around, not a defect that a single control closes.
The answer to who is harmed changes by system. In direct injection, the attacker is the primary user, so the exposure sits in what that user can make the model do. In indirect injection, the attacker is usually a third party who controls content the system reads, and the harm often falls on the legitimate user whose session retrieves the poisoned data. Three questions shape the risk for any deployment: whether the model reads untrusted content at all, who the primary user is, and what permissions the model's tools hold, such as write access to email, tickets, payment systems or code repositories. A chatbot that only summarizes internal documents and a coding agent with repository write access carry very different consequences from the same attack technique.
The organization owns its controls and decides which trust boundaries, tool permissions and data sources are acceptable. An implementation partner can prepare the systems, separate trusted and untrusted data paths, configure logging, and assemble the test results and design records that show how the decisions were made. Only an independent assessor issues any report or opinion on those controls. NIST AI 100-2e2025 is a voluntary technical reference, not a certification scheme or a required standard, so no body accredits an organization against it. Before work begins, confirm with your legal and compliance leads which contractual or regulatory obligations actually require evidence about prompt injection in your environment.
Who is harmed in each case and why it matters for scope
NIST separates attacks where the attacker is the user from attacks where the attacker plants instructions in content the system reads later. Each card below names a party that carries exposure, so the compliance lead can record which exposures apply to a given system.
The attacker as primary user
In direct prompt injection, the attacker is the person typing into the model. AI 100-2e2025 treats their in-context instructions as overriding or being appended to higher-trust instructions such as the system prompt. Check whether any user-facing input can reach the system prompt boundary, and whether the model holds tools or data the attacker should not reach.
The primary user as the victim
In indirect injection, the user who asked a legitimate question is often the one harmed. A poisoned web page or document can steer the answer, leak data from the session, or push the user toward a harmful action. Scope this exposure by listing every interface where end users act on model output without independent review.
The owner of retrieved data sources
Anyone who can write to a web page, wiki, mailbox, ticket queue or RAG document store that the system retrieves is an attacker candidate. This party may be a supplier, a customer or an internal staff member. Inventory each retrieval source, record who can modify it, and treat externally writable sources as untrusted input.
The owner of tools and downstream systems
When a model calls APIs, sends email, or updates records, instructions hidden in tool output can trigger those actions. AI 100-2e2025 names AgentDojo for measuring agent vulnerability to injection through returned tool data. Document which systems the model can reach, which actions need human approval, and who owns each downstream permission.
The organization facing data theft or code execution
AI 600-1 describes indirect injection in LLM-integrated applications as a route to steal proprietary data or run malicious code. The organization running the application bears this loss even when it never received the attack directly. Scope it by mapping which data stores and execution environments sit behind the model.
The compliance lead documenting exposure
The compliance lead must record, for each system, which direct and indirect exposures apply and which mitigations were chosen. NIST says current mitigations do not fully protect against prompt injection, so the record should state the residual exposure and the design assumption. Keep this with the AI inventory and the risk decision it supports.
Which mitigation families NIST names and where each falls short
AI 100-2e2025 lists fine-tuning task-specific models as a mitigation for indirect prompt injection. The approach narrows what the model is trained to do so that it performs one defined task, which can reduce the range of behavior an injected instruction can trigger.
- What it reduces: the breadth of actions available to an injected instruction when the model only performs a defined task.
- Build cost: training data must be curated and labeled, and the model must be evaluated again after each retraining or base model change.
- Limit: a narrower model can still follow instructions embedded in the content it processes, so injected text may still change outputs within the task.
- Drift: new task requirements or an updated base model can reintroduce capabilities that the narrowing was meant to remove.
Confirm that the fine-tuned model is tested with injection payloads after every training run, not only at initial release.
AI 100-2e2025 also lists training models to follow hierarchical trust in prompts. The model is trained to weight instructions by source, so that system instructions outrank user input and user input outranks retrieved content.
- What it reduces: the chance that lower-trust content overrides higher-trust instructions, when the model has learned to prefer the higher-trust source.
- Opaque dependence: effectiveness depends on learned model behavior, which the deploying team usually cannot inspect directly.
- Limit: the trust ordering is a learned tendency, so it can fail on unusual phrasing or on conflicting instructions that the training did not cover.
- Verification gap: vendor training may not match a specific application's system prompts, retrieval sources or tool outputs, so the ordering needs local testing.
Ask the model provider what trust hierarchy the model was trained on and how that behavior was evaluated, then test the ordering against your own data flows.
AI 100-2e2025 names detection schemes as a mitigation family. These are classifiers or monitors that flag suspected injected instructions in inputs or outputs before the model acts on them or before the result reaches a user.
- What it reduces: known injection patterns and instruction-like content that a detector recognizes before the system takes action.
- Detection gaps: attackers can reword, encode or split instructions across sources, so a detector may miss forms it was not built to recognize.
- False positives: legitimate text, such as documentation about prompts, can be blocked, and tuning thresholds means accepting a trade-off between missed attacks and blocked work.
- Measurement need: logs of flagged and unflagged events are needed to show how the detector performs over time.
Confirm what the detector is tested against, how often its labels are reviewed, and what downstream control still applies when a missed injection gets through.
AI 100-2e2025 lists filtering instructions from third-party data as a mitigation. The control removes or blocks instruction-like content from retrieved web pages, documents, emails or tool results before the model reads them.
- What it reduces: the volume of instruction-like text that reaches the model from sources the organization does not control.
- Utility cost: content must still be useful for the task, so filtering that removes too much degrades answers.
- Limit: instructions written as ordinary prose may not look like instructions to a filter, so some pass through.
- Coverage gap: filters placed at one retrieval point do not cover content that reaches the model through another path, such as a tool or plugin, unless each path is covered.
Map every source the application reads from, including tools and plugins, and confirm that each one has its own filter or is excluded from the model's inputs.
AI 100-2e2025 names spotlighting, a prompt design that separates trusted instructions from untrusted data. The untrusted content is marked clearly, for example by encoding it or wrapping it in delimiters, so the model can tell data apart from instructions.
- What it reduces: the chance that the model treats retrieved text as instructions, by making the data channel explicit in the prompt.
- Interpretation limit: the same model reads the markers and the untrusted content, so the markers help but do not enforce a boundary.
- Mimicry risk: content that copies the delimiters can imitate trusted structure, so the scheme must be tested with payloads built to match it.
- Enforcement gap: a clearer prompt does not stop a tool from acting on an injected instruction, so spotlighting needs permission limits alongside it.
Test spotlighting with injection payloads that copy your delimiters, and confirm that tool permissions still constrain what the model can do if the markers fail.
Designing on the assumption that prompt injection is possible
This sequence turns NIST's design assumption into concrete architecture decisions, so that a successful injection has a bounded effect instead of a free hand over the system. It does not stop injection; it limits what an injected instruction can reach and documents what remains.
Inventory every input the model can read
The application owner, with the engineers who built the integration, lists each channel that reaches the model: user prompts, system prompts, retrieved web pages, RAG documents, tool outputs, email bodies and file uploads. Each entry names its source and who can write to it. The output is a single input register that covers every path into the context window. Done means the register has no unnamed channels and the team can explain how a tool result enters the prompt.
Classify each input as trusted or untrusted
Working from the register, the security owner labels each input trusted only when the organization controls its origin and integrity, and untrusted when an outside party could have written it. Retrieved content and tool output are untrusted by default under NIST's framing, because an attacker who controls a resource the system fetches can place instructions in it without touching the application. The output is a classification record with a reason for each label, reviewed by someone outside the build team. Done means every input has a label and no untrusted source is treated as instructions.
Separate model instances by permission
The architecture team decides which model instances may read untrusted data and which may call tools, write to stores or send messages. NIST's guidance points to using multiple LLMs with different permissions, so a model that reads external content should hold no privileged action rights, and a privileged model should never ingest raw untrusted text. The output is a permission matrix that maps each model instance to its data sources and allowed actions. Done means no single instance can both read untrusted content and perform high-impact operations without an intermediate control.
Route untrusted data through defined interfaces
Engineers replace open-ended passing of external content with narrow, typed interfaces, such as a constrained summary schema, a fixed set of extracted fields or a validated structure that the privileged component accepts. Free-text handoffs are removed or reviewed. The output is an interface specification and the code paths that enforce it, with logs showing what crossed each boundary. Done means test cases confirm that out-of-schema content is rejected or quarantined rather than forwarded.
Apply layered mitigations and test them
The team adds the mitigation families AI 100-2e2025 describes, such as instruction filtering on third-party data, detection schemes and trust-separating prompt designs, and then tests them. Red-team exercises should follow the Generative AI Profile's guidance on prompt injection, and open-source tools named in NIST material, including Garak, PyRIT and AgentDojo for tool-output injection, can be used to run repeatable checks. The output is a test report showing which injections succeeded, which were blocked and which remain. Done means each mitigation has a recorded test result, and failures feed back into the permission matrix.
Document residual risk and owner acceptance
Because NIST states current mitigations do not fully protect against prompt injection, the accountable executive records what the layered controls leave open, the worst credible effect of a successful injection and the monitoring that would detect it. The residual risk register names an owner, the decision made and the conditions for revisiting it. The output is a signed risk acceptance tied to the AI RMF MANAGE and MEASURE work already in the program. Done means no residual risk exists without a named owner and a review trigger.
Testing for prompt injection with the tools NIST names
NIST names a small set of tools and one generative AI action for testing how systems respond to injected instructions. Each card covers a different test job, from measuring agent exposure to tool-returned data to running structured red-team exercises. Together they let a team check whether its mitigations hold before and after deployment.
AgentDojo for tool-returned injection
AI 100-2e2025 names AgentDojo as a framework for measuring how vulnerable AI agents are to prompt injection carried in data returned by tools. This is the indirect case in practice: an agent reads a web page, email or file through a tool and follows instructions hidden in it. Use it to test whether an agent with real tool access can be steered away from the user's task. Confirm the current version and license with the project before you adopt it.
Garak for adversarial vulnerability scans
NIST names Garak as an open-source aid for identifying adversarial ML vulnerabilities. It is used to run repeatable probe sets against a model or application, which helps teams see which prompt patterns get past their guardrails. Treat results as a baseline, not a proof of safety, because NIST says mitigations remain incomplete. Confirm the current release and license with the project before you adopt it.
PyRIT for scripted attack campaigns
NIST also names PyRIT, an open-source aid for identifying adversarial ML vulnerabilities. Where a scanner runs fixed probes, a scripted red-team framework lets testers chain attempts, adapt to responses and test multi-turn attacks against a target system. Use it to test whether a system with separated trusted and untrusted inputs still resists injected instructions in realistic sequences. Confirm the current version and license with the project before you adopt it.
AI red-teaming under MS-2.7-007
GAI Profile action MS-2.7-007 suggests AI red-teaming to assess resilience against generative AI attacks such as prompt injection, along with misuse that facilitates attacks on other systems. The action calls for testing that covers both direct prompts and injected data in retrieval sources. Document scope, testers, findings and fixes so the results can serve as Measure evidence under the AI RMF. Red-team skills and tools are a team decision, so define the test plan in writing before testing starts.
What evidence shows prompt injection is being managed
An assessor reviewing prompt injection controls will ask for records that show the organization knew where untrusted content enters its AI systems and decided what to do about it. The core set is a threat model that lists direct and indirect injection paths for each application, trust-boundary documentation that shows which data sources, tools and agents sit on the untrusted side of each boundary, and red-team results that test those paths. Teams should confirm with their assessor which format these records must take, since NIST does not prescribe one. Test logs from tools such as Garak and PyRIT are useful supporting evidence, but they show what was run and what happened, so they should be tied back to the specific threat model entries they test.
The records that carry the most weight are the residual risk decisions. NIST's AI RMF places these under Measure and Manage: Measure asks that prompt injection risk be tracked with metrics and reviewed over time, and Manage asks that risk responses and residual risk be documented, including who accepted the remaining exposure and on what basis. For this topic, a residual risk entry should state which injection paths remain open after mitigations such as spotlighting or input filtering, why NIST's statement that current mitigations do not fully protect against prompt injection leaves that path open, and which design limits, such as restricting model access to untrusted data through defined interfaces, were applied instead. A decision that reads "mitigated" without this detail is the weakest record an assessor can receive.
Be clear about what no one can give you. NIST does not validate or endorse any organization's approach to using the AI RMF, and the Playbook describes no certification path, so no NIST document alone proves that prompt injection is managed. Any claim of compliance has to rest on the organization's own records and on whatever independent review its customers or regulators accept. Confirm with those parties which records they expect and whether they want a third-party assessment. DeployOpen can prepare threat models, test runs and documentation for that review, but the independent assessor issues any opinion on it.
Questions buyers ask about prompt injection and NIST
Does NIST require controls against prompt injection?
No. The NIST AI Risk Management Framework is voluntary. Neither private nor public sector organizations are required to use it, and NIST's FAQ says so directly. The AI RMF does not set prompt injection controls as a mandatory requirement. Its characteristic on secure and resilient systems is the place where security work connects to AI risk, and NIST AI 100-2e2025 explains the attack classes. Whether you must address prompt injection usually comes from a contract, a customer security questionnaire, a sector regulator or your own risk assessment. Ask the party making the request which document or clause they are relying on, and record the answer in your AI inventory so the obligation is traceable.
Is there a NIST certificate for prompt injection defenses?
No. NIST does not certify products, organizations or AI systems against the AI RMF or against the guidance in AI 100-2e2025. NIST's use cases page states that it does not validate or endorse any individual organization or its approach to using the framework. Any vendor or consultant who offers a NIST certificate for prompt injection resistance is describing something NIST does not issue. Evidence of a sound approach is what you can show: documented threat models, red-team results, design decisions about trust boundaries and monitoring records. If a customer asks for an assessment report, the independent auditor or assessor they engage issues it, not the organization that built the system.
How much effort does prompt injection testing take, and what drives it?
Effort depends on how many systems take untrusted input, how many tools and data sources the model can reach, and how often the application changes. A single chat assistant with no tools is a smaller test than an agent that reads email, browses the web and writes to a ticketing system. Testing covers direct attacks by users and indirect attacks through retrieved documents or tool output. The main cost drivers are the number of attack paths, the permissions granted to the model, the depth of red teaming, and whether testing is repeated after each model or prompt change. Ask any provider for a written scope listing the applications, data sources, tools and attack categories covered, and for the retest conditions. NIST gives no standard duration for this work.
What must our organization provide, and what can a partner do?
Your organization owns the system, the data, the access model and the decisions about acceptable risk. A partner can plan and run adversarial testing, document findings, help harden tool permissions and logging, and prepare evidence. Your team should supply architecture diagrams, a list of data sources and tools the model can call, the identities and privileges each component uses, logs, and a named owner who accepts residual risk. Be aware that DeployOpen does not issue audit reports or certifications, and it does not decide what risk your business accepts. Independent auditors and certification bodies do that work. Confirm the split of responsibilities in the written scope before work starts.
Can our in-house security team test prompt injection without outside help?
Yes, many teams can start in-house, especially if they already run application security testing. The skills needed are threat modeling for LLM applications, familiarity with how models combine instructions and data, and the discipline to record and retest findings. The difficulty is coverage and independence. An internal team that built the feature may miss attack paths it assumed were closed. Use the design guidance in AI 100-2e2025, such as assuming injection is possible when untrusted input reaches a model and limiting what the model can reach, as the basis for your test plan. Outside testers are useful when you need an independent view, when the system is high impact, or when a customer expects evidence from a third party.
How do we confirm that a testing tool or benchmark is still current?
Check the project's own repository, release notes and documentation before relying on any tool, because the status of open-source projects changes. NIST AI 100-2e2025 names AgentDojo for measuring agent vulnerability to injection through tool output, and the open-source tools Garak and PyRIT for identifying adversarial ML weaknesses. Naming a tool in a NIST publication does not confirm that it is maintained, that its license fits your use, or that it supports your model provider. Record the version you tested with, the date you checked, and the license terms. Then include a periodic review in your testing procedure so that a tool that stops being maintained is replaced before the next assessment.
Where should we start if a customer asks about prompt injection?
Start by mapping where untrusted content enters the system and what the model can do with it. Many customer questionnaires ask whether you have tested for direct and indirect injection, whether tools run with least privilege, and whether there is monitoring. Answer from evidence: a current inventory of AI systems, a threat model, test results with dates, and a description of design limits. The NIST AI RMF Playbook offers documentation prompts for related subcategories, though NIST notes it is not a checklist. If you need an independent test or a hardened, self-hosted environment for the model and its tools, scope that work in writing with a provider so the deliverables are clear before any fees are agreed.
What to confirm before you scope the work
Before you scope a prompt injection assessment, confirm which models in your environment see untrusted content. The first check is to list every model endpoint that receives text from outside your trust boundary, including customer messages, uploaded files, emails, web pages fetched at runtime and documents pulled from retrieval indexes. AI 100-2e2025 explains that indirect injection works because a model combines data and instruction channels, so a model that never sees third-party content has a much smaller exposure than one that summarizes inbound email. For each endpoint, record the system prompt, whether it holds any secrets or privileged instructions, and whether the same model also serves users with different permissions.
Next, map the tools each model can call and the data each tool can reach. Tool output is itself untrusted input, and the agent-related guidance in AI 100-2e2025 names tool-returned data as a route for injected instructions. Ask which tools can write, send, delete or transact, which can read sensitive records, and whether any action runs without a human approval step. Confirm whether the application enforces the least privilege of the end user or runs every call under a shared service account. A short table of model, tool, permission and approval gate is often enough to show where an injected instruction could cause harm.
Finally, confirm who owns each data source and whether any sector regulator names NIST guidance. Each retrieval source, whether a wiki, ticketing queue, shared drive or vendor feed, needs a named owner who can say who can edit it and how content is reviewed. Those owners decide whether a source counts as trusted, and they will be the people affected when indirect injection succeeds. Separately, check with your regulator, contract counterparties and internal risk committee whether they reference the NIST AI RMF or its profiles. That answer determines whether you test to NIST's suggested actions or to a narrower internal standard, and whether the results need to be documented in a particular format. Ask any assessment partner to confirm the same points in writing before work begins.
How DeployOpen works on this
DeployOpen scopes prompt injection work around the attack paths NIST describes, direct and indirect, against the systems you run in your own environment. We test, document and hand over what the written scope covers, and an independent auditor or assessor decides what any evidence proves.
How we scope
- Map which LLM applications take user prompts and which pull in web pages, RAG documents or tool output as untrusted data.
- Agree in writing which direct and indirect injection paths, tools and data sources are in and out of the test.
- Confirm the permissions each model holds, since those set how far an injected instruction could reach.
How we build
- Deploy the test tooling inside your cloud account, data center or air-gapped network, not on our infrastructure.
- Prepare test corpora for direct prompts and for poisoned documents placed where your retrieval pipeline will find them.
- Configure red-teaming runs with AgentDojo for tool-output injection and Garak or PyRIT for adversarial prompt coverage, where the scope calls for them.
How we hand over
- Deliver test plans, attack inventories, raw run results and a findings report mapped to each tested injection path.
- Document the design assumptions in use, including where untrusted input reaches a model and what interfaces limit it.
- Transfer test scripts and configurations so your own security team can rerun them after changes.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous testing are available as a retainer, which suits systems that change often.
- Ask your independent assessor which evidence they will accept, and confirm that scope in writing before testing begins.
