NIST AI 600-1 information security risk: prompt injection and data poisoning

MS-2.7-001 suggests applying established security measures to assess how likely and how severe each threat to a generative AI system is. The list in the Generative AI Profile covers backdoors, compromised dependencies, data breaches, eavesdropping, man-in-the-middle attacks, reverse engineering, autonomous agents, model theft or exposure of model weights, and inference, bypass and extraction attacks. The action is a threat assessment, so its output is a ranked view of which threats apply to your system and how much harm each would cause.

• Model weights and training data: Record where weights, checkpoints and training sets are stored, who can read them, and whether a theft or exposure would matter for your business.
• Dependencies and pre-trained models: List libraries, model files and datasets pulled from outside your organization, since compromised dependencies appear in the same action.
• Agents and tool access: Note which actions an autonomous agent can take, because the action names agents as a threat source.
• Extraction and inference paths: Identify which endpoints expose outputs that could be used to infer training data or copy model behavior.
• Likelihood and magnitude: Score each threat with a documented method, so later tabs have a baseline to measure against.

Illustrative example: a team that hosts an internal model may find that its weights are the highest-magnitude threat, even if its public API is the most exposed surface. Confirm the scoring method with your risk owner before you rely on the result.

MS-2.7-004 suggests identifying metrics that show whether your security measures are working. The Generative AI Profile names data provenance, the number of unauthorized access attempts, inference, bypass and extraction events, penetrations, and provenance verification as examples. The action does not set targets. It asks you to choose metrics that match the controls you have, record them over time, and show whether they change after a control is added.

• Access and authentication logs: Count unauthorized access attempts against model endpoints, vector stores and training pipelines, and keep the source of each attempt.
• Provenance verification: Track the share of training data and model artifacts with verified origin, and record each failed verification.
• Extraction and bypass events: Log queries that match known extraction or bypass patterns, and review them on a set schedule.
• Penetration results: Record the number of findings, their severity and the time to close them, so trends are visible.
• Metric ownership: Assign each metric to a named owner and document how often it is reviewed.

The evidence from this action is a metric register, the logs behind it and a record of how the metrics were reviewed. Confirm with your security team which logging sources already exist, because a metric you cannot collect cannot show effectiveness.

MS-2.7-007 suggests AI red teaming to test resilience against three groups of attack: abuse that helps attack other systems, such as malicious code generation or enhanced phishing; generative AI attacks, such as prompt injection; and machine learning attacks, such as adversarial prompts, data poisoning, membership inference, model extraction and sponge examples. This action is the only one of the three that tests the system directly, so its output is observed behavior under attack rather than a register of risks.

• Prompt injection: Test direct prompts and indirect prompts hidden in content the system retrieves, and record which instructions the model follows.
• Data poisoning: Test whether tampered training or fine-tuning data changes outputs in ways a clean run would not.
• Membership inference and extraction: Test whether the model reveals whether a record was in its training data, or reproduces protected content at scale.
• Sponge examples: Test inputs designed to drive up compute use and latency, and measure the effect on availability.
• Findings and retests: Record each finding, the fix applied and a retest result, so the red team report shows closure.

The Playbook suggests running red-team exercises under adversarial conditions and sharing attack patterns with incident databases. Confirm with an independent tester whether the scope, rules of engagement and reporting format fit your governance process.

The Generative AI Profile presents these actions as suggestions, and the AI RMF says organizations choose which categories and subcategories to use. Your organization decides which of the three actions apply to each system, based on what the system does, the data it touches and the threats your risk owners accepted. A low-exposure internal tool may need a lighter threat assessment and fewer metrics than a customer-facing agent with tool access.

• Start with the threat assessment: Use MS-2.7-001 to decide which threats are in scope before you pick metrics or tests.
• Match metrics to controls: Use MS-2.7-004 only for metrics you can collect from the controls you run.
• Scale red teaming to exposure: Use MS-2.7-007 for systems that accept untrusted input, retrieve external content or take actions.
• Connect the actions: Feed red-team findings back into the threat assessment and metric register.
• Document the decision: Record which actions you applied, which you excluded and the reason for each choice.

Before you rely on this split, confirm with your risk owner and any external auditor how they expect the three actions to be documented, since the choice itself is part of the evidence.

Your organization owns the risk decisions, the controls and the access that makes them work. Under the Generative AI Profile, the Measure actions on information security (MS-2.7-001, MS-2.7-004 and MS-2.7-007) ask you to assess the likelihood and magnitude of threats such as prompt injection and data poisoning, and to decide which metrics and red-team results are acceptable. No partner can accept that residual risk for you.

• Risk decisions: Setting the risk tolerance for prompt injection, data poisoning and model weight exposure, and recording who signs off on residual risk.
• Controls: Choosing and operating the countermeasures, such as authentication, throttling and access restrictions on retrieval sources and training data pipelines.
• System access: Holding administrative credentials, approving changes to training datasets and controlling who can alter the model, prompts or retrieval corpus.
• Ownership of findings: Triaging red-team results and deciding which fixes ship and which risks are accepted.
• Third-party obligations: Requiring security audits of third-party AI resources and keeping the contracts that support them.

Before engaging anyone, confirm internally who owns each of these decisions and who holds the keys to the training pipeline and the vector store.

An implementation partner such as DeployOpen prepares the systems, runs security testing and organizes the evidence that your team and an assessor will review. In a self-hosted deployment inside your own cloud accounts, data centers or air-gapped network, the partner works within the access you grant. It does not decide your risk tolerance and it does not issue any report or certificate.

• System preparation: Hardening the model serving stack, logging layers and data pipelines so that the controls you chose can be operated and observed.
• Security testing: Running application, API and AI red-team exercises that exercise prompt injection, indirect injection through retrieved content and data poisoning scenarios.
• Evidence organization: Collecting test results, metrics such as unauthorized access attempts, and provenance records into an evidence set mapped to the relevant subcategories.
• Remediation support: Fixing issues within the agreed scope and retesting after changes.
• Documentation: Writing runbooks and records that describe how each control operates.

Ask any partner to state in writing which systems it will touch, which credentials it will hold, and which deliverables it will hand over, because the scope determines what you can later show an assessor.

An independent assessor reviews the evidence and, where its engagement provides for it, issues a report. For AI RMF work, there is no NIST certification or accreditation to obtain, so an assessor's output is a report on the scope and criteria you and it agreed, not a NIST-issued attestation. The assessor should be independent of the people who built and tested the controls.

• Independence: Keeping the assessor separate from the team that designed the controls and from the partner that ran the red-team exercises.
• Scope and criteria: Agreeing in advance which controls, subcategories and Generative AI Profile actions are in scope.
• Testing: Sampling evidence, reviewing logs and, where relevant, reperforming tests.
• Reporting: Issuing any report under the terms of its own engagement, which may cover findings, exceptions and limitations.
• Claims discipline: Making sure your public statements describe the assessment accurately and do not imply NIST endorsement.

Before hiring, ask what criteria the assessor will use, what type of report it issues and whether it has relevant AI security experience.