What NIST AI 600-1 asks for in pre-deployment testing and AI red teaming
When a customer, contract or regulator asks whether a generative AI system was tested before release, the honest answer starts with the framework's status. The Generative AI Profile, NIST AI 600-1, is voluntary and certifies nothing. It asks organizations to test generative AI in a way matched to the deployment context and to run structured red teaming. The answer to "was it tested?" therefore depends on the system, its intended context of use and the expertise of the people who did the probing. Only an independent auditor, assessor or certification body can issue a report on that testing, and an implementation partner's documentation does not substitute for one.
The profile is direct about why pre-deployment testing is hard. Its Appendix A.1.4 says that pre-deployment TEVV processes for generative AI may be inadequate or mismatched to the deployment context. It notes that testing through video games or human standardized tests does not guarantee validity, and that jailbreaking or prompt engineering tests may not systematically assess validity or reliability. It also describes a gap between laboratory and real-world results, since current testing often relies on benchmark datasets and in silico techniques, a gap likely made worse by prompt sensitivity and varied contexts of use. A test suite that passes benchmarks and jailbreak attempts should not be presented as proof of fitness for a specific use.
For red teaming, the profile defines AI red-teaming as a structured exercise to probe a system for flaws and vulnerabilities, such as inaccurate, harmful or discriminatory outputs. It describes four types: general public, expert, combination, and human/AI. It also notes that generative AI-led red-teaming can be more cost effective than human red-teamers alone. Quality depends on the red team's background and expertise, and red teams should understand the deployment context. Results need further analysis before they inform governance decisions. Confirm with the team that will run the exercise which type fits the system, who is on the team, and how findings will reach the go or no-go decision.
The four types of AI red teaming and when each fits
NIST AI 600-1 describes four types of AI red teaming: general public, expert, combination, and human/AI. Each surfaces different flaws, and none is sufficient alone. The right mix depends on who the system will face after release and how much domain judgment the findings require.
General public red teaming
Broad groups of people without special training try to make the system produce inaccurate, harmful or discriminatory outputs. This type surfaces how ordinary users might misuse or misread the system, including phrasings no internal team would think of. Check that participants reflect the actual user population, that findings are logged with the prompts that produced them, and that the organization has a plan for triaging a large volume of reports before any go or no-go decision.
Expert red teaming
Specialists with domain knowledge, such as clinicians, auditors, security researchers or lawyers, probe the system for failures that matter in their field. NIST says output quality depends on the background and expertise of the red team, so a generic tester may miss a plausible but wrong legal citation or clinical recommendation. Confirm that experts understand the deployment context and that their findings are analyzed before they feed governance decisions.
Combination red teaming
This model pairs public participants with expert testers in one exercise, so breadth and depth are covered together. It suits systems whose users range from novices to professionals, where a single group would leave gaps. Ask the provider to show how the two groups' findings are reconciled, how duplicate issues are removed, and which findings are routed to the MANAGE process for treatment.
Human/AI red teaming
Here, generative AI is used to generate attack prompts and variations at scale, with human reviewers judging the results. NIST notes that generative AI-led red teaming can be more cost effective than human red teamers alone, which makes repeated testing practical. Human judgment remains necessary for assessing harm and context, so check that reviewers are qualified, that the attacking model's own limits are documented, and that the approach still covers the attack classes listed in MS-2.7-007.
How a red-teaming exercise moves from scope to governance decision
The sequence turns a loose request to test a generative AI system into adjudicated findings that a decision owner can weigh before release. Each phase produces a written artifact that the next phase depends on.
Define deployment context and threat scope
The organization, with its AI system owner, writes down the intended use, the users and affected groups, the deployment setting, and the abuse and failure modes the exercise must probe. This includes the attacks the Generative AI Profile names under MS-2.7-007, such as prompt injection, malicious code generation, enhanced phishing, data poisoning, membership inference, model extraction and sponge examples, where they apply to the system. The step closes with a signed scope statement that lists in-scope components, out-of-scope items, rules of engagement and the decision the results will inform. If the scope cannot name the deployment context, the red team has nothing to test against, so the step is not complete until the context is written down.
Select red team type and expertise
Choose one of the four types NIST describes in AI 600-1: general public, expert, combination, or human/AI red-teaming. Match the type to the threat scope, and confirm that the people involved have domain expertise and understand the deployment context, because NIST notes that output quality depends on the team's background. Where generative AI-led red-teaming is used alongside people, record that choice and the reason for it. The output is a red team roster with stated qualifications, a documented method, and a conflict-of-interest check confirming that testers are not the people who built the controls under test.
Run structured probes against the system
The red team runs structured probes that follow the scope, using a recorded test plan, fixed prompts or scenarios, and a log of every input, output, system configuration and model version. Probes should cover the deployment setting, not only generic jailbreak prompts, since NIST cautions that jailbreaking alone may not systematically assess validity or reliability. Testers record each flaw they find, such as inaccurate, harmful or discriminatory outputs, along with the conditions that produced it. The output is a complete probe log and a list of candidate findings, each tied to a scope item and reproducible by someone other than the tester.
Analyze and triage results
A reviewer who did not run the probes checks each candidate finding for reproducibility, severity, affected users and the likelihood of the same failure in production. Duplicates are merged, false positives are documented and dropped, and findings are grouped by the AI RMF category they touch, such as MEASURE 2 for trustworthiness evaluation or MANAGE 1 for risk response. NIST states that results need further analysis before they inform governance decisions, so this step is where raw output becomes evidence. The output is an analyzed findings register with severity, confidence, the analyst's rationale and open questions for the system owner.
Adjudicate findings with the system owner
The accountable owner and the relevant risk, legal and domain reviewers review the analyzed findings and decide which are accepted, mitigated, deferred or treated as blockers. Each decision is recorded with the evidence behind it, any residual risk that remains, and the monitoring that will watch the risk after release. Adjudication is the point where the organization, not the red team, takes responsibility for the outcome. The output is a signed adjudication record that links every high-priority finding to a decision and an owner.
Feed adjudicated findings into the go or no-go decision
The adjudicated record goes to the body that holds release authority, which makes the go or no-go determination that the AI RMF places under MANAGE 1. The decision memo states the scope tested, the red team type used, the high-priority findings and how each was resolved, the residual risks accepted, and the conditions attached to release. Findings that remain open are assigned to monitoring under MEASURE 3 and MANAGE 4. The output is a dated decision record that a reviewer can trace back to the scope and probe log, and that is retained with the system's documentation.
What MS-2.7-007 asks red teams to probe
MS-2.7-007 asks a red team to probe a generative AI system for two different kinds of attack surface at once. The first is abuse that uses the model to attack other systems, such as malicious code generation or enhanced phishing. The second is the set of attacks specific to machine learning, including prompt injection, adversarial prompts, data poisoning, membership inference, model extraction and sponge examples. The action is a suggestion in the Generative AI Profile, not a test script, so the organization decides how deep each probe goes. The reader's first task is to decide which of these attack classes apply to the system's architecture, such as whether it calls tools, retrieves documents or accepts uploads from outside users.
Each attack class should produce a specific artifact. For abuse-enablement probes, keep the prompts attempted, the outputs classified as harmful or not, and the guardrail or filter that responded. For prompt injection, keep the injected instructions, the data sources they arrived through and whether the system acted on them. For data poisoning, membership inference and model extraction, the evidence is usually a test plan that names the training data, query budget and rate limits being examined, plus results showing whether leakage or copying was detected. For sponge examples, keep resource consumption measurements under crafted inputs. Confirm with your security team which logging is already available, since missing logs often decide whether a result can be shown to an assessor at all.
A red team report is only useful if it names who ran each probe and how they were qualified. The Generative AI Profile ties result quality to the team's background and its awareness of the deployment context, so the report should record those details. Before results shape a go or no-go decision, have someone outside the testing team review the findings. Ask any external provider to show how each attack class in MS-2.7-007 was covered, which probes were out of scope and why, and how findings will be tracked to remediation. Those answers, rather than the number of prompts run, are what show the test was done as the Profile intends.
Why Texas law and Colorado's former AI Act pointed to red teaming
Texas HB 149, the Texas Responsible Artificial Intelligence Governance Act enacted in the 89th Legislature, amends the state's public information provisions in a way that names adversarial testing or red-team testing among the ways a defendant can discover a violation. The provision does not require red teaming. It recognizes that testing of this kind can surface the facts a defendant needs to understand how an AI system behaved. Read the enrolled text directly before drawing conclusions about your own obligations.
- What the text names: Adversarial testing or red-team testing appears as one of several discovery routes, not as a mandated program.
- What it does not say: The provision does not set a method, team composition, frequency or pass criteria for red teaming.
- Who it affects: Organizations that build, deploy or sell AI systems that could become the subject of a Texas proceeding.
- Where the evidence goes: Test plans, scope statements, findings and remediation records are what would need to be produced and explained.
- Why the wording matters: A red-team exercise that is poorly documented gives a defendant little to point to.
Before relying on this provision, have counsel confirm how Section 552.105(e) applies to your organization, your role in the AI system and your jurisdictional exposure.
Colorado's SB24-205 set obligations for developers and deployers of high-risk artificial intelligence systems. Its defense provisions referred to adversarial testing or red teaming as defined or used by NIST. That reference tied the statutory language to the NIST definition, so the NIST description of structured probing for flaws and vulnerabilities became the reference point for what counted. Colorado later replaced this act, so confirm its current status before treating it as operative.
- The reference point: The defense pointed to NIST's meaning of adversarial testing or red teaming, not to a separate Colorado definition.
- Why NIST's definition matters: NIST describes red teaming as a structured exercise to probe an AI system for flaws and vulnerabilities, such as inaccurate, harmful or discriminatory outputs.
- What the defense supported: Testing conducted in line with that meaning could be presented as part of a reasonable-care record.
- What the defense did not do: It did not certify a system or create a safe harbor that applies without facts about the testing performed.
- Versions matter: Confirm which NIST text the statute references and whether the reference points to the July 2024 Generative AI Profile.
Ask counsel whether the replacement law changed or removed this reference and whether any retained obligations still depend on it.
Across both jurisdictions, red teaming functions as evidence of care. Neither law turns it into a certification, an audit report or a required attestation. A court, regulator or opposing party would look at what was tested, who tested it, how results were analyzed and what changed afterward. Organizations that want to rely on red teaming in a legal setting should plan for that review from the start.
- Scope records: Document the system, deployment context and the risks the exercise targeted.
- Team qualifications: Record the background and domain expertise of the people who ran the test, since output quality depends on it.
- Analysis before action: Keep the analysis that turned raw findings into governance decisions.
- Remediation trail: Show which findings led to changes, accepted risks or deployment decisions.
- Independence: Note whether internal staff, an outside firm or a mix ran the exercise.
Confirm with counsel which records must be retained, for how long and in what form, because the answer depends on the specific legal obligation.
The legal relevance of red teaming depends on facts that only the organization and its counsel can establish. Statutory text, effective dates, amendments and enforcement posture can all change. The safe reading is that these provisions give red teaming a recognized evidentiary role, while leaving the legal conclusion to qualified counsel. An implementation partner can prepare test records and systems, but it does not give legal advice on how a statute applies.
- Applicability: Whether a given system falls within the statute's definitions.
- Current status: Whether the cited provision is in force and whether a replacement law changed the reference.
- Role-based duties: Whether you are a developer, deployer or both, since duties can differ.
- Interaction with other laws: How sector rules and contracts apply alongside state requirements.
- Evidence standards: What a regulator or court would expect the test record to show.
Request a written opinion from counsel that cites the current text of each provision before you rely on it in a contract, filing or compliance statement.
What a program needs in place before testing counts as evidence
Testing only supports a decision when the organization can say what the system is for, what could go wrong and who tried to break it. The five items below connect the deployment context, the risks named in MAP and MEASURE, the test plan, the red team and the record of what was decided. Each card says how to check the item before results are relied on.
Documented deployment context
Testing has to mirror the setting where the model will run. The Generative AI Profile warns that pre-deployment testing may be mismatched to deployment contexts, so the program needs a written description of intended users, intended tasks, input sources, integrations and the environment the model will face. Check that the description is dated, owned and updated when the use case changes. A test that does not match this description tells the reader little about the deployed system.
Named risks from MAP and MEASURE
A red team needs a target list. Each risk to test should trace back to a MAP entry describing the system's categorization and impacts, and to a MEASURE metric chosen for that risk. Check that every risk to be probed has an owner, a metric and a reason it was selected. Risks that appear only in a general concern list rarely produce testable hypotheses.
Test plan with defined scope
The test plan states which components are in scope, including third-party models, datasets and plug-ins, which attack types will be attempted and what counts as a finding. It also records exclusions and the reason for each. Check that the plan names the testing methods (benchmarks, field testing, participatory feedback or red teaming) and explains what each can and cannot show. Benchmarks and jailbreak tests alone do not establish validity or reliability.
Red team with documented expertise
The Generative AI Profile says red-team output quality depends on the team's background and expertise, and that teams should understand the deployment context. Record which of the four red-teaming types was used (general public, expert, combination or human/AI), who was on the team and what domain knowledge they bring. Check that the team's qualifications are written down before results are accepted, because an unqualified team can produce findings that look thorough but miss the failures that matter.
Record of findings and decisions
Red-team results need further analysis before they inform governance decisions. Keep a log that links each finding to the risk it tests, its severity, the analysis performed, the go or no-go decision and any residual risk accepted. Check that someone with decision authority signed off on each release decision and that the log can be produced to an assessor or regulator. Without this record, testing shows activity but not evidence that the organization acted on it.
Who owns red teaming, who prepares evidence and who issues reports
Your organization owns the AI system, its risk tolerances and every release decision. The Generative AI Profile expects testing results to be analyzed before they inform governance decisions, and that analysis belongs to people accountable for the system, not to the testers who produced the findings. Your team also decides what counts as an acceptable residual risk and whether the system goes live.
- Scope and purpose: Define the intended use, deployment context and users that the red team must reflect.
- Red team selection: Decide which of the four red-teaming types fit the system, and confirm the testers have domain expertise and knowledge of the deployment context.
- Findings triage: Review each flaw, decide severity and assign remediation owners before anything is closed.
- Go or no-go decision: Record the release decision, the risks accepted and the people who approved them.
- Governance records: Keep the test plan, raw results and decisions together so later reviewers can trace them.
Before the next release, confirm that someone inside the organization can explain every accepted finding without referring to the tester's report.
An implementation partner such as DeployOpen can plan and run security testing within a written scope, and can prepare systems, logs and records so that your team's evidence is complete and traceable. Its work supports your decisions but does not replace them. DeployOpen does not issue audit reports, attestations or certifications, and NIST does not offer an AI RMF or Generative AI Profile certification.
- Test design: Help map the scope to the Generative AI Profile actions, such as MS-2.7-007, and agree the test types and boundaries in writing.
- Testing execution: Run application, API, AI system and agent testing where the scope includes it, and document methods and inputs.
- Evidence preparation: Organize test logs, configuration records, versions and remediation tickets into an evidence pack.
- Remediation support: Help fix technical issues in the self-hosted environment and retest them.
- Handover: Deliver reports and records to your team, with a clear list of what was tested and what was not.
Ask any partner to state in its proposal which deliverables are test results and which are assessment conclusions, and confirm that it will not issue a certificate.
Independent auditors, assessors or certification bodies review evidence and, where a formal report or attestation is needed, issue it. Their independence depends on not having designed or run the controls they test. For AI RMF work, the framework itself is voluntary and has no assessor program, so any independent review is a contractual or internal arrangement, and its scope must be agreed up front.
- Evidence review: Test whether the records support the claims your team makes about red teaming and pre-deployment testing.
- Sampling and walkthroughs: Check that the process ran as documented, not just that documents exist.
- Reporting: Issue a report only in the form and scope the engagement defines, and only for the period or system covered.
- Qualification checks: Confirm the assessor's credentials and the report types it can issue for your customer or regulator.
- Limits: Note what the review did not test, such as deployment-specific misuse that was outside the test scope.
Before engaging an assessor, ask the customer or regulator which report or evidence they will accept, since that determines the scope you need.
Questions buyers ask about pre-deployment testing and red teaming
Does NIST AI 600-1 require red teaming?
No. NIST AI 600-1 is the Generative AI Profile for the AI Risk Management Framework, and the AI RMF is voluntary. The profile describes AI red teaming as a structured exercise that probes a system for flaws and vulnerabilities, such as inaccurate, harmful or discriminatory outputs, and it lists red teaming among the forms of structured public feedback. Its suggested action MS-2.7-007 recommends red teaming to assess resilience against abuse that facilitates attacks on other systems, generative AI attacks such as prompt injection, and machine learning attacks such as data poisoning and model extraction. Whether your organization needs it depends on the contracts, regulations and customer commitments that apply to your system. Confirm those requirements with your legal and compliance teams before scoping the work.
Can benchmarks and jailbreak tests replace red teaming?
No. Benchmarks and jailbreak tests measure specific behaviors, but they do not establish that a system is valid, reliable or safe in your deployment. NIST AI 600-1 notes that pre-deployment testing for generative AI may be inadequate or mismatched to the deployment context, and that jailbreaking or prompt engineering tests may not systematically assess validity or reliability. It also describes measurement gaps between laboratory and real-world settings, a gap likely made worse by prompt sensitivity and varied contexts of use. Benchmarks remain useful as one input. Red teaming adds scenarios drawn from your users, data and workflows, which fixed datasets rarely capture. Ask your team which benchmarks were run, what they did not cover, and which field tests show behavior in the actual deployment.
Who can red team a generative AI system, and how do we judge expertise?
NIST AI 600-1 describes four types of AI red teaming: general public, expert, combination, and human/AI. It states that the quality of results depends on the background and expertise of the red team, and that red teams should show domain expertise and awareness of the deployment context. When evaluating a provider or internal team, ask for evidence of domain knowledge in your sector, experience with adversarial techniques against language models and related systems, and a written method that explains how findings are recorded and reproduced. Ask how the team handles sensitive outputs and whether it uses generative AI tools to widen coverage. The profile notes that generative AI-led red teaming can be more cost effective than human red teamers alone, but human judgment is still needed to interpret results.
Does AI red teaming get certified?
No. NIST has not created a certification or conformity assessment for AI red teaming, and NIST does not validate or endorse any individual organization or its approach to using the AI RMF. An independent assessor can review your red-teaming records, methods and governance as part of an audit against a specific standard or contract, but that review produces an assessment or report, not an AI RMF certificate. Organizations sometimes ask for red-team reports as evidence in procurement. Ask any vendor exactly which document they will deliver, who wrote it, and whether it is independent of the team that performed the testing. DeployOpen performs security testing, including AI and agent security testing, and prepares evidence for assessments, but it does not issue audit reports or certifications.
What does red teaming need from our internal team?
Red teaming needs internal people who understand the system and can act on results. Plan for these inputs before testing starts:
- A named owner who can decide which findings require changes to the system, its prompts, filters or access controls.
- Documentation of intended use, users, data sources, and known limits, so testers test realistic scenarios.
- Access to logs, staging environments and the system components in scope, with rules for handling sensitive outputs.
- Staff who can triage results, because NIST AI 600-1 says results need further analysis before they inform governance decisions.
The effort depends on how many components are in scope and how many findings require remediation. Ask the provider to state the scope, the test environment, and the team's responsibilities in writing before work begins.
Do Texas or Colorado still reference red teaming?
Texas does, and Colorado's referenced language has changed. Texas HB 149, the Texas Responsible Artificial Intelligence Governance Act, appears in Section 552.105(e) of the enrolled text, which lists adversarial testing or red-team testing among the ways a defendant can discover a violation. Colorado's SB24-205 included a defense that referred to adversarial testing or red teaming as defined or used by NIST. Colorado later replaced its AI Act, so confirm the current text of any Colorado obligation before relying on that reference. Read the enrolled Texas text and the current Colorado act directly, and have counsel confirm how each applies to your organization. Testing records are useful under either law only if they document scope, methods, findings and remediation.
How DeployOpen scopes red teaming and prepares evidence
DeployOpen is a Chainscore Labs company that runs security testing of applications, APIs, cloud and infrastructure, AI systems and agents, and red teaming, inside the client's own environment. For a generative AI system, the scope conversation starts with the deployment context: which model and application components are in play, which data and tools the system can reach, who the intended users are, and which abuse cases matter most to the organization. Those decisions determine whether the exercise is a structured red team of the kind AI 600-1 describes, a narrower security assessment of the surrounding application and infrastructure, or both. The organization still owns the risk decisions. DeployOpen plans the work, runs it, and reports what it found.
Work is fixed-scope. The scope, deliverables, test boundaries, access arrangements and timeline are agreed in writing before any testing starts, and a quote follows from that written scope. Ask the proposal to name the red-team type, the expertise and domain background of the testers, the systems and data in bounds, the rules for handling harmful outputs, and what the final deliverable contains. Ask also who is paid for what, because access setup, model hosting, and any follow-up retesting are cost items that can differ from the testing itself. Confirm the scope with your own legal and security teams before testing begins.
DeployOpen prepares test records and evidence for independent review: the test plan, the prompts and scenarios used, the observed outputs with timestamps, the severity and reproduction notes, and the remediation tracking. That material supports the organization's own analysis, which AI 600-1 says must happen before results inform governance decisions. DeployOpen does not issue audit reports, attestations or certifications, and no testing engagement guarantees that a system is free of flaws or meets any framework. Independent auditors and assessors decide what the evidence shows, and the organization decides whether to release the system.
