AI risk assessment vs AI red teaming: which comes first
Risk assessment sets out what can go wrong across an AI system and how much each failure matters, while red teaming proves or disproves specific failure paths on a named system. If you must show AI governance and test evidence to an auditor, regulator or board, you need both, but they answer different questions and produce different artifacts. The assessment produces a documented, prioritized risk register with owners and treatments. A red team produces tested evidence that a particular attack either worked or did not against a particular build, model version and configuration. Choose risk assessment alone when you are deciding whether a use case should proceed, how it is governed, or where testing effort should go. Choose red teaming alone when a single system is already live, carries high exposure, and you need to know whether its guardrails hold under adversarial pressure.
Risk assessment usually comes first because it determines what red teaming should target. EU AI Act Article 9 requires a documented risk management system for high-risk AI that identifies and estimates known and reasonably foreseeable risks across the lifecycle. ISO/IEC 42001:2023 places AI risk assessment in clauses 6.1.2 and 8.2, separate from impact assessment in clauses 6.1.4 and 8.4. NIST AI 600-1 lists adversarial role-playing, red-teaming or chaos testing under MAP 5.1 as a way to find unforeseen failure modes, which means red team output is an input to risk identification, not a substitute for it. Without the assessment, a red team has no agreed scope, no severity criteria and no owner to act on what it finds. For an illustrative internal chatbot with retrieval access to HR records, the assessment would rank data exposure through prompt injection as a material risk, and the red team would then test whether that path is actually open.
Red teaming can run before a formal assessment when one system carries high exposure, such as a customer-facing agent with write access to payments, and the team needs evidence now. Treat those early findings as provisional until NIST AI 600-1 style additional analysis confirms their severity and feeds them into the register. Run both together for release testing of high-risk systems, with the assessment defining metrics and thresholds and the red team reporting results against them. Confirm with your certification body or auditor which artifacts they expect, and confirm with your legal team how EU AI Act Article 9(6) and 9(8) testing thresholds apply to your system.
How the two approaches differ across scope, method and outputs
This table compares AI risk assessment and AI red teaming on goal, scope, owner, method, evidence and sign-off. Read it to decide which activity answers your governance question and which one answers your testing question, since most organizations need both.
| Dimension | AI risk assessment | AI red teaming |
|---|---|---|
| Primary goal | Identify, estimate and evaluate known and foreseeable AI risks across the system's lifecycle, then decide on controls | Find unforeseen failure modes and exploitable behavior in a specific system by attacking it in a controlled way |
| Scope | The AI use case, its context, affected people, data and dependencies, often covering several models or deployments | A defined system, model endpoint or application version, with rules of engagement that set what may be attacked |
| Typical owner | Risk, compliance or AI governance function, with product and legal input | Security testing or red team function, often an external team, reporting to the system owner |
| Core method | Structured workshops, documented risk identification, likelihood and impact estimates, and review against defined criteria | Adversarial role-play, prompt and tool-abuse attempts, chaos testing and scenario-driven attacks, as NIST AI 600-1 MAP 5.1 describes |
| Main inputs | Use case documentation, intended purpose, data sources, post-market or operational data, prior incidents and impact assessments | Target access, scope, threat scenarios, risk hypotheses from the register, and any prior test results |
| Standards referenced | ISO/IEC 42001 clauses 6.1.2 and 8.2 (AI risk assessment), NIST AI RMF MAP and MEASURE, EU AI Act Article 9 | NIST AI 600-1 action MP-5.1-005, MITRE ATLAS technique catalog, OWASP GenAI lists for test planning |
| Evidence produced | Risk register entries, risk criteria, treatment decisions, residual risk statements and review records | Test plan, attack logs, reproduction steps, observed failure modes and severity notes tied to specific inputs |
| Success criteria | Risks are identified, rated against agreed criteria, assigned treatments and reviewed on a defined cycle | Failure modes are found, documented and reproducible, with coverage recorded against the agreed scenarios |
| Time horizon | Continuous across the lifecycle, revisited when the use case, data or post-market evidence changes | Bounded to an engagement or release window, repeated when the system or its model changes |
| Sign-off | Accountable risk owner and governance body accept or treat each risk, with records kept for assessors | System owner accepts findings and fix plans; results go to governance only after additional analysis, per NIST AI 600-1 |
What each approach finds and what it misses
AI risk assessment and AI red teaming answer different questions about the same system. Risk assessment decides which risks matter and who carries them, while red teaming shows how a named system actually fails, so each approach has blind spots the other covers.
Risk categories and stakeholders
Risk assessment names the categories of harm a system could cause, such as discrimination, privacy loss, unsafe advice or over-reliance, and identifies who is affected, including customers, staff and third parties. It also records the context of use that defines those harms. Check whether the assessment names the people harmed by each risk, not only the technical component that fails. Ask who owns each risk and who signs off its acceptance.
Control gaps and residual risk
A risk assessment compares the controls an organization has against the risks it identified and records what remains after those controls. This is the input for decisions to accept, mitigate, transfer or avoid a risk. The output is a decision record, not a list of exploits. Check that every residual risk has a named owner, a review date and a link to the controls that were tested or assumed.
Exploitable behavior nobody modeled
Risk assessment works from known and reasonably foreseeable risks, so it can miss a failure mode that no one thought to model. A model that follows instructions hidden in a retrieved document, or an agent that calls a tool in an unexpected sequence, may fall outside the risk register. Hands-on testing is the step that surfaces these paths. Confirm that the assessment is revisited when new tools, data sources or user groups are added.
Concrete failure modes on a named system
Red teaming produces specific evidence: a jailbreak that bypasses a safety filter, an injection path through a document upload, or an unsafe tool call that writes to a production record. Each finding ties to a version, configuration and test date. Check that the red team scope names the exact model, prompts, tools and data connections under test, because results do not transfer cleanly to a different version or deployment.
Portfolio-wide and non-technical harms
A red team test of one application rarely reveals risks that span many systems, such as a shared model provider with weak data terms or a pattern of biased outcomes across credit decisions. Harms to workers, reputation or public trust may also fall outside what an adversarial prompt can show. Ask whether the risk register covers the full portfolio of AI use cases, and who owns risks that cut across systems.
Results need analysis before governance
NIST AI 600-1 states that AI red teaming results should receive additional analysis before they feed governance and decision making, policy updates and AI risk management efforts. A raw list of successful attacks is not a risk rating. Ask the red team to classify each finding by affected harm, likelihood in context and existing mitigation, then have the risk owner decide whether the register changes.
How red team results feed the AI risk register
Treat a red team finding as an input to the risk register, not as a register entry. NIST AI 600-1 says AI red-teaming results should receive additional analysis before they are incorporated into governance and decision making, policy updates and AI risk management efforts, so the first step is triage and confirmation. Log each finding with the system version, the exact prompt, tool call or data path used, and the tester's observed output. Then confirm the finding is reproducible on the target configuration and check whether it is representative: a jailbreak that works once on a test model with a loose system prompt is a different risk from one that works across user roles and languages in production. Record the conditions under which it holds so the risk owner can judge scope.
Next, map each confirmed finding to a risk category in your own taxonomy and decide whether it belongs to an existing register entry or needs a new one. Then re-rate likelihood and impact on the scales your risk owners use, because a red team's severity view often differs from the business view of harm. Choose a treatment (accept, mitigate, transfer or avoid) and name the control owner. Set a retest trigger for each treated risk, such as a model or prompt change, a new tool or data source, or a post-market incident. Article 9(1) and (2) of the EU AI Act describes risk management as a continuous, iterative process that evaluates post-market data and adopts targeted measures, which is why the trigger matters as much as the initial rating.
Confirm with your risk owners how the register template, likelihood and impact scales, and acceptance thresholds are defined before the first red team report arrives. Ask who signs off a re-rating, how red team severity maps to your scales, and which findings must be escalated to the AI governance committee. Ask your red team provider to deliver findings in a format that carries the reproduction steps, conditions and affected system version, so triage can run without a second engagement. ISO/IEC 42001 separates AI risk assessment from AI system impact assessment, so confirm which of the two a given finding should update.
In what order to run assessment and red team testing
This sequence ties governance-level risk identification to hands-on adversarial testing, so each red team exercise targets the risks that matter most and its results change the register and controls.
Define the AI system and intended use
The system owner, with risk and compliance leads, writes down what the system does, who uses it, the data it touches, the decisions it informs and the environments it runs in. Include any tools, plugins or agent permissions, since these widen the attack surface. The step is done when the description is signed off and every later step refers to the same version of it. Illustrative example: a customer support assistant that reads order records and can issue refunds up to a set limit.
Run the risk assessment to set priorities
The risk owner identifies known and reasonably foreseeable risks for the defined system, estimates likelihood and impact, and ranks them in the risk register. Under ISO/IEC 42001, this is the AI risk assessment, which is separate from the AI system impact assessment. Use the NIST AI RMF Map and Measure functions as a structure if your organization follows it. The step closes when each risk has an owner, a rating and a proposed way to test it, and the highest-ranked risks are named for red teaming.
Set test metrics and pass thresholds in advance
Before any adversarial testing starts, the test lead and risk owner agree on the metrics, the test cases tied to each ranked risk and the thresholds that count as acceptable. EU AI Act Article 9(6) and 9(8) require high-risk AI systems to be tested against prior defined metrics and probabilistic thresholds, during development where appropriate and in any event before placing on the market. Confirm with legal counsel which obligations apply to your system. The step closes with a signed test plan that records the thresholds before any results exist.
Run red teaming on the highest-ranked risks
A red team runs adversarial role-play, red-teaming or chaos testing against the system, as NIST AI 600-1 action MP-5.1-005 describes, aimed at the top-ranked risks first. Testers record inputs, outputs, tool calls and the conditions that produced each failure. Automated scanning can widen coverage, while human testers probe context-specific misuse. The step closes when every planned test case has a logged result and a severity judgment, including tests that passed.
Analyze results before they enter governance
The risk owner and an independent reviewer examine the red team output before it changes the register or policy. NIST AI 600-1 says red-teaming results should receive additional analysis before they inform governance, policy updates and risk management decisions, so raw findings are not treated as final ratings. Compare each result against the pass thresholds set earlier and separate confirmed risks from test artifacts. The step closes with a reviewed findings summary that maps each confirmed issue to a register entry.
Feed results back into the register and controls
The risk owner updates the risk register with new or re-rated risks, assigns treatment decisions and adds or changes controls such as guardrails, access limits, human review points and monitoring. Each control is linked to the test case that motivated it. The step is done when the register and control list show the same version and every confirmed finding has an owner and a treatment decision recorded.
Retest after changes and keep the record
After any material change to the model, prompts, tools, data or intended use, the team reruns the affected test cases and reassesses the related risks, since the EU AI Act describes risk management as a continuous, iterative process across the lifecycle. Confirm with your assessor which records they expect, such as test plans, logs and decisions. The step closes when retest results meet the agreed thresholds or the change is formally rejected or rolled back.
When to choose risk assessment, red teaming or both
A new AI program usually needs governance evidence before it needs hands-on attack work. The risk assessment leads because it establishes which AI systems exist, what they are used for, who is affected, and which risks are known and reasonably foreseeable. Those identified risks become the register that later testing is measured against. Red teaming adds value only once there is a specific system or model to attack, so it usually follows the first inventory and risk rating rather than preceding it.
- Inventory first: A risk assessment starts from a list of AI use cases, owners and data sources, which many programs do not yet have.
- Governance output: The assessment produces risk ratings, owners, treatment decisions and review dates that an ISO/IEC 42001 management system can carry.
- Targeted red teaming: Once a high-impact use case is named, a focused exercise can test whether the foreseen failure modes actually occur.
- Feedback loop: Red team findings should be analyzed before they change policy or the risk register, as NIST AI 600-1 advises.
Confirm with the internal risk function which framework will own the register, and ask the competent or notified body, where one applies, whether it expects a specific form of documented evidence.
For a high-risk AI system that must be placed on the EU market, both approaches are needed, and the risk management process comes first. EU AI Act Article 9 describes a continuous, documented risk management system across the lifecycle, and Articles 9(6) and 9(8) require testing against prior defined metrics and probabilistic thresholds before placement on the market. The risk assessment defines what must be measured and what thresholds mean. Red teaming then tests whether the system breaks in ways the assessment did not anticipate, which feeds back into the risk file before release.
- Defined metrics: The assessment sets the metrics and thresholds that the pre-market tests must be measured against.
- Adversarial depth: Red teaming probes misuse and unforeseen failure modes that metric-based testing can miss.
- Documented iteration: Each red team finding should trigger a documented update to the risk identification and treatment record.
- Timing: Testing during development is expected where appropriate, and the final test must occur before the system is placed on the market.
- Evidence trail: Keep the link between each finding, the risk it updated and the release decision.
Confirm with the notified body or competent authority which conformity route applies to your system and what documentation they will review.
A deployed system that has had an incident or a major model change needs a targeted red team first, because the question is whether the change created a new exploitable behavior. Red teaming leads here since it tests the actual system as it now behaves. The risk assessment follows, reviewing whether the existing register still describes the system, because Article 9 expects the risk process to use post-market data and to adopt targeted measures. The assessment is often a short update rather than a full cycle.
- Scope the change: Test the swapped model, new tools, changed prompts or retrieval sources, not the whole program.
- Reproduce the incident: Use the incident conditions as the first test cases so the fix can be verified.
- Update the register: Revise likelihood and impact ratings and add new failure modes that the testing revealed.
- Post-market data: Feed monitoring data and user reports into the risk review, as Article 9 describes.
- Retest after fixes: Confirm that mitigations hold against the same attacks and do not shift the failure elsewhere.
Confirm with the internal risk function whether the change is material under your own change policy, and with any competent authority whether it requires notification.
An organization with many AI systems needs both approaches running on different cadences. A portfolio-level risk assessment sets the common taxonomy, risk tiers and ownership model for every system, while red teaming is prioritized by tier so that the highest-impact systems receive the deepest exercises. Without the portfolio view, red team effort tends to land on whatever system is most visible rather than most consequential. Without the testing, the register describes intent rather than behavior.
- Tiering: Use the risk assessment to rank systems, then schedule red teaming by tier.
- Shared catalog: Map findings to a common vocabulary so that results from different teams can be compared in the register.
- Separate analysis: Keep the governance review of red team results distinct from the exercise itself, as NIST AI 600-1 recommends.
- Reuse: Common attack patterns, such as those in MITRE ATLAS, can be reused across systems with system-specific scenarios added.
- Ownership: Name who owns the register, who runs exercises and who accepts residual risk.
Confirm with the internal risk function how the portfolio tiers map to any regulatory classifications, and whether the competent body expects evidence at system or program level.
What drives cost and effort for each approach
AI risk assessment and AI red teaming draw on different people and produce different artifacts, so their costs grow from different inputs. The cards below cover the drivers that move effort for each approach, who is paid for what, and which costs recur after the first engagement.
Number of systems in scope
A risk assessment under ISO/IEC 42001 clauses 6.1.2 and 8.2 covers the AI management system and each use case it governs, so effort grows with the count of AI systems and their owners. Red teaming is scoped per system, so ten systems mean ten attack plans, not one shared exercise. Ask the provider whether a quote is per system, per use case, or per program, and what is shared across them. Confirm the inventory of AI systems before you request any quote.
Model, tool and data access
A system that only answers questions needs less adversarial work than one that calls tools, reads a retrieval store and writes to business systems. Each additional access path adds attack surfaces, test accounts and environment setup for the red team. Assessment effort rises too, because the risk register must record what each tool can change. Ask for the access matrix the test will assume, and whether test identities and sandboxed data are included in the fixed scope.
Depth of adversarial testing
A quick run of known attack categories costs far less than a scenario-led exercise that builds context-specific attacks and judges harm across multiple turns. Automated runs scale cheaply in compute but still need humans to review flagged outputs. Depth also depends on the stop point: a single release gate differs from a full campaign. Ask what depth is included, how many attack scenarios are planned, and who reviews the results before they reach governance.
Domain and specialist expertise
Assessors need knowledge of the business process, the affected people and the regulatory context, such as the EU AI Act Article 9 obligations for high-risk systems. Red teamers need adversarial ML and application security skills, which are scarce and priced accordingly. Many organizations pay for both at the start, then move routine checks in-house. Ask who on the provider side holds each skill, and which of them you would need to staff internally.
Internal workshop and owner time
Risk assessments consume workshop hours from product owners, legal, privacy and operations staff, who describe intended use and foreseeable misuse. Red teaming consumes engineering time to provision environments, explain guardrails and triage findings. This internal time is often larger than the external fee and is rarely written into a quote. Ask the provider to list the stakeholder sessions and environment support they expect from your team.
Evidence, reporting and recurring work
Assessment outputs feed a risk register, documented controls and management review, which must be updated as systems change. NIST AI 600-1 says red teaming results need further analysis before they enter governance, which adds reporting work. Most of the recurring cost comes from retests after model or prompt changes and from post-market monitoring, not from the first exercise. Ask which outputs are one-time, which are refreshed, and what a retest after a model swap includes.
Questions buyers ask before hiring help for AI risk and red teaming
Can AI red teaming replace an AI risk assessment?
No. A red team test examines one system against specific adversarial scenarios, while a risk assessment identifies and evaluates risks across the use case, the population affected and the lifecycle. The two answer different questions. Red team findings are evidence that feeds the assessment, and NIST AI 600-1 says red team results should receive additional analysis before they shape governance decisions, policy updates and risk management. Without the assessment, a team can fix the attacks it tested and still miss the risks it never looked for. Confirm with any provider which artifacts they deliver: a risk register, test report, or both.
Which should we do first, the risk assessment or the red team test?
In most cases the risk assessment comes first, because it defines what the system is for, who it affects and which harms matter most. Those answers set the test objectives and the success criteria. EU AI Act Article 9 describes risk management as a continuous process across the lifecycle, and Article 9(6) and 9(8) require testing against defined metrics and probabilistic thresholds, so the metrics need to exist before testing starts. A red team test can still come earlier for a prototype if the goal is discovery rather than release evidence. Confirm the sequence against your release calendar.
What does the work need from our team?
Expect to supply several roles, whatever the provider:
- A business owner who can state the intended use, acceptable harms and who signs off on residual risk.
- Engineers who can explain the architecture, data flows, model versions, guardrails and any tools or agents the system can call.
- Someone who can grant test access to a non-production copy with realistic data controls.
- A risk or compliance lead who maps outputs to ISO/IEC 42001 clauses 6.1.2 and 8.2 and to the Article 9 file.
If these roles are missing, timelines slip more often than the testing itself. Confirm the staffing assumptions in the written scope.
Can we run an AI risk assessment ourselves?
Often yes for the governance layer, if you have people who understand the system and the regulatory context. ISO/IEC 42001 asks for a defined AI risk assessment process and for it to be applied and kept current, which an internal team can run with proper documentation. Adversarial testing is harder to run in-house without specialist skills in attack techniques and in judging harmful outputs. Many organizations split the work: internal teams own the risk register and decisions, and an external team tests the system. DeployOpen plans and documents this kind of work under a fixed scope agreed in writing, but whether it suits your team is a decision to make with your own risk owner.
How often should we retest an AI system?
Retest when something material changes, and set a regular cadence on top of that. Triggers include a new model or model version, a changed system prompt or retrieval source, new tools or agent permissions, a new user population, and new incidents in post-market monitoring. Article 9(1) and 9(2) of the EU AI Act treat risk management as continuous, so periodic review is expected. Ask the provider how they handle regression tests: which attacks are kept as a fixed suite and which are rewritten each cycle. The right frequency depends on how often your system changes, which your engineers can estimate better than any general rule.
What should we ask for in a quote?
Ask for the written scope before the price. A useful quote lists:
- The system boundary, including models, tools, data sources and any agents.
- The objectives and the harm categories tested, with the reference frameworks used.
- The deliverables: risk register entries, a test report, retest criteria and evidence files.
- The test environment, access method and data handling rules.
- Who performs each activity, and what requires your staff.
- What is one-time and what recurs, such as retainers for retesting.
Prices depend on scope size and number of systems, so compare quotes on identical scope. DeployOpen does not publish a price list and issues quotes from the written scope.
How do the outputs map to ISO/IEC 42001 and EU AI Act Article 9?
The risk assessment output maps most directly to ISO/IEC 42001 clauses 6.1.2 and 8.2, which cover AI risk assessment, and it should be kept separate from the AI system impact assessment in clauses 6.1.4 and 8.4. Red team results map to the risk identification and treatment steps, and to the evidence that testing covered identified risks. For Article 9, the file should show identified risks, evaluation, the measures adopted, testing against prior metrics and post-market review. This mapping is preparation for an assessment, not a certification. Auditors and notified bodies decide conformity, so confirm with them which evidence they expect.
Which standards and laws call for each
The first decision is which obligation you are actually answering, because the standards treat AI risk differently. ISO/IEC 42001:2023 puts AI risk assessment in clauses 6.1.2 and 8.2, and it treats AI system impact assessment as a separate exercise in clauses 6.1.4 and 8.4. A risk assessment asks what could go wrong for the organization and how likely and severe each outcome is. An impact assessment asks what the system could do to people and groups affected by it. Your risk register should cover the first, and your impact assessment should cover the second. Confirm the exact wording and the scope of each clause against the standard itself, and ask your certification body how it expects the two records to be linked before you build either one.
For high-risk AI systems under the *EU AI Act*, Article 9 is the controlling text. Article 9(1) and (2) require a documented risk management system that runs as a continuous, iterative process across the lifecycle, covering identification and estimation of known and reasonably foreseeable risks and the adoption of targeted measures. Article 9(6) and 9(8) add testing against predefined metrics and probabilistic thresholds, before the system is placed on the market. Red team findings can serve as test evidence here, but the article does not name red teaming. Check with counsel whether your system is high-risk and which conformity route applies. The EU AI Act Article 9 text is the natural starting point for that review.
NIST AI 600-1 is the place where red teaming is named directly. Under MAP 5.1, action MP-5.1-005 calls for adversarial role-playing, red-teaming or chaos testing to find unforeseen failure modes. The same profile says red team results need further analysis before they go into governance decisions, policy updates or risk management. DeployOpen prepares test plans, evidence packs and control documentation for these obligations, but it does not issue certifications, attestations or audit reports. Auditors and certification bodies decide whether your management system conforms, and regulators decide whether your AI system meets Article 9. Ask each of them early which evidence they will accept and in what form.
How DeployOpen scopes, delivers and supports AI security testing
DeployOpen runs AI security testing as a fixed-scope engagement inside your own infrastructure, so governance-level risk assessment and hands-on adversarial testing can be planned as separate, connected work. Scope, deliverables and timelines are agreed in writing before any testing starts.
How we scope
- Written scope states which AI system, models, data flows and risk owners the assessment covers.
- Scope separates the governance risk assessment from the adversarial test of a named system.
- Rules of engagement set permitted attack types, test data, environments and escalation contacts before testing begins.
How we build
- Test environments are set up inside your cloud account, data center or air-gapped network.
- Risk register entries and red team test cases are mapped to the same identified AI risks.
- Adversarial testing combines scripted attack sets with expert-crafted scenarios, and findings are reviewed before scoring.
How we hand over
- Deliverables include the risk assessment record, test plan, evidence of executed attacks and findings.
- Red team results receive additional analysis before they are proposed for risk register or policy updates.
- Documentation explains which controls were tested, what passed, what failed and what needs retesting.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retesting after model, prompt or tooling changes can be scoped as a follow-on engagement.
- Ask us to confirm which compliance evidence we prepare and which independent auditors or certification bodies must issue it.
