Does the EU AI Act require red teaming for every AI system?
The AI Act does not require red teaming by name for every AI system. What it does require depends on the system's risk class and on whether your organization is a provider or a deployer. Three obligations matter most for testing: pre-market testing against defined metrics under Article 9, resilience against attacks such as data poisoning, adversarial examples and model flaws under Article 15(5), and documented adversarial testing, including red teaming, for providers of general-purpose models with systemic risk under Article 55(1) and Annex XI Section 2. If a customer or contract asks you to prove testing, start by confirming which of these your system triggers, because the evidence you need follows from that answer.
Article 9 asks high-risk providers to test against prior defined metrics and probabilistic thresholds suited to the intended purpose, during development where appropriate and in any event before the system is placed on the market. Article 15(5) asks for measures against attempts to alter use, outputs or performance by exploiting vulnerabilities. The Act names the attack classes but leaves the method open, so a test plan that covers data poisoning, evasion and confidentiality attacks against your own system is a defensible starting point. Note that Article 42(3), added in 2026, can deem some high-risk systems covered by the Cyber Resilience Act to meet the Article 15 cybersecurity requirements, so confirm with counsel whether that route applies to your product before you design the test program.
The organization owns these controls and the test evidence, and an implementation partner can help design the test plan, run the adversarial tests and assemble the records. Only an independent assessor or notified body issues conformity judgments where one is required, and most Annex III systems use internal control instead. For systemic-risk models, the Act's documentation duties make adversarial testing a formal record, and the NIST AI RMF and NIST AI 600-1 describe red teaming as a practice you can adopt voluntarily to find failure modes, which is useful even where the Act does not name it.
How the rules differ for systemic-risk general-purpose models
Red teaming is not a general requirement for every AI system, but it becomes an explicit, documented duty for providers of general-purpose AI models with systemic risk. The cards below separate what Article 55(1) and Annex XI require from what applies to other providers and who supervises each duty.
Standardized state-of-the-art evaluation
Article 55(1) requires providers of models with systemic risk to evaluate their models using standardized, state-of-the-art protocols. This duty is in addition to the baseline general-purpose model obligations in Articles 53 and 54. Check whether your evaluation protocols are named, versioned and repeatable, and whether they are chosen against the current state of the art rather than a fixed internal checklist.
Documented adversarial testing
Within the evaluation duty, providers must carry out and document adversarial testing, which includes red teaming. The test is expected to probe the model for failures that a normal evaluation would miss. Record the attack scenarios, the testers, the results and the mitigations that followed, so the evidence can be produced for the AI Office on request.
Internal and external testing records
Annex XI Section 2 asks for documentation of any internal or external adversarial testing, such as red teaming, alongside the evaluation strategies and results. Independent testers count as external testing, and their scope, findings and remediation must be recorded in the technical documentation. Confirm which tests your organization runs in-house and which are commissioned from a third party, because both must appear in the file.
Model adaptations and architecture records
Annex XI also requires documentation of model adaptations, including alignment and fine-tuning, and of the system architecture. Adversarial test results only mean something against a known model state, so each test should be tied to a specific model version and its training changes. Ask engineering teams to link every red-team round to the adaptation it assessed.
Union-level systemic risk assessment
Article 55(1) also requires providers to assess and mitigate systemic risks at Union level. This goes beyond the risks to one customer or one deployment, and it asks what the model could do at scale across the EU. Confirm that your risk assessment covers misuse, downstream dissemination and foreseeable failure modes that adversarial testing has surfaced, and that mitigations are recorded against each identified risk.
Code of practice and AI Office supervision
The Commission states that the Safety and Security chapter of the General-Purpose AI Code of Practice applies only to providers of models with systemic risk under Article 55. The AI Office supervises these rules for general-purpose models. Other providers of general-purpose models should not assume the same red-teaming chapter applies to them, but should confirm their own obligations before relying on the code as evidence.
What NIST guidance says about red teaming
The NIST AI RMF Playbook is a companion resource to the AI RMF 1.0 (NIST AI 100-1). Under MEASURE 2.7, which concerns the security and resilience of AI systems, the Playbook suggests a set of actions rather than a fixed test. Organizations read these as options to adapt to their own systems and risk profile.
- Track security test results and metrics: The Playbook suggests tracking red-team results, anomalous event rates, downtime and time to bypass, so that security performance can be measured over time.
- Run red-team exercises under adversarial conditions: The suggested actions include testing the system as an attacker would, under realistic misuse conditions, rather than only under normal inputs.
- Apply countermeasures: Examples include authentication, throttling and differential privacy, selected according to the threats found in testing.
- Share attack patterns: The Playbook suggests sharing attack patterns with incident databases so other organizations can learn from them.
- Require security audits of third-party AI resources: Where a model or component comes from a supplier, the Playbook suggests requiring a security audit of that resource.
Before relying on these actions, confirm the current wording and numbering in the NIST AI RMF Playbook, since it is updated outside the AI Act process. Check whether your metrics can be produced from your own logs and test records.
NIST AI 600-1 is the Generative AI Profile for the AI RMF. Its MAP 5.1 actions address likely impacts of generative AI, including harms that a model or system could cause in ways its designers did not expect. Two actions, MP-5.1-002 and MP-5.1-005, bear directly on adversarial testing of generative systems.
- Content provenance harms (MP-5.1-002): The profile asks organizations to identify harms such as deepfakes or tampered content, then rank them by likelihood and impact. The ranking gives testing a priority order.
- Adversarial role-playing and red teaming (MP-5.1-005): The profile suggests conducting adversarial role-playing, red-teaming or chaos testing to find failure modes that were not anticipated in design.
- Unforeseen failure modes: The emphasis is on discovering failures outside the expected use, such as prompt injection or misuse by users who try to push the system beyond its intended purpose.
- Link to the risk register: Results should feed the same risk register that records likelihood, impact and the chosen treatment, so each finding has an owner and a decision.
- Scope to your deployment: A tester working on a self-hosted model needs different scenarios from one testing a vendor-hosted chat interface, because the attack surface and logging differ.
Confirm the exact action text in the current NIST AI 600-1 document before citing it in evidence, and record which actions you adopted and why.
NIST's AI RMF and its Playbook are voluntary guidance. They have no certification, and neither NIST nor the Playbook is a legal test under the EU AI Act. The AI Act sets its own obligations, such as the testing against defined metrics in Article 9 and the adversarial resilience expectations in Article 15, and it does not adopt NIST actions as a checklist.
- Different legal basis: Compliance is judged against the Act's articles and any harmonised standards that apply, not against NIST action numbers.
- NIST as a method source: Organizations can use NIST actions to design tests and evidence that support their AI Act file, but the mapping is their own engineering judgment.
- Assessor expectations: Market surveillance authorities and notified bodies will ask whether testing matched the intended purpose and produced records. They will not ask for a NIST Playbook number.
- Documentation of choices: Record which NIST actions you used, what you skipped and why, so the reasoning can be reviewed.
- Check for changes: Harmonised standards and Commission guidance may change the expected evidence, so confirm current requirements with the AI Office, the relevant national authority or your legal counsel.
Use NIST guidance to shape your test plan, then confirm the legal requirements for your system with qualified counsel before you describe your testing as AI Act compliant.
How an assessor tests adversarial testing evidence
The sequence builds a traceable path from the intended purpose to the test record, so an assessor can see that adversarial testing was planned against defined metrics, executed, acted on and repeated after changes. Each step produces a document the assessor can check against the next one.
Define intended purpose and test metrics
The provider documents the intended purpose of the system, the users and the conditions of use, then sets the metrics and probabilistic thresholds that testing must meet. Article 9 expects these to be defined before testing starts, so the assessor can compare results against criteria fixed in advance. The output is a signed metrics specification tied to the risk management file. The step is done when every metric names a threshold, a measurement method and the owner who accepts a failure.
Scope adversarial tests against identified threats
The security or AI engineering lead maps the threats that apply to this system: data poisoning, model poisoning, adversarial examples, model evasion, confidentiality attacks and flaws in the model itself. Each threat is paired with a test method, whether that is a red-team exercise, a scripted probe suite or a review of training data sources. The output is a test plan that lists what is in and out of scope and why. Your reviewer should be able to trace every threat in the risk file to at least one test case.
Prepare the test environment and tooling
The team sets up a test environment that matches the production configuration as closely as is practical, with the same model version, data pipelines and access controls. Tool versions, prompts, seeds and configuration are recorded so a later run can be reproduced. The output is an environment record and a tool inventory with versions. Confirm with your tool vendors which versions and licenses apply before you commit results to the file.
Execute tests and log every attempt
Testers run the planned cases, usually with a mix of manual red teaming and automated probes, and log every attempt, not only the ones that succeed. For generative systems, a probe tool such as garak writes a JSONL report of all attempts and evaluations and a separate hit log of attempts that found a vulnerability. The output is a complete, time-stamped test log that a reviewer can sample. An assessor will ask whether failed or abandoned attempts are recorded, so the log must be complete.
Record results and assign mitigations
Findings are compared against the metrics set in step one and each gap is ranked by likelihood and impact. The team records the mitigation chosen for each finding, such as authentication, throttling, input filtering, data changes or a documented residual risk accepted by a named owner. The output is a results report with a finding register that links each finding to its mitigation and owner. The step is complete when no finding is open without a decision.
Retest after material changes
The team reruns the relevant test cases after the model, training data, configuration or deployment environment changes in a way that could alter behavior or risk. Each retest is compared with the earlier baseline so the change in results is visible. The output is a retest record that names the change that triggered it and shows the before and after results. Ask your assessor early which changes they expect to trigger a retest, since the answer depends on your change control process.
Assemble the conformity file
The provider collects the test plan, metrics specification, environment record, logs, results report, mitigation decisions and retest records into the technical documentation that supports the conformity assessment. For general-purpose models with systemic risk, Annex XI also expects documentation of internal or external adversarial testing. The output is an indexed evidence package with a short narrative that links each threat to its test and its outcome. An independent assessor or notified body decides whether the file is sufficient, so the organization should confirm scope with them before finalizing it.
Which open-source tools produce red-team evidence
Open-source tools can generate the attack runs and logs that an assessor will ask to see when testing the AI Act's adversarial and robustness duties. They produce output, not verdicts. Whether that output shows compliance depends on the metrics, thresholds and review process your organization defines and can explain.
garak for language model probing
garak is an Apache-2.0 scanner that probes language models for prompt injection, jailbreaks, data leakage, hallucination, misinformation and toxic generation. It fits systems built on a general-purpose model, where Article 15(5) attacks such as prompt manipulation are most likely. Confirm which probe families you run, pin the version, and record which probes were excluded and why.
JSONL attempt report
garak writes a JSONL report that records every attempt and its evaluation, not only the failures. That completeness matters because an assessor can check whether the test set was run in full rather than curated after the fact. Store reports with the model version, configuration and date so each run can be reproduced against the same system state.
Hit log of successful attacks
The hit log lists the attempts that found a vulnerability. It is the starting point for the Article 9 risk management record, because each hit needs a documented decision: fix, mitigate, accept with rationale, or retest. Define who triages hits and how long a hit may stay open before escalation.
Metrics and thresholds you define
Article 9 requires testing against prior defined metrics and probabilistic thresholds suited to the intended purpose. A tool reports rates and hits but does not set an acceptable failure rate for your use case. Write the metric, the threshold, the sample size and the approver before the first run, and keep that document as evidence of pre-market testing.
Classifier robustness checks
For non-language models, such as classifiers used in high-risk decisions, ask the team which robustness test methods they use against adversarial examples and evasion. Confirm the tool or method, its inputs and its outputs, and whether results can be compared across model versions. Do not assume a language-model scanner covers these models.
Data poisoning checks
Article 15(5) names data poisoning and model poisoning. Confirm with your engineering team how training and retrieval data are checked for tampering, which provenance records exist, and what tests show that poisoned inputs change behavior. Treat these as categories to confirm, since no specific tool is established here for this purpose.
Common questions about AI Act adversarial testing
Does the EU AI Act require red teaming for every AI system?
No. The AI Act does not name red teaming as a mandatory activity for all AI systems. It requires testing against defined metrics before a system is placed on the market (Article 9), and resilience against attacks such as data poisoning, adversarial examples and model evasion (Article 15(5)). Red teaming becomes explicitly expected in one place: providers of general-purpose AI models with systemic risk must evaluate models using standardized protocols, including documented adversarial testing (Article 55(1)). Outside that case, the Act leaves the choice of test method to the provider, so the question for your system is which tests show that Articles 9 and 15 are met for its intended purpose.
Who is responsible for AI Act adversarial testing, the organization or a partner?
The organization that holds the obligation owns the testing and the conclusions drawn from it. For a high-risk system, that is usually the provider. A specialist partner can plan, execute and document tests, including adversarial tests against a self-hosted model or its surrounding application, and hand over reports and raw results. The partner does not take on the compliance duty, and no partner can issue an assessment result. The organization must review the findings, decide on mitigations, and keep the technical documentation that shows what was tested and why. Ask any partner to state in writing which deliverables it produces and which decisions remain with your team.
Must an external red team test our AI system under the AI Act?
The AI Act does not say that an external red team is required. It requires testing and resilience measures, and it does not specify who must perform them. Internal teams can test, provided the tests are documented, repeatable and matched to the intended purpose. Whether an independent party is needed for a given system is a question to put to your conformity assessment route and your notified body or market surveillance contacts, if applicable. Confirm this with your assessor before you commit budget to an external engagement, and record the decision in your risk management file.
What fines apply if an AI system fails adversarial testing requirements?
Article 99 sets maximum fines by category. Breaches of prohibited practices carry up to EUR 35 million or 7% of worldwide annual turnover. Breaches of most operator obligations, which include the Article 9 and Article 15 duties, carry up to EUR 15 million or 3%. Supplying misleading information to authorities carries up to EUR 7.5 million or 1%. In each case the higher figure applies. These are ceilings, not expected amounts, and the actual penalty depends on the authority's assessment of the breach, its severity and cooperation. Testing failures usually matter most in the way they surface during market surveillance, so keep test records that show what was checked and when.
What does adversarial testing ask of our internal engineering and security staff?
Adversarial testing needs people who know the system well enough to define realistic attack scenarios and who can read the results. In practice, engineers need to document the intended purpose, the data pipeline, the model versions and the deployment context, because these set the test metrics and thresholds. Security staff need to agree on attack classes such as poisoning, evasion, confidentiality attacks and prompt-based manipulation, and to decide who triages findings. Product and compliance owners sign off on acceptance criteria. Plan time for these internal reviews, since they often take longer than the testing itself. Ask a partner to show how its test plan maps back to your documentation before work begins.
Can an open-source testing tool alone satisfy Article 15 of the AI Act?
No. A tool produces evidence, but Article 15 concerns the system's resilience and your documented approach to it. For language models, an open-source probe such as garak (Apache-2.0) can test for prompt injection, jailbreaks, data leakage and similar weaknesses, and it writes a JSONL report of every attempt and a hit log of successful ones. Those outputs help show what was tested. They do not show that the test scope, thresholds, mitigations and monitoring are adequate for the intended purpose. Treat tools as one input to a documented testing program, and confirm with your assessor how their outputs will be evaluated.
Who owns the testing, and who signs off
The organization owns the test plan, the metrics and the records. The AI Act places the testing duty on the provider of the system, and that duty cannot be handed to a vendor. Article 9 expects the provider to test against defined metrics and probabilistic thresholds suited to the intended purpose before the system is placed on the market, and Article 15 expects resilience against attacks on the system. Someone inside the organization must decide which attacks matter for the intended purpose, which thresholds count as a pass, who approves a failed result and what happens next. An implementation partner such as DeployOpen can plan the security tests, run them against the system and prepare the evidence, but the choice of metrics and the acceptance decisions stay with the provider's accountable owner, usually the team that holds the technical documentation.
Independent parties issue conformity outcomes, not the testers. A penetration test report or a red team debrief is internal evidence. It does not certify compliance. Most Annex III high-risk systems use the internal-control route in Annex VI, so the provider's own declaration of conformity rests on its evidence, while notified bodies are involved only for some biometric systems and Annex I products under Article 43. Confirm with your notified body or national authority which route applies to your system before you commission work. Your readiness file should show who performed each test, who reviewed the results and who accepted the residual risk.
Enforcement sits with national authorities and the AI Office. National market surveillance authorities enforce the Act for AI systems. The AI Office enforces the general-purpose AI model rules under Article 88, and under Article 75(1) as amended in 2026 it is exclusively competent for AI systems built on a provider's own general-purpose model and for AI in very large online platforms and search engines. Ask your counsel which authority would supervise your system, because that answer decides which records an inspector will request and in what form.
How DeployOpen tests AI systems and prepares the evidence
DeployOpen runs security testing for AI systems and agents, and prepares the test records that an AI Act file needs to show adversarial testing was planned, run and acted on. The organization owns its risk decisions and its conformity position, and any independent assessor or notified body issues conclusions on compliance.
How we scope
- We agree in writing which AI systems, models, endpoints and data flows are in scope before testing starts.
- The written scope names the attack classes to test, such as prompt injection, data poisoning paths and adversarial inputs, and the metrics each one is measured against.
- Engagements are fixed-scope, and ongoing testing and support can be added as a retainer once the initial scope is agreed.
How we build
- We set up the test environment inside your own infrastructure so model outputs, logs and findings stay under your control.
- We run adversarial test suites and red-team exercises against the defined metrics and record each attempt, result and threshold decision.
- Where it suits the system, we use open-source probing tools such as garak and keep their raw reports as primary evidence.
How we hand over
- You receive a test plan, the raw test logs, a findings register with severity and remediation status, and a retest record for each fix.
- The documentation maps each result to the relevant Article 9 and Article 15 testing evidence so your assessor can trace it directly.
- We walk your team through the results and confirm which items remain open, so accountability for residual risk stays with your organization.
How we support
- We help you decide which tests must be repeated after a model update, fine-tune or change of intended purpose.
- Retainers cover re-testing on a schedule you set, plus updates to the test suite as attack methods change.
- Before any external assessment, confirm with your assessor which artifacts they expect and whether they want the raw logs or a summary.
