AI red teaming vs LLM penetration testing

AI red teaming or LLM penetration testing: which one to commission first

Choose AI red teaming when the main risk is what the model says or does under adversarial pressure, such as harmful output, policy bypass, or manipulation of its behavior. Choose LLM penetration testing when the main risk is how the application, its APIs and its data stores can be abused. Most teams with an LLM feature in production or about to launch have both risks, so the first decision is which one is more likely to cause harm in the first release, and which one the customer or regulator is asking you to evidence. Ask the vendor or internal owner to write down the top three harms for the feature before you pick a provider, because the answer usually points to one test.

Plan to combine them when the feature has both a behavioral surface and a conventional attack surface, which is typical when the model can call tools, read retrieval stores or act under a user's identity. NIST SP 800-115 notes that organizations often combine techniques to reach an in-depth assessment at an acceptable level of risk, and that principle applies here. A practical split is to commission the penetration test first to cover the web, API, authorization and data layers, then run AI red teaming against the model and prompt paths, with results from each feeding a single risk register. Confirm with the provider how each engagement will be scoped, who will hold the rules of engagement, and how findings from one test will be retested in the other.

Before you sign, confirm three things in writing: the named scope (model behavior, application layers, or both), the team's documented expertise in adversarial ML and application security, and the output format, since NIST AI 600-1 says red-team results need further analysis before they inform governance and risk management. Also check whether the EU AI Act obligation in Article 55(1) applies to you, since it targets providers of general-purpose AI models with systemic risk and does not by itself settle which test a customer will request. Customer security questionnaires usually ask for an application penetration test report, so an AI red teaming report alone may not answer that request.

How AI red teaming and LLM penetration testing differ side by side

This table compares the two approaches on goal, scope, technique catalog, team, success criteria and output. Read each row as a question to put to a provider: the rows show which engagement answers your question, and the last rows show how much of each test a human must drive.

Dimension	AI red teaming	LLM application penetration testing
Primary goal	Find ways to make the model behave harmfully or outside its intended use, including misuse that helps attack other systems	Find exploitable weaknesses in the LLM application and the web, API and data layers around it
Primary question	Can the model, or the system built on it, be pushed into unsafe output or actions?	Can an attacker reach data, functions or other users through the application?
Target boundary	The model and its deployed behavior, including prompts, guardrails and content policies	The full application: front end, APIs, identity, retrieval sources, tool connectors and hosting, as agreed in scope
Technique catalog	MITRE ATLAS techniques such as LLM Prompt Injection: Indirect (AML.T0051.001) and LLM Jailbreak (AML.T0054), with NIST AI 600-1 as the risk profile	OWASP LLM Top 10 2026 for application risks, with OWASP WSTG and NIST SP 800-115 for test method and structure
Team make-up	Diverse members with domain and harm expertise, as NIST AI 600-1 recommends, plus people who can judge output severity	Application penetration testers with web, API and cloud skills, plus someone who understands LLM-specific attack paths
Success criteria	Reproducible harmful outputs or unsafe behavior, rated by severity and tied to a stated harm	Validated exploit paths with evidence of impact, rated against the affected asset and the data it exposes
Reporting and output	Prompts or scenarios, model responses, harm analysis and guidance for governance, which NIST AI 600-1 says needs further analysis before use	Findings with reproduction steps, impact and remediation, in the style of a conventional penetration test report
Manual-led	Largely human-driven, with automated tools used to widen coverage. Confirm the split for each engagement	PCI SSC guidance describes penetration testing as essentially manual and says running an automated tool alone is not enough. Confirm how much of the engagement is human-driven
Typical buyer question	Does the model resist misuse and harmful generation before release or after a material change?	Does the deployed application meet the security bar we set for its data and users?

What each approach finds and what it misses

The six risk areas below overlap in practice, but they fail differently under each approach. AI red teaming probes model behavior and harm at scale, while LLM penetration testing follows the application, API and data path around the model. Read each card for the approach that usually surfaces the issue and the gap that the other approach tends to leave open.

Prompt injection and jailbreaks

Direct and indirect prompt injection, plus jailbreaks that drop safety behavior, are the core of OWASP LLM01 and map to ATLAS techniques such as LLM Prompt Injection: Indirect (AML.T0051.001) and LLM Jailbreak (AML.T0054). AI red teaming usually surfaces these first because it generates many attack variants and judges whether the model complied. LLM penetration testing tends to miss jailbreak variety but catches injection paths through documents, emails or tool results that the application ingests. Ask the team to test both direct prompts and attacker-controlled content the app reads.

Harmful or misleading outputs

Harmful content, dangerous instructions and confident but wrong answers are harms that depend on context and judgment. NIST AI 600-1 notes that red-team output quality depends on the team's background and expertise, and that results need further analysis before they feed governance. AI red teaming with domain experts usually finds these, while a conventional penetration test rarely scores output quality. Check whether the scope defines harm categories and who decides what counts as a failure.

Excessive agency in tool use

When a model can call tools, send email or change records, the risk moves from what it says to what it does. The 2026 OWASP LLM Top 10 moves Excessive Agency up to LLM03, which makes it a priority for scoping. A penetration tester usually finds over-broad tool permissions, missing confirmation steps and service accounts with too much reach, because these are authorization problems. AI red teaming may show that a model will attempt an unsafe action but not whether the backend enforces the limit. Test both the model's intent and the enforced permissions.

Insecure output handling and API flaws

Model output rendered into HTML, SQL, shell commands or downstream APIs can create classic injection and access control flaws, and the surrounding API can expose object-level authorization gaps. LLM penetration testing usually finds these because it applies web and API techniques to the surrounding layers. AI red teaming tends to miss them because it focuses on the model's responses rather than how the application parses and trusts them. Confirm that the test covers the rendering path, the API endpoints behind the chat interface and session handling.

Data exposure through retrieval or context

Retrieval pipelines and system prompts can leak documents, credentials or other users' data. The 2026 OWASP LLM Top 10 adds Hidden Context Exposure at LLM08, which covers this category. LLM penetration testing usually finds weak tenant isolation in the retrieval store and overly broad index permissions, while AI red teaming may extract hidden instructions through repeated questioning but may not check every data source. Ask for test accounts in each role and a review of what the retriever can read on a user's behalf.

Gaps in model-level safeguards

Safety training, content filters and refusal behavior can differ across model versions, languages and fine-tunes. NIST AI 600-1 action MS-2.7-007 calls for red teaming that assesses resilience to abuse that facilitates attacks on other systems, such as malicious code generation or phishing content. AI red teaming usually surfaces these gaps because it measures model behavior across many prompts. LLM penetration testing typically treats the model as a black box and misses them. Ask whether the vendor's safety evaluations are repeated after any model or prompt change.

How scope changes for a chatbot, a retrieval app and an agent

A text-only assistant answers questions and generates content without retrieving private data or calling tools. Most of the risk sits in the model's behavior and in how the application wraps it. The AI red team scope centers on jailbreaks, harmful or policy-violating output, misuse such as generating phishing text or malicious code, and abuse that could help attack other systems, which NIST AI 600-1 asks testers to consider under action MS-2.7-007. The penetration test scope is narrower but still real: user authentication, session handling, rate limits, the chat API, logging of prompts that may contain personal data, and the secrets and hosting stack behind the front end.

Model behavior: test whether safety behavior holds across languages, personas and multi-turn conversations.
Misuse and abuse: check whether the assistant produces content that would help an attacker, such as phishing drafts.
API and session layer: test authentication, authorization of conversation history, and abuse controls on the chat endpoint.
Logs and stored prompts: confirm where prompts and completions are kept, who can read them, and how long they persist.
Hosting and secrets: review API keys for the model provider, container configuration and network exposure.

Before testing, confirm with the team whether the assistant is connected to any data source or tool, because that answer changes which tab applies next.

How an engagement runs from scoping to retest

The sequence turns a broad question about an AI system into agreed rules, targeted attack objectives, evidence-backed findings and a retest that shows whether fixes held. Each step closes with an output the client can check before the next one starts.

Agree rules of engagement
The client and the testing team agree written rules of engagement before any testing begins. NIST SP 800-115 provides a rules of engagement template in Appendix B, and teams often adapt it to cover model endpoints, prompt logs, test accounts, rate limits, data handling and who can stop the test. The step closes when both parties have signed the rules and named an escalation contact who can pause testing.
Define harm categories and attack objectives
The team and the system owner list the harms that matter for this system, such as data disclosure, unsafe actions, misinformation or abuse that helps an attacker reach other systems. NIST AI 600-1 action MS-2.7-007 frames this kind of abuse, including prompt injection and malicious content generation. The step closes with a written list of objectives, each tied to a business impact and an owner who will review results.
Build the test plan from checklists or scenarios
The team turns the objectives into a plan built from checklists, scenarios or both. Checklists give repeatable coverage of known weakness classes, such as OWASP LLM Top 10 entries or ATLAS techniques, while scenarios follow a realistic user or attacker path through the application, its data sources and its tools. The step closes when the plan lists which web, API and data layer tests run alongside the model-focused tests, and the client has approved it.
Run testing with human judgment
Testers run the planned attacks against the agreed environment and record every prompt, response and system action. Automated tools can generate volume, but a skilled tester decides which lines of attack to pursue and judges whether an output is harmful in context. The step closes when each objective has recorded evidence or a documented reason it could not be tested.
Validate and analyze findings
Testers reproduce each candidate finding, confirm the conditions that trigger it and remove duplicates or one-off model variance. NIST AI 600-1 says red-team outputs need additional analysis before they feed governance and risk management, so the team also rates impact and notes which results depend on the team's own background and expertise. The step closes with a validated list where every finding has a reproduction path and a severity rationale.
Report findings and remediation
The team writes a report that lets engineers reproduce each issue and lets risk owners decide on treatment. It should separate model behavior findings from conventional web, API and data layer weaknesses, state the test scope and limits, and recommend fixes such as input and output controls, tool permission changes or data access reductions. The step closes when the client confirms receipt and the findings are entered into its tracking system.
Retest fixes and close the engagement
After the client applies fixes, the team reruns the reproduction steps for each closed finding and a sample of the adjacent attacks to check for regressions. The retest result is recorded as fixed, partially fixed or not fixed, with evidence attached. The step closes when every finding has a retest status and the client has accepted the final report, including any risks it chose to accept in writing.

What you receive from each engagement

This table compares the artifacts an AI red teaming engagement and an LLM application penetration test typically produce, and which items you must request in writing. Use it to check each proposal against the same list before signing, because the two engagement types often use different labels for similar work.

Deliverable	AI red teaming	LLM application penetration test
Scope and rules of engagement	Defines model behaviors, harm categories, prohibited content and permitted attack methods, often with the provider's model access level. Ask for written limits on what may be generated and stored.	Defines hosts, endpoints, accounts, data stores and the LLM integration points. Ask for explicit inclusion of prompt pipelines, tool calls and retrieval sources.
Test plan	Objectives tied to specific harms or misuse cases, with the adversarial techniques and team roles described. Confirm whether the plan references NIST AI 600-1 actions.	Methodology mapping to PTES phases or OWASP WSTG sections, plus authentication levels and test windows. Confirm whether the plan covers both the web layer and the model layer.
Attack scenarios or checklist coverage	Scenario narratives such as jailbreak or indirect prompt injection attempts, each with an objective and a success condition. Ask how many scenarios were attempted and how coverage was judged.	Checklist-based coverage of the web, API and data layers, with LLM-specific cases added. Ask for a coverage list showing which checks were run and which were skipped.
Evidence of reproduced results	Prompts, model responses, conversation logs and the number of attempts per scenario. Ask whether results were reproduced on a second run, since outputs can vary.	Request, response and proof-of-exploit records for each finding, with steps to reproduce. Ask for captured credentials or data handling notes.
Severity and harm rating method	Harm rating based on the nature, severity and reach of the output, judged by people with domain expertise. Ask for the written rubric and who signed off on each rating.	Severity rating tied to exploitability and business impact, often expressed with CVSS or the client's own scale. Ask which scoring version is used and how LLM-specific findings are scored.
Mapping to OWASP LLM Top 10 2026 and MITRE ATLAS	Findings mapped to ATLAS technique IDs such as AML.T0051.001 (indirect prompt injection) and AML.T0054 (jailbreak), with LLM Top 10 entries where relevant. Confirm the mapping version used.	Findings mapped to OWASP LLM Top 10 2026 entries and, for the surrounding stack, to OWASP WSTG or PTES sections. Ask whether ATLAS IDs are included for model-level findings.
Remediation guidance	Guidance on system prompts, filters, output handling, model choice and monitoring, with the trade-off each option carries. Ask whether fixes are validated against the original scenario.	Fix guidance for code, configuration and architecture, including input handling for LLM tool calls and least privilege for connected data. Ask for prioritized fixes with owners.
Retest scope and governance summary	Retest of the same scenarios after fixes, plus an executive summary for AI governance and risk owners. Confirm whether results are routed to the risk register.	Retest of confirmed findings and an executive summary for security leadership. Ask whether the retest window is in the written scope and whether governance material is included.

What drives cost and effort for each test

Both engagements are priced by the same underlying variables: how much surface is in scope, how much judgment the team must apply, and how often the work repeats. The cards below separate what you pay once from what recurs, so you can compare written scopes rather than headline numbers. DeployOpen does not publish prices, and a quote follows from the written scope.

Number of models and endpoints

Each model, model version, and inference endpoint is a separate target with its own configuration, guardrails, and rate limits. An AI red team test has to cover each one that users or agents can reach, while an LLM penetration test also covers the API gateway and routing in front of them. Ask the quote to list every model and endpoint by name, version, and hosting location, including any fallback model the application switches to under load.

Tools and data sources the system reaches

Every tool, plugin, database, and retrieval index the application can call expands the attack paths that must be tested, including indirect prompt injection through documents the system reads. Read access to a document store is a different test from write access to a ticketing system. Ask for an inventory of each connector with its permissions, and state whether tests may write to live data or only to seeded test records.

Domain expertise for harm judgment

Deciding whether a model output is harmful, misleading, or a policy breach often needs people who know the domain, such as clinicians for medical advice or credit specialists for lending decisions. NIST AI 600-1 says red-team output quality depends on the team's background and expertise, and that results need further analysis before governance uses them. Judgment takes senior reviewer time, so ask who reviews outputs, what their qualifications are, and how disagreements are resolved.

Harm categories in scope

Each harm category, such as data disclosure, unsafe content, bias, or abuse that helps attackers build malicious code or phishing content, needs its own test design, prompt set, and success criteria. Broad categories multiply the work because each needs review by a person. Ask the quote to name the categories in writing and to say which ones are excluded, so that both parties agree what a passing result means.

Staging versus production access

Testing in production exposes real customers and data to adversarial inputs, and it often needs change windows, monitoring coordination, and incident handling agreed in advance. A staging copy that mirrors the production model, prompts, retrieval data, and tool permissions costs more to build and keep current, but it lets the team test more aggressively. Ask who builds and maintains the non-production environment, and who pays for it when configurations drift.

Retest cycles and one-time setup

Setup work, including scope definition, rules of engagement, test harness build, and baseline harm evaluation, happens once per system. Retests recur after prompt, model, or tool changes, and a changed system version can invalidate earlier results. Ask the quote to separate the initial engagement from each retest, to state what triggers a retest, and to say what is included in the retest report.

When the EU AI Act and customers ask for adversarial testing

The EU AI Act's adversarial testing duty is narrower than many buyers assume. Article 55(1) requires providers of general-purpose AI models with systemic risk to perform and document adversarial testing of the model itself. It is a model-level obligation, not a blanket rule that every LLM application must undergo AI red teaming. If you build a chatbot or retrieval tool on top of a third-party model, the duty may sit with the model provider, while your own obligations depend on your role under the Act and the risk category of your use case. Confirm with your regulator, legal counsel or the model provider's documentation which party holds the obligation before you buy a specific test.

Customer questionnaires and security reviews usually ask a different question: how did you test this system? Here the vocabulary matters. NIST AI 600-1 gives you language for AI red teaming scope, including abuse that facilitates attacks on other systems and generative AI attacks such as prompt injection. The OWASP LLM Top 10 and MITRE ATLAS give you shared identifiers, such as LLM01 prompt injection and ATLAS technique AML.T0051.001 for indirect prompt injection, so a reviewer can map your report to their own checklist. Use these references to describe your scope precisely, and ask your customer which list they want mapped in the report.

Penetration testing obligations come from other frameworks, and they do not name AI systems. DORA Article 25(1) lists penetration testing and scenario-based tests for financial entities, and Article 26 sets threat-led testing for entities a competent authority identifies. PCI SSC's penetration testing guidance says an automated tool run alone does not satisfy a penetration testing requirement, and CIS Control 18 asks for internal and external penetration tests at least annually, with validation of detections afterward. Check each requirement against your own scope: whether the LLM application, its API and its data stores sit inside the cardholder data environment or the ICT estate, and whether an AI test counts toward the required test or only supplements it. Confirm these points with your auditor, assessor or customer contract before you scope the work.

Questions buyers ask before hiring outside help for AI red teaming or LLM penetration testing

What drives the cost of outside AI red teaming or LLM penetration testing?

Cost follows the written scope, so the main drivers are how many applications, models, agents and data sources are in play, how many integration points and user roles need testing, and how much human expert time the work requires. No supplied figure gives a price, and DeployOpen does not publish one, so the quote follows from the agreed scope. Ask each provider to split the quote into planning, testing, analysis, reporting and retest effort, and to state which parts are one-time and which recur. Ask whether tooling licenses are included or passed through to you.

How long does an LLM application security test take?

Duration depends on scope size, the number of attack scenarios, how many environments and test accounts are available, and how much analysis the findings need before they feed governance. No supplied fact gives a duration for this work, so ask each provider for a timeline tied to a written scope. Ask what delays the start, such as access approvals, staging readiness or waiting for system owners to answer questions, and whether the schedule includes a retest window.

What does the testing team need from our staff?

Plan for named system owners who can explain the model, prompts, tools and data flows, a technical contact for environment questions, and test accounts with the same roles your real users hold. You will also need a staging or non-production environment where the team can send adversarial inputs without affecting customers or live data. Confirm with the provider whether they need read access to logs, configuration or system prompts, and who approves each access request before testing begins.

Can our own team run AI red teaming with open-source tools?

Yes, many teams start with open-source tools, but the tool produces attack traffic and scores, not a judgment about harm or business impact. NIST AI 600-1 says the quality of red-teaming output depends on the background and expertise of the team, and that results need additional analysis before they feed governance and risk management. Ask whether your staff have the skills to craft context-specific attacks, interpret results and reproduce findings. Tool choice belongs in a separate evaluation of license, maintainer and attack coverage.

Does one engagement cover both AI red teaming and LLM penetration testing?

One engagement can cover both, but the two approaches ask different questions. AI red teaming probes model behavior, misuse and harms, with techniques catalogued in MITRE ATLAS, while an LLM application penetration test examines the web, API and data layers around the model using a conventional testing method such as the OWASP Web Security Testing Guide. NIST SP 800-115 notes that organizations often combine techniques to reach an in-depth assessment at an acceptable level of risk. Ask the provider to show how each part is scoped, staffed and reported.

How often should we retest after a model or prompt change?

Retest whenever the model version, system prompt, retrieval sources, tool permissions or guardrails change in a way that could alter behavior. Frequency beyond that depends on your risk appetite and any applicable requirement. The EU AI Act's Article 55(1) requires providers of general-purpose AI models with systemic risk to perform and document adversarial testing, so confirm whether that applies to you. Ask providers whether retesting is offered as a retainer, how it is scoped against the change, and whether continuous testing is included or quoted separately.

How do we write a scope that a provider can quote accurately?

Write the scope as a list of systems, not a goal. Name each model, application, agent, tool connector, API, data store and user role in scope, and state which are out of scope. Describe the objectives, such as the harms or abuse cases you care about, and the rules of engagement, including allowed techniques, testing hours, data handling and escalation contacts. NIST SP 800-115 provides a rules of engagement template in Appendix B that you can adapt. DeployOpen agrees scope, deliverables and timelines in writing before work starts.

How DeployOpen scopes, delivers and supports AI and LLM security testing

DeployOpen runs security testing for AI systems, including AI red teaming and LLM application penetration testing, and prepares the controls and evidence that go with it. Work is scoped in writing before it starts, run against systems inside your own cloud accounts, data centers or air-gapped networks, and handed over with documentation your team can keep using.

How we scope

Written scope states which model behavior, misuse and harm objectives are in and out of the test
Application, API, data and retrieval layers are listed separately so each has a defined test boundary
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins
Rules of engagement cover test accounts, data handling, rate limits and any production access

How we build

Test plans map to the OWASP LLM Top 10 categories and, where useful, MITRE ATLAS techniques
Red team scenarios combine objective-driven prompts with checklist coverage of the web and API layers
Testers with application security and AI backgrounds run the work, with domain expertise matched to your use case
Findings are validated and reproduced before they go into the report

How we hand over

Reports separate model behavior findings from conventional application and API vulnerabilities
Each finding includes reproduction steps, affected components and the test evidence behind it
Results are documented so your governance and risk teams can review them before they update the risk register
Retest scope and remediation guidance are agreed in writing at handover

How we support

Ongoing support, upgrades and continuous testing are available as retainers
Retainers can repeat test scenarios after model, prompt or application changes
Your team keeps the scenario library, configuration and documentation after the engagement
Confirm with DeployOpen which retainer terms and retest windows apply to your environment before signing