LLM application security testing

Why LLM application security testing is now a launch and audit requirement

If a customer security review, a compliance program, an insurer or a launch date has asked you to test an LLM-powered application, the test you need covers the OWASP Top 10 for LLM Applications 2026 plus the web and API layers around the model. The model is only one part of the attack surface. The chat interface, the APIs that feed it, the identity and session handling, and the tools it can call all need testing as well. DeployOpen runs this test inside your scope and delivers findings, but it does not issue audit reports or certifications. Independent auditors and certification bodies make those decisions, so ask your assessor early which evidence they expect from the test.

The main risk categories are easy to state and hard to catch with a standard web test. Prompt injection (LLM01) occurs when instructions hidden in user input, documents or tool outputs override the developer's intent. Sensitive information disclosure (LLM02) covers the model revealing personal data, credentials or confidential records it can reach. Improper output handling (LLM10) appears when the application passes model output to a browser, interpreter or database without validation. Excessive agency (LLM03) describes a model holding more tools or permissions than its job needs. Unbounded consumption (LLM06) covers runaway token and compute costs. Hidden context exposure (LLM08) covers system prompts and internal instructions that users can extract. A scanner that checks headers and injection points in forms will not generate the adversarial prompts or observe how the model behaves across a conversation, so it will miss most of these.

What can be in scope for an LLM application test

An LLM application test covers more than the model. The scope is set by how the model is wired into prompts, interfaces, data and actions, and by which users and environments the test may touch. Each card below is a decision to settle in writing before any work starts, and the scope agreed for one application will not carry over to another.

Model integration and system prompt handling

Scope which model endpoints, hosted or self-hosted, sit behind the application and how the system prompt is stored, assembled and kept out of user reach. Testers look at whether the prompt holds secrets, business rules or internal tool descriptions that a user could extract, which maps to LLM07 System Prompt Leakage in the 2025 edition and to the broader LLM08 Hidden Context Exposure in the 2026 edition. Confirm which model versions and guardrail layers are in the path.

Chat and API front end

Define which user interfaces, public or internal APIs, streaming endpoints and webhooks feed prompts to the model. The front end is where prompt input arrives and where model output is rendered, so unsafe handling of model text in browsers, downstream services or logs is in scope here. Ask for an inventory of every route that accepts free text and every route that returns generated content.

Retrieval and tool connections

List each data source the model can read, such as document stores, search indexes, databases and ticketing systems, and each action it can take, such as sending email or calling an internal API. Indirect prompt injection and excessive agency both enter through these connections, and the test checks what the model can reach compared with what the requesting user is allowed to reach. Agent-style tool use may need its own scoping, covered in the sibling agent testing topic.

User roles and tenants

Agree which user roles, customer tenants and permission levels are included, because the model often answers from data that a given user should not see. Testers check whether retrieval, memory and tool calls respect the caller's identity, and whether one tenant's context can appear in another tenant's answer. Each role in scope needs its own test account with realistic data, so the boundary can be exercised rather than assumed.

Environments and reasons for each

Decide whether testing runs against staging, a pre-production copy or production. Staging is usually preferred for active abuse tests because prompt-based attacks can produce unexpected outputs, consume paid model tokens and write data to downstream systems. Production testing may be needed to confirm the live guardrails and configuration, and it should be limited to read-only or clearly marked test records. Confirm that staging uses the same model version, prompt and retrieval configuration as production.

Rules of engagement and data handling

Set rate limits, test window, cost ceilings for model calls, and the handling of any real personal or confidential data that testers may see in outputs. Agree how prompts, transcripts and screenshots are stored, who can read them and when they are deleted, and what happens if a test triggers a live alert or a third-party model provider flags the traffic. These terms belong in the written scope and should be confirmed with the model provider's own usage policies.

What the client prepares before scoping

Before scoping, gather a written architecture description covering the model, prompt flow, retrieval, tools and data stores. Add a data classification for everything the model can see, test accounts for each role in scope, and named technical and business contacts who can respond during testing. Testers can then propose a scope and a quote from these inputs. Without them, scope is an estimate that will need to change once the test starts.

How the engagement runs, phase by phase

The sequence moves from a written scope and agreed rules of engagement through threat mapping, hands-on testing of the application and model behavior, and reporting, ending with a retest that confirms fixes held.

Agree scope and written rules of engagement
DeployOpen and the client define the application, its model integrations, tools, data sources, user roles and environments in scope, along with what is out of bounds, such as production data and third-party model providers. Testing accounts, test data, rate limits, contact points and stop conditions are written down before any traffic is sent. The timeline is set in the written scope. This step closes when the client has signed the scope and rules of engagement, and both sides hold the same list of assets.
Map architecture and threats to OWASP and ATLAS
The testers review architecture diagrams, system prompts access, retrieval sources, tool definitions and trust boundaries, then map likely weaknesses to the OWASP LLM Top 10 2026 categories, including LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Excessive Agency and LLM06 Unbounded Consumption. Each category is tied to MITRE ATLAS techniques such as AML.T0051.001 for indirect prompt injection and AML.T0054 for jailbreak. NIST AI 100-2 E2025 supplies the attack taxonomy and NIST AI 600-1 frames the risk approach. This step closes with a threat map that the client approves, showing which tests will run and why.
Test the web and API layers
Testers apply OWASP-aligned methods to the authentication, authorization, session, input validation and API endpoints that surround the model, including object-level access checks on conversation and document identifiers. They also check how the application handles file uploads, cross-origin settings and error messages. Findings are recorded with the request context needed for reproduction by the client's developers. This step closes when every planned endpoint and role combination has a recorded result, pass or finding.
Test model behavior and tool use
Testers run controlled scenarios that probe whether the model follows injected instructions from documents, web content or tool outputs, whether it reveals system prompt content or data from other users, and whether its outputs are passed unsafely to downstream components. For agents, testers check whether tool calls stay within granted permissions and whether spending or request volume has limits. Tests run in a non-production environment with synthetic data, and each scenario records which guardrails held. This step closes when each mapped category has evidence of pass or failure.
Validate findings and assess impact
Each candidate finding is reproduced, checked for false positives and rated against the business impact the client agreed in scoping, such as exposure of customer records or unauthorized financial actions. Testers record the conditions under which a weakness appears, since model behavior can vary between runs. Findings that cannot be reproduced are documented as such rather than dropped. This step closes when every reported finding has a reproduction path, an impact statement and a severity rating the client has reviewed.
Deliver the report and remediation guidance
The report summarizes scope, methods, mapped OWASP and ATLAS references, findings with evidence, and prioritized remediation advice for developers, such as input and output controls, least-privilege tool scopes and rate limits. A walkthrough with the engineering and security teams explains each item. The client should confirm with its auditor or customer what format of evidence they need, since DeployOpen issues findings, not audit opinions. This step closes when the client has accepted the report and assigned owners to each finding.
Retest fixes and close the engagement
After the client applies fixes, testers repeat the original scenarios for each finding and check for regressions in nearby controls. Results are recorded as resolved, partially resolved or open, with the evidence used for each decision. The retest scope, timing and any additional fees are set in the written scope before the work starts. This step closes with an updated findings register and a final statement of which issues remain open.

What the test looks for in an LLM application

This table maps the weakness categories a full LLM application test examines to their OWASP LLM Top 10 2026 identifiers, showing what each one looks like in a typical application and why it matters to the buyer. Read each row as a question the test must answer about your system, and use the last column to decide which owners need to see the findings.

OWASP LLM ID and category	What the weakness looks like in this application	Why it matters to the buyer
LLM01 Prompt Injection	Untrusted text from users, uploaded files or web pages changes how the model behaves, including overriding instructions set by the developer	Attackers can steer the model away from its intended task, so every downstream action it can take is in scope
LLM02 Sensitive information disclosure	The model reveals personal data, credentials, contract terms or other records it can reach through its context, logs or tool results	Disclosure is a privacy and contractual exposure, and it often arrives through answers rather than through a visible data breach
LLM03 Excessive agency	The model holds tools, permissions or data scopes broader than the task needs, so a manipulated answer can trigger a real action	Agency sets the blast radius of every other finding, so buyers should know exactly which actions the model can take
LLM05 Data and model poisoning	Documents, training or fine-tuning data, or feedback loops that the application ingests can carry content that skews answers or behavior	Relevant where the application ingests data it does not fully control, such as customer uploads or scraped sources
LLM06 Unbounded consumption	Requests that generate very large outputs, long chains of tool calls or repeated expensive queries drain compute budgets or degrade service	Cost overruns and denial of service are often invisible until invoices or availability complaints arrive
LLM08 Hidden context exposure	System prompts, hidden instructions, internal tool descriptions or retrieval metadata can be drawn out of the application	Exposed internals make later attacks easier and can reveal business logic the vendor treats as confidential
LLM10 Improper output handling	Model output is passed to a browser, interpreter, database or downstream service without validation or encoding	Model text is untrusted input to other systems, so classic injection and rendering flaws can reappear through the model
Web and API layer around the model	Broken authentication, missing object-level authorization, weak session handling and unsafe file upload paths in the surrounding application	Attackers often reach the model through ordinary web and API weaknesses, so these are tested alongside AI-specific risks

How the test changes for chatbots, agents and API-only products

A conversational front end with retrieval answers user questions from documents, tickets or knowledge bases. Here the top of scope is prompt injection (LLM01) carried in through content the model reads, sensitive information disclosure (LLM02) and hidden context exposure (LLM08), because the risk is that the model reveals system instructions or retrieved content that the asking user is not entitled to see. Vector and embedding weaknesses (LLM09) also move up, since access control on retrieval determines what can leak. Agency is usually low, so the test spends less time on tool actions and more on what the retriever returns.

Indirect injection through content: testers plant or review documents, web pages and tickets the retriever may ingest, and check whether instructions in them change answers.
Cross-user retrieval: accounts with different permissions query the same assistant to confirm retrieved chunks respect tenant and role boundaries.
System prompt and context exposure: tests check whether instructions, internal identifiers or retrieved text can be extracted through normal conversation.
Output rendering: the test checks how markdown, links and HTML in answers are handled by the browser, under improper output handling (LLM10).
Cost abuse: long sessions and large retrieval requests are checked against any quotas (LLM06).

Confirm with the client whether the retrieval index holds data from more than one tenant, because that single fact changes the scope of the test.

What drives the cost and duration of an LLM application test

This table lists the scope factors that move the price and duration of an LLM application test, why each one matters, and what to request in a quote. Quotes follow from the written scope, so use the rows to check that each factor is stated and priced. The main takeaway is that the one-time engagement and the recurring retest or retainer are separate costs, and both should be defined before work starts.

Cost driver	Why it moves cost or duration	What to ask for in a quote
User roles and tenants	Each role and tenant boundary adds authorization paths to test, and cross-tenant checks multiply with every added role.	A list of roles, tenant models and which combinations will be tested.
Model-connected features and tools	Each feature that sends user content to the model, or lets the model call a function, adds inputs, outputs and trust boundaries to review.	A feature inventory with each tool, data source and output path named.
Write access for agents	Agents that create, change or delete data or trigger external actions need controlled test environments and more scenarios for excessive agency.	A statement of which actions the agent can take, which are write actions, and where testing will run.
Depth of model behavior testing	Prompt injection, sensitive information disclosure and output handling tests vary widely in breadth, from a sampled set of scenarios to a full harm-scenario suite.	The scenario categories covered, the number of test cases per category, and the method for judging results.
Number of environments	Each environment that needs testing, such as staging with production-like data, adds setup, access and verification work.	The environments in scope, what data they hold, and whether production testing is excluded.
Web and API surface	The web and API layers around the model are tested with the same methods used for any application, so endpoint count and authentication flows set the effort.	An endpoint or route count, the API specifications provided, and the authentication methods in use.
Retest after fixes	A retest checks only the findings in the agreed list, so cost depends on how many findings are fixed and how many rounds are needed.	The number of retest rounds included, the time window for them, and whether retest is in the original fee.
Ongoing testing retainer	Continuous testing recurs on a schedule. Its cost depends on release frequency, new features and the test categories covered each period.	The release cadence assumed, the test categories per period, and what triggers an out-of-cycle test.

Which frameworks ask for LLM testing and what they expect

Several regulations, certification schemes and customer questionnaires now ask whether an LLM application has been tested for adversarial abuse. Each one looks for different evidence, so the scope and reporting of the test should be set against the specific request. DeployOpen prepares the test evidence, while the auditing or certifying body decides whether it meets the requirement.

EU AI Act Article 15(5) resilience

Article 15(5) requires high-risk AI systems to be resilient against attempts to exploit their vulnerabilities, including data poisoning, model poisoning, adversarial examples, model evasion and confidentiality attacks. For an LLM application, an assessor will look for evidence that prompt injection, data leakage and tampering with retrieved content were tested and that the results drove changes. Confirm with your legal or compliance team whether your system is classified as high-risk before scoping the test to this article.

EU AI Act Article 55(1) adversarial testing

Article 55(1) applies to providers of general-purpose AI models with systemic risk and requires documented adversarial testing. Most application teams are deployers or providers of systems built on someone else's model, so this duty usually sits with the model provider rather than with your application. Ask your model vendor for their adversarial testing documentation, and use it to confirm which risks your application test must still cover.

HITRUST agentic AI factor

HITRUST CSF v11.9.0 added an Agentic AI selectable factor to the HITRUST AI Security Certification, mapped to the OWASP Top 10 for Agentic Applications 2026. If your LLM application calls tools or acts on a user's behalf, expect assessors to ask how goal hijack, tool misuse and privilege abuse were tested. Confirm the current factor requirements directly with HITRUST before you define the test scope.

NIST AI 600-1 voluntary profile

NIST AI 600-1 is the Generative AI Profile of the AI Risk Management Framework. It is voluntary, but many organizations use it to show that generative AI risks were identified, measured and managed. Evidence that a buyer or assessor will look for includes a documented risk map for the application, test results tied to those risks, and records of how findings were tracked to closure.

Customer security questionnaires

Enterprise customers increasingly ask LLM vendors whether prompt injection, data leakage between tenants and abuse of model usage were tested. Expect questions about test date, scope, the methodology used and whether critical findings were retested. Answer with a summary that names the scope and the categories covered, and avoid sharing unverified claims. Ask your customers which of their questions need a written report and which need a summary they can cite.

What the report gives your auditor, customer and engineering team

Every LLM application test should produce one report that three audiences can use. The executive summary tells leadership which risks matter for the business, in plain terms, with the number of open findings by severity. Each finding then maps to an identifier from the OWASP Top 10 for LLM Applications 2026, such as LLM01 Prompt Injection, LLM03 Excessive Agency, LLM06 Unbounded Consumption or LLM10 Improper Output Handling, and to MITRE ATLAS technique IDs where one applies, such as AML.T0051.001 for indirect prompt injection. The report should also state the severity rating method in its own text, so a reviewer can see how a finding was rated and against which factors, such as exploitability, data exposed and the agent permissions involved. Ask for the rating method in writing before testing starts, and check that it matches the scale your risk register uses.

Each finding needs enough detail for an engineer to fix it and for a reviewer to confirm the fix. Expect a description of the affected component, such as the prompt template, retrieval tool, output renderer or the service account an agent uses, plus reproduction notes written at the level of the weakness category and the observed behavior. Good reproduction notes describe what was sent, what the system did and which control failed to stop it, without publishing working attack strings or exploit code. Remediation guidance should name the control to change, for example output encoding, tool scoping, spend limits or access checks on retrieved content, and should say what evidence would show the change worked.

The retest is where a finding is closed or kept open. Ask for a retest letter that lists each original finding ID, the state at retest (fixed, partially fixed, accepted risk or still open), the version or configuration tested, and the date. A retest letter proves what was checked on that date, against that build. It does not prove that the application is secure, that no other path to the same weakness exists, or that the fix holds after the next model, prompt or dependency change. DeployOpen does not issue audit reports, attestations or certifications. Your auditor or certification body decides whether the test evidence meets its criteria, so confirm with them, before testing starts, the report format, finding ID scheme, severity method and retest evidence they expect.

What to check before you choose an LLM test provider

A provider's answers to these questions show whether it can test an LLM application end to end, from prompt handling and tool actions to the web and API layers around the model. Each card names a decision you can verify in writing before you sign a scope.

Evidence against the OWASP lists

Ask which items of the OWASP LLM Top 10 2026 and the OWASP Top 10 for Agentic Applications the provider tests, and how each maps to a test case. Expect coverage of prompt injection, sensitive information disclosure, excessive agency, improper output handling and unbounded consumption, plus the supply chain and poisoning items where they apply. Ask for the test plan, not a general statement that the provider follows OWASP.

Written scope and rules of engagement

Before any testing starts, the provider should document the in-scope prompts, models, tools, data stores, endpoints and accounts, along with the permitted test techniques and time windows. Rules of engagement should name who can stop the test and how the provider reports a finding that touches live customer data. A vague scope is a common source of disputes and of testing that goes beyond what the client approved.

Controlled environment for tool actions

An LLM application that can send email, change records or call APIs can cause real effects during a test. Ask where tool-connected actions run, whether they hit a staging copy of each downstream system, and which accounts and credentials the test uses. The provider should use scoped test identities and be able to show that write actions cannot reach production systems.

Handling of sensitive test data

Testers often see prompts, retrieved documents, user records and model outputs that contain personal or regulated data. Ask how the provider stores evidence, who can access it, how long it is kept and how it is deleted at the end of the engagement. Ask whether test data is synthetic where possible and whether the provider will sign the confidentiality and data processing terms your legal team requires.

Testers and supervision

Ask who will run the test, what experience they have with LLM applications, agents and the web and API layers around them, and who reviews their findings before they reach you. Ask whether a named senior engineer signs off on the report and how the provider handles a finding it cannot confirm. Clear answers here matter more than team size.

Report format and sample

Request a sample redacted report before you buy. Check that each finding lists the affected component, the weakness category, reproduction steps at a level your engineers can act on, severity with its reasoning, and a recommended fix. Also ask what the provider states it does not test, such as model training, third-party model internals or production load, because an unstated exclusion is a gap you will discover later.

Retest pricing and schedule

Ask whether a retest of fixed findings is included in the quote, how many retest rounds apply, and how the provider schedules them against your release calendar. Retesting is usually a separate line item, so confirm its cost and timing in writing. A retainer for continuous testing is a different commitment from a one-time retest and should be priced separately.

Common questions about LLM application security testing cost, time and effort

How much does an LLM application security test cost?

The cost depends on the written scope, not on a standard price. DeployOpen does not publish a price list, and a quote follows from the agreed scope. The main drivers are the number of components in scope: the chat or API front end, the web and API layers around it, each tool or action the model can trigger, retrieval sources, and any agent or multi-tenant boundaries. Scenario depth matters too. Testing prompt injection against a single assistant with no tools is a different effort from testing an agent that writes to business systems. Ask for a quote that lists each in-scope component, the test types covered (for example, prompt injection, sensitive information disclosure, output handling, excessive agency, unbounded consumption and the web and API layers), the reporting deliverables and any retest allowance. Most engagements are one-time fixed-scope work. Ongoing testing after model or prompt changes is usually a separate retainer.

How long does an LLM application penetration test take?

The duration is set in the written scope before work starts, and it follows from the components and test types agreed. A narrow application with one model, few tools and a single user role takes less time than a multi-tenant product with retrieval over sensitive documents and agents that act on users' behalf. Schedule factors also matter: access to a staging environment that mirrors production, test accounts for each role, and agreed windows for high-volume requests, since testing unbounded consumption can affect shared model budgets. Ask the provider to show the testing phases, the point at which findings are shared during the engagement, the report delivery date and how a retest is timed. Avoid any timeline that is not tied to a defined scope.

What does our team need to provide for the test?

Most of the effort on the client side is preparation, and it directly shortens the engagement. Expect to provide:

Architecture notes showing the model provider, hosting location, retrieval sources, tools and the trust boundaries between users, tenants and data
Test accounts for each user role, with the permissions they actually hold, plus any API keys or service identities the application uses
A named technical contact who can answer questions about system prompts, tool schemas and logging, and a security contact for urgent findings
Change windows and rate limits agreed in advance, so that consumption tests do not disrupt production or exhaust shared budgets
Confirmation of which logs, traces and monitoring alerts exist, so the team can check whether attacks were visible

Ask the provider to list these inputs in the scope document so that gaps are found before testing starts.

Can our internal team test the application instead?

Internal teams can and should run routine checks, such as regression tests on prompts and guardrails, input fuzzing in development, and monitoring for unusual tool calls. What an external test adds is independence and a documented method that an auditor or customer can rely on. An external tester approaches the system without the assumptions built into its design, applies a test plan mapped to the OWASP LLM Top 10 2026 and related lists, and reports findings with reproduction context and remediation advice. Many organizations use both: the internal team tests continuously, and an external provider tests at fixed points, such as before a launch or after a major change to tools or model. Whichever route you take, make sure the people testing have authorization in writing and that the test uses an environment where actions on tools and data are contained.

Is a prompt injection test alone enough for an LLM application?

No. A prompt injection test covers LLM01, the top risk in the OWASP LLM Top 10 2026, but it does not cover the rest of the application. A full LLM application test also examines sensitive information disclosure (LLM02), excessive agency (LLM03), improper output handling (LLM10), unbounded consumption (LLM06) and hidden context exposure (LLM08), along with the web and API layers that sit around the model, such as authentication, authorization between users and tenants, and input validation. Prompt injection is a focused engagement and is useful when the main concern is manipulation of model behavior. If your application has tools, data access or multiple tenants, the broader test is usually the better fit because injected instructions often matter only in combination with those capabilities. Confirm the scope against the list of test categories in the statement of work.

How is retesting handled after we fix the findings?

Retesting checks whether each reported finding is closed and whether the fix did not open a new weakness nearby. Most engagements include a defined retest step, but the allowance, the number of rounds and the period in which retests are accepted should be written into the scope before work starts. Ask the provider how each finding is tracked, whether retests are performed on the same environment and version, and how the updated report records the status of each item. Be aware that changes to the model, system prompt or tool definitions can alter behavior, so a retest should use the same scenarios as the original finding. For continuing change, a retainer can cover periodic retesting. Your auditor or customer will generally want to see both the original findings and the retest status.

What should we ask for in the test report?

Ask for a report that serves both engineers and the people who need to rely on it. It should state the scope and the components tested, the methodology and the framework references used, such as the OWASP LLM Top 10 2026 and, where relevant, MITRE ATLAS technique identifiers. Each finding should include severity with the reasoning behind it, the affected component, the conditions needed to reproduce it at a defensive level, evidence, and clear remediation steps. Also request a summary for leadership, a list of tests that held, and any limitations of the test. Confirm with your auditor or customer whether they need a specific format. DeployOpen prepares testing evidence but does not issue audit reports or certifications; independent auditors and certification bodies make those determinations.

How DeployOpen scopes, tests, reports and retests LLM applications

DeployOpen scopes LLM application tests in writing before any testing starts, then tests the model-facing layer and the web and API layers around it. Each engagement ends with documented findings and retest support so your team can show what was fixed.

How we scope

Agree in writing which LLM features, tools, data sources and user roles are in the test.
Confirm which OWASP LLM Top 10 2026 categories apply to your design, such as prompt injection and excessive agency.
Define test accounts, rate limits and cost guardrails so usage-based testing does not exceed your agreed budget.
Name the environment to test, whether staging or a production-like copy, and any third-party model APIs involved.

How we build

Map the application's trust boundaries, including what untrusted content reaches the model and what actions it can trigger.
Test prompt injection through user input, documents and tool outputs, recording which guardrails held and which failed.
Check output handling, sensitive information disclosure and unbounded consumption alongside the usual web and API tests.
Run the web and API layers against the OWASP Web Security Testing Guide and ASVS, including authorization on every tool call.

How we hand over

Deliver findings with reproduction context, affected components, severity and a remediation path your developers can act on.
Map each finding to the relevant OWASP LLM Top 10 2026 entry and, where useful, to MITRE ATLAS techniques.
Provide an executive summary for risk owners and a technical report for engineering, as set out in the written scope.
Share evidence in a form your internal audit or compliance team can use, while noting DeployOpen does not issue audit opinions.

How we support

Retest agreed findings after fixes and record which issues are resolved, partially resolved or still open.
Advise on guardrail and permission design so fixes address root causes, such as tool scopes and output validation.
Offer ongoing support, upgrades and continuous testing as a retainer after the fixed-scope engagement ends.
Confirm with your team how retest windows and follow-up support are priced and timed before the written scope is signed.