Automated vs human-led AI red teaming

Automated or human-led AI red teaming: which one to choose first

If your team is shipping an LLM feature and needs release evidence, start with automated tooling for breadth and regression, add human experts for context-specific harm, and combine both for release testing. Automated tools such as garak and promptfoo can run large volumes of adversarial prompts against each build, which makes them good at catching known weaknesses returning after a change. Human testers are better at finding harms that depend on your product, your users and your business rules, such as a support bot disclosing account details to a convincing caller. NIST AI 600-1 notes that human-led and AI-led red teaming may each be better suited to eliciting different types of harms, so the choice depends on which harms your feature can cause.

Automated testing alone is usually enough for an internal prototype with no sensitive data, no tool access and a narrow set of expected inputs, provided you can show that the harm categories you care about were probed and the results were recorded. Human-led testing alone can make sense early in design, when the team is still deciding what harms matter and needs expert judgment before writing any test cases. A combination is needed once the feature is customer-facing, can take actions, or touches regulated data. In that case, ask each team to document which harms cannot be measured quantitatively and where structured human feedback would be most useful, as the MEASURE 1.1 actions in NIST AI 600-1 describe. Confirm with your testing provider how they will record judgments of harm so the evidence can be reviewed later.

How automated and human-led AI red teaming compare

This table compares automated scanners and model-driven attack generation with expert human red teamers across goal, coverage, cost and release gating. Read each row as a trade-off: automated methods widen coverage and repeat cheaply, while human-led work finds context-specific harms that a fixed probe set may not reach.

Factor	Automated (scanners and model-driven attack generation)	Human-led (expert red teamers)
Goal	Broad, repeatable coverage of known weakness classes such as prompt injection, data leakage, toxicity and jailbreaks, using tools such as garak and promptfoo	Find harms that depend on the application's context, users and business process, and judge whether an output is actually harmful
Scope	Wide: can run thousands of adversarial prompts across a model or application in one pass	Narrow and deep: focused on the workflows, data and decision points that matter most to the system's owner
Method	Static, dynamic and adaptive probes (garak) or model-generated attacks and CI-triggered evaluations (promptfoo)	Crafted, multi-turn attacks built from domain knowledge, with findings verified or refined by the tester
What it finds	Known vulnerability patterns and regressions after a model or prompt change; high volume of candidate failures	Context-specific failures, chained misuse and harms that need domain expertise to recognize, including those that need judgment to score
What it misses	Novel attacks outside its probe library, business-logic abuse, and harms that need human interpretation to recognize	Breadth: a team cannot cover thousands of prompts, so untested paths remain untested unless automation runs alongside
Skills needed	Engineers who can configure, tune and read the tool, plus someone who triages results and removes false positives	Specialists in the relevant domain, such as cybersecurity, medicine or biotech, as NIST AI 600-1 describes for expert red teaming
Repeatability	High: the same probes can run on every build, model version or configuration change	Lower: results depend on the tester and time available, so record attack paths and prompts to make retests possible
Cost pattern	NIST AI 600-1 says generative AI-led red teaming can be more cost effective than human red teamers alone; cost is mostly setup, compute and triage time	Recurring expert time per engagement, scaling with scope depth; cost rises with the number of domains and attack paths tested
Effort to run	Lower per run once the harness exists; ask who maintains probe sets and how false positives are handled	Higher per run: scoping, rules of engagement, testing and reporting. Ask for the written scope and expected deliverables before work starts
Fit for release gating	Strong for blocking regressions on every build, provided the probe set is updated and failures have a defined owner	Better for periodic depth checks before major launches or new use cases, with automated suites handling each release

What NIST AI 600-1 says about each approach

NIST AI 600-1 treats automated and human-led red teaming as complementary, not interchangeable. The profile, which is the Generative AI Profile of the AI Risk Management Framework, notes that red-teaming led by generative AI can be more cost effective than human red-teamers alone. It also says human-led or AI-led red-teaming may each be better suited to eliciting different types of harms. For your test plan, that means the harm categories in scope should decide the mix. Ask the team running the exercise which harms each method was chosen to find, and record that choice in the test plan so a reviewer can see why a method was used.

The profile describes two human models that map to different staffing decisions. General public red-teaming can be more effective with large groups, which suits broad probing of user-facing behavior. Expert red-teaming uses specialists in domains such as medicine, biotech or cybersecurity, which suits harms that only a domain expert can recognize as harmful. The profile also describes combined approaches in which experts modify or verify prompts written by general participants to expand coverage. When you scope a combined exercise, confirm who reviews participant-written prompts, how their judgments are recorded, and how the expert's changes are tracked against the original prompt.

The MEASURE 1.1 actions connect this to documentation. MS-1.1-008 covers documenting risks that cannot be measured quantitatively, and MS-1.1-009 covers defining where structured human feedback exercises such as red-teaming would be most useful. Use these actions to write down which harms your automated tests cannot score reliably and where human judgment is therefore required. Treat the profile as guidance for designing and documenting your testing, not as a route to certification. It does not produce an audit result, and an assessor will not accept a red-team exercise as evidence of compliance on its own. Confirm with your own assessor or regulator how they expect this documentation to be presented.

What automated and human-led red teaming find and miss

Automated tools and expert humans probe the same AI system but fail in different places. The cards below separate what each approach reliably surfaces from what it tends to miss, so you can decide where each belongs in release testing.

Known probe classes at scale

Automated scanners such as garak run static, dynamic and adaptive probes for prompt injection, jailbreaks, data leakage, toxicity, hallucination and misinformation. This coverage matters most when you change the model, system prompt or retrieval layer and need a repeatable baseline across thousands of prompts. Check which probe families the tool covers, how often it is updated, and whether results are stored so you can compare runs between releases.

Repeatable regression testing

Model-driven attack generation fits into CI/CD pipelines, so a failing prompt set can block a release the same way a failing unit test does. Promptfoo, which is MIT-licensed and usable as a CLI or in pipelines, is one example. The condition that makes this valuable is a stable set of pass criteria that the team has agreed on in advance. Without those criteria, a green run tells you little.

Business logic abuse

Automated tools rarely understand what your application is supposed to do for your customers. A tester who knows the approval limits, account roles and workflow order can find an agent that approves a refund it should route to a human, or a chatbot that reveals another customer's order status. Ask whether your test plan includes scenarios written from your business rules, not only from generic harm categories.

Subtle and domain-specific harms

NIST AI 600-1 notes that human-led and AI-led red teaming may each be better suited to eliciting different types of harms, and that expert red teamers in fields such as medicine or cybersecurity can judge outputs that a classifier misreads. A clinical summary that is plausible but wrong for a specific dosage is an example of a harm a generic toxicity score will not flag. Confirm which harm categories your domain experts must review by hand.

Multi-turn social engineering

Attacks that build trust across a conversation, change persona gradually or split a harmful request into harmless steps are hard for single-prompt scanners to reproduce. Experienced testers adapt to the model's replies in real time, which is where human-led sessions earn their cost. Ask for a record of how each finding was reached, including the turns that preceded it, so the fix targets the right behavior.

Documenting unmeasurable risk

NIST AI 600-1 asks organizations to document risks that cannot be measured quantitatively and to define where structured human feedback, such as red teaming, is most useful (MS-1.1-008 and MS-1.1-009). Automated scores rarely capture these risks well. Confirm that each release record names the risks that were judged by people, who judged them, and what criteria they used, so auditors and risk owners can trace the decision.

How garak and promptfoo fit into a test program

garak, the Generative AI Red-teaming and Assessment Kit, is an Apache 2.0 tool maintained in the NVIDIA GitHub organization. It works like a vulnerability scanner for language models: it sends a library of probes to a target model and records how the model responds. The probes cover weaknesses such as hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks, and they are grouped into static, dynamic and adaptive styles. Static probes use fixed prompts, dynamic probes generate variations, and adaptive probes adjust based on earlier responses.

License: Apache 2.0, so the tool can be run inside your own environment and modified, subject to the license terms.
What it probes: Probe families for the weakness classes named above. Confirm the current probe list against the release notes before you set a coverage target.
Pipeline use: Run it as a scheduled or gated job against a model endpoint in a staging environment, then store the output as a test record.
What your team owns: Choosing which probes apply to your use case, writing custom probes for your business data and workflows, and deciding which failures are material.
Result handling: Triage of hits, false positives and model behavior that changes between versions, which the scanner cannot judge for you.

Before you rely on garak for a release gate, confirm its supported targets, probe families and output format against the current release notes.

How to combine automated and human-led testing in release testing

This sequence pairs broad automated probing on every build with targeted expert testing, so each release is checked for regressions and for the harms that only a skilled person is likely to surface. The output is a documented record of coverage, findings and release decisions that the organization has set in advance.

Define and document the risks to measure
Before any test runs, the AI risk owner and the test lead list the harms the system must be measured against, such as data leakage, prompt injection, toxic output or unsafe advice in the business context. NIST AI 600-1 places this under MEASURE 1.1, which includes documenting risks that cannot be measured quantitatively and defining where structured human feedback such as red-teaming would be most useful (MS-1.1-008 and MS-1.1-009). The step closes when each risk has a named owner, a test method and a note on whether it needs human judgment.
Run automated scans on every build
On each build, the team runs a scanner such as garak or promptfoo against a fixed set of probes that covers the documented risks. The purpose is regression detection: a change that reopens a previously passing weakness should fail fast. The engineering or platform team owns the pipeline, and the output is a per-build report of pass, fail and new-probe results, stored with the build identifier so results can be compared over time.
Route flagged and uncovered areas to experts
Failed probes, ambiguous outputs and risks with no automated probe go to expert red teamers with domain knowledge, such as cybersecurity or the regulated domain of the application. Automated tools cannot reliably judge harm in context, and NIST AI 600-1 notes that human-led and AI-led red-teaming may each be better suited to eliciting different types of harms. The step closes with a triage list that records each item, its reason for escalation and the expert assigned.
Have experts modify and verify automated prompts
Experts take the automated probe outputs and adapt them into context-specific attacks, then verify whether flagged responses really cause harm. NIST AI 600-1 describes combined approaches in which experts modify or verify prompts written by general participants to expand coverage. The output is a set of verified findings with reproduction steps, the exact prompts used and an explicit judgment on severity, so that scanner false positives and missed harms are both visible.
Feed findings back into probe sets
Verified findings become new or updated probes, so the automated suite learns from expert work and the next build tests the same ground. The AI security or test engineering lead reviews each new probe for duplication and for false positives before it joins the suite. The step closes when the probe set has a change log that links each added probe to the finding that produced it.
Update the AI risk register
Verified findings, accepted residual risks and risks still unmeasured are recorded in the organization's AI risk register, with owners and status. This keeps the testing results connected to governance decisions rather than left in tool output. The risk owner signs off on each change, and the step is complete when every verified finding has a register entry and a recorded disposition.
Decide release gates and cadence
The organization sets its own release gates, including which findings block a release, which need a documented exception and how often expert testing runs beyond the per-build scans. Those thresholds and cadence are the organization's decisions to define, based on its risk appetite and regulatory obligations, and DeployOpen does not set them. Confirm with your legal, compliance and product owners what applies to your system. The step closes with a written gate policy approved by the accountable owner.

What you receive from each approach

This table compares what an automated run leaves behind with what a human-led engagement delivers, so you can see which evidence your release process and assessors will actually use. Read each row as a question to ask a vendor or partner before signing. The main takeaway is that automated output is largely raw data, while human-led output is interpreted judgment that needs a written scope to be useful.

Deliverable	Automated run	Human-led engagement
Scope and plan	Usually a configuration file listing probes, models and thresholds	Scoped test plan agreed in writing before work starts, including the systems and harms in bounds
Harm coverage	Probe categories such as prompt injection, leakage or toxicity, as defined by the tool	Harm taxonomy written for your application, mapped to the risks in your own register
Raw evidence	Probe logs and per-attempt records, often large and hard to read without tooling	Reproduced transcripts of the attacks that succeeded, with the steps needed to repeat them
Results summary	Pass and fail counts per probe, which show frequency but not business impact	Severity judgments that explain why a failure matters for your users, data and obligations
Remediation	Usually none beyond the failing probe name; fixes are left to the team	Remediation guidance tied to each finding, such as prompt, guardrail, retrieval or access control changes
Retest	Rerun of the same suite, which confirms the probes pass but not that the attack path is closed	Retest of reported findings, agreed in the scope, with updated transcripts and status
Pipeline artifacts	Reports and configuration in CI, stored with each build for comparison over time	Usually delivered as documents and a read-out; artifacts for CI are added only if the scope asks for them
Deliverables fixed by	The tool version and configuration you choose, which you control	Deliverables fixed in the written scope before work starts, so ask for the list and format in the proposal

What drives cost and effort for each approach

Automated tooling and human-led red teaming fail on budget in different places. The tools carry little or no license cost, so the spend shifts to compute, engineering and triage, while expert time drives the human side. Read each card against your own release cadence and the systems in scope.

Open-source tool licensing

Both garak (Apache 2.0) and promptfoo (MIT) can be used without a license fee, so the question is what you pay around them. Confirm each project's current license terms in its repository before you standardize on it, and check whether any vendor add-on or hosted service you consider is priced separately. Ask any vendor or integrator quoting this work to separate tool costs from services.

Compute for probe runs

Automated runs send thousands of adversarial prompts to the target model and often to a judge model, so compute and model inference costs recur with every run. Costs rise with probe count, attempts per probe, number of models tested and how often you run the suite. Ask for an estimate of model calls per release gate, and confirm whether the target runs on your infrastructure or a hosted endpoint.

Engineering time to wire CI

Connecting promptfoo or garak to a pipeline, storing secrets, setting pass and fail thresholds and keeping the configuration current is one-time work plus ongoing upkeep. Upkeep recurs as models, prompts and tool versions change. Confirm who owns the pipeline configuration and who reviews changes to the test suite, because an untended suite gives false confidence.

Custom probes and triage

Generic probes miss application-specific harms, so writing custom probes for your data, tools and business rules takes skilled time. Results also need triage, since automated judges produce false positives and misses that someone must review before a finding goes to a developer. Ask how the vendor or team sizes triage effort and who signs off on a failed gate.

Expert time per harm domain

Human-led red teaming is paid for in expert hours, and NIST AI 600-1 points to specialists in domains such as medicine, biotech or cybersecurity for certain harms. Each harm domain you add, such as clinical advice or cyber misuse, usually adds a specialist to the team. Ask which domains the quote covers and whether specialists are named or drawn from a pool.

Systems, modalities and retest rounds

Cost grows with the number of systems, the modalities they accept (text, images, audio, tool calls) and the rounds of retesting after fixes. Each retest round is recurring effort, and a fixed-scope engagement fixes these counts in writing before work starts. Ask for the count of systems and modalities in the quote, the number of retest rounds included and the reporting depth expected for each.

Questions buyers ask before choosing AI red teaming

Do automated red teaming tools alone satisfy a release gate?

Not on their own, unless your gate only asks for broad coverage of known prompt attack patterns. NIST AI 600-1 says generative AI-led red teaming can be more cost effective than human red-teamers alone, but it also says human-led or AI-led red teaming may each be better suited to eliciting different types of harms. A gate that depends on a single method inherits that method's blind spots. Write the gate criteria first, then decide which harm categories need an expert judging the output. Confirm with your risk owner whether they accept automated results as sufficient evidence for each category.

How much human time does a combined AI red teaming program need?

The answer depends on how many harm categories you test, how often models or prompts change, and how much judgment each output needs. Automated runs are cheap to repeat, so their cost is mostly setup, configuration and reviewing flagged results. Human effort concentrates in expert testing, where specialists in areas such as cybersecurity or medicine craft context-specific attacks and judge harm. NIST AI 600-1 notes that combined approaches let experts modify or verify prompts written by general participants, which lets you keep expert hours focused on verification and the highest-risk categories. Ask any provider for a written breakdown of hours by method, and do not accept a total without it.

Can our team run garak or promptfoo ourselves?

Often yes, for the automated part of the work. garak is an Apache 2.0 tool that probes LLMs for weaknesses such as prompt injection, data leakage, toxicity and jailbreaks using static, dynamic and adaptive probes. promptfoo is MIT-licensed and usable as a CLI, a library or inside CI/CD pipelines. Both are open source, so the license cost is zero, but running them still takes engineers who can select probes, set thresholds, manage model endpoints and triage results. Expert judgment on whether an output is harmful, and attacks designed for your specific workflow, usually remain a separate skill set. Confirm your team's capacity for both before deciding.

How long does a human-led AI red teaming engagement take, and what drives it?

Duration is set by the agreed scope, not by a standard length. The main drivers are the number of systems and model endpoints in scope, the number of harm categories and use cases, whether the target includes tools, retrieval or agent actions, how much access the testers receive, and how many rounds of retesting follow fixes. Fixed-scope work is agreed in writing before it starts, so timelines should appear in the statement of work with their assumptions listed. Ask the provider which assumptions their timeline depends on, such as access timing and availability of your subject matter experts.

What does the work need from our internal staff?

Plan for named people from several groups. Product owners need to define the intended use, prohibited outputs and acceptable risk for each use case. Engineers need to provide test environments, model endpoints, logging and a way to reproduce results. Security or AI risk staff need to decide how findings are triaged and who accepts residual risk. Domain experts are needed to judge outputs in areas such as clinical or financial content. Ask for a list of access requests, decision points and review meetings in the proposal, so your staff time is visible before the work begins.

How do red team results connect to the AI risk register?

Each finding should map to a risk statement, a use case and a control or decision, not sit in a separate report. NIST AI 600-1 MEASURE 1.1 actions include documenting risks that cannot be measured quantitatively and defining where structured human feedback exercises such as red-teaming would be most useful (MS-1.1-008, MS-1.1-009). Use those items as the bridge: record which risks red teaming was chosen to evaluate, what was found, what was accepted or mitigated, and what retest is required. Confirm with your AI risk owner the format they need, and whether the register is maintained in your GRC tool or in a spreadsheet.

Which should we ask about first when comparing providers?

Ask how they split work between automated and expert testing for each harm category, and what evidence you receive at the end. A credible proposal names the tools it runs, such as garak or promptfoo where relevant, explains which findings need expert judgment, and lists the internal inputs it needs. Ask what is one-time, such as scope setup and the first assessment, and what recurs, such as regression runs after model or prompt changes. Request a written scope and quote that states these assumptions, since DeployOpen and other providers quote from written scope rather than a published price list.

How DeployOpen scopes and delivers AI red teaming

DeployOpen plans and runs AI red teaming as part of its security testing work, inside the client's own infrastructure. Scope, deliverables and timelines are agreed in writing before work starts, and the work can cover automated probing, expert-led testing or a combination of both.

How we scope

Written scope names the AI system, its model access paths and the harm categories to be tested.
Scope states whether automated probing, expert-led adversarial testing or both are included.
Scope records which test data, environments and accounts the team may use and how they are isolated.

How we build

Automated probe suites are configured for the agreed harm categories and run against the target.
Expert testers write context-specific attacks for business workflows that generic probe sets may miss.
Findings are judged for harm and impact by people, with automated results used to widen coverage.

How we hand over

Deliverables document methods, probe configurations, attack scenarios, evidence and reproduction steps.
Reports separate automated results from expert-validated findings so release owners can act on them.
Test configurations are handed over so the client can rerun the suites in its own CI pipeline.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after the initial engagement.
Retesting after model, prompt or guardrail changes can be agreed in the written scope.
Ask for the retainer terms and retest triggers in writing before the engagement starts.