garak migration from manual LLM testing

What this migration involves

Moving from manual LLM testing to garak means turning jailbreak and prompt injection checks that people run by hand into scans that anyone on the team can repeat and compare. garak is a free, Apache 2.0 licensed command-line scanner developed in the NVIDIA GitHub organization. You install it from PyPI with pip, and the PyPI package requires Python 3.11 or newer. The migration covers installing and pinning the tool, mapping each existing manual test to a garak probe or a custom probe, and writing each run to a JSONL report with a garak.log file that persists across runs.

The usual triggers are an audit or customer request asking for repeatable evidence of adversarial testing, a change to the model, system prompt or retrieval setup that could alter behavior, and several teams running similar checks in different ways with different results. Before starting, confirm which models and endpoints are in scope, who owns the prompt library being ported, and where reports will be stored. Those three answers determine most of the work.

In-house or partner delivery depends mainly on how many targets are involved and who owns the results. One model behind one application, with a security engineer who will review the reports, is usually a manageable internal project. Many targets across several teams, or results that must be defended to an auditor or a board, usually justify outside help with scoping, baselines and handover. Ask any partner to state in writing which targets, probes and report formats are included.

What the work includes

Each engagement is scoped and agreed in writing before work starts, and the deliverables below define what you receive. The list can be trimmed or extended to match your existing test suite and reporting needs.

Inventory of manual tests

A register of every current manual jailbreak and prompt injection test, with the prompt text, the expected outcome, the owner who runs it and the system it targets. This inventory is the baseline for everything else, so gaps in ownership or undocumented expectations surface here, before any scan is configured.

Mapping to garak probes and detectors

Each inventoried test is matched to a garak probe and its detectors where a close equivalent exists, with notes on where the match is partial. Check that the detector's pass or fail logic matches the expected outcome you wrote down, because a probe that fires on a different condition can give false confidence.

Documented gap list

A written list of manual tests that garak does not cover, with a recommendation for each: keep as a manual test, write a custom probe, or retire it. Ask for this list explicitly, since it defines what automated scanning will not tell you about your application.

Scan configuration per target

A configuration file for each model or application endpoint, specifying the generator, the probes selected, the generations per prompt and the output locations. Confirm which generator your targets require, since REST endpoints need a separate endpoint description and authentication settings.

Baseline run and reports

A first full scan on each target, delivered with its JSONL report of every attempt and evaluation and the garak.log debugging file. Keep these artifacts together with the exact garak version and configuration used, so later runs can be read against the same reference.

Run comparison procedure

A written method for comparing a later run with the baseline, covering probe-level failure rates, newly failing probes and changes in the set of probes run. It should state which differences trigger review, and who reviews them, so that a version change or model update is not mistaken for a regression.

Runbook and handover

An operations runbook covering how to start scans, read reports, store artifacts and handle failed runs, followed by a handover session with the team that will operate the scans. Confirm who owns scan schedules, API key rotation and report review after handover, since these decisions determine whether the program keeps running.

What drives effort and duration

This table lists the factors that most change how long a garak migration takes and how much review work follows, why each one matters, and how to keep each one manageable.

Driver	Why it changes the work	How to keep it down
Manual test cases and reproducible prompts	Each manual case must be read, judged for whether it can be expressed as a prompt, and mapped to a probe or custom prompt set. Cases that depend on a human judging a free-form answer take longer to convert.	Inventory existing cases first and mark which are single-prompt and reproducible. Remove duplicates and retire cases that test the same weakness, then port the remaining set in priority order.
Number of model and application targets	Each target needs its own generator configuration, credentials, baseline run and result review. Application targets with guardrails or routing add more variables than a bare model endpoint.	Limit the first baseline to priority targets, such as the production model and the customer-facing application. Add other targets once the baseline and reporting are stable.
Prompts that depend on multi-turn context or tools	garak probes are largely single-prompt. Tests that need prior turns, session state, retrieved documents or tool calls require custom probes, a harness that keeps context, or a manual step, all of which add build and validation time.	Separate these cases early. Keep the ones that need multi-turn or tool behavior on the manual track unless a custom build is justified, and confirm current probe coverage in the garak user guide.
Endpoint variety and authentication methods	Each endpoint type (native provider API, gateway, custom REST service) and each authentication scheme (API key, OAuth token, mutual TLS, session cookie) needs a YAML definition, secret handling and response parsing checks. Non-standard response shapes often need custom parsing.	Standardize on the fewest endpoint patterns your estate uses, such as one gateway route for several models. Confirm the authentication flow with the application owner before writing configuration.
Number of environments (development, staging, production)	Each environment has different endpoints, keys and permitted load. Results from one environment cannot be assumed to hold in another. Production scans need change approval and rate control, which adds coordination work.	Start with staging or a pre-production copy for the baseline. Run production scans only with agreed windows and limits, and keep the same configuration files across environments so differences are visible.
Result retention for assessors	garak writes JSONL reports and a continuing garak.log per run. Assessors may ask for report history, the probe and garak versions used, and proof of who reviewed each finding, so storage, naming and version pinning need to be designed before the first run.	Define a storage location, retention period and run naming scheme with the compliance owner. Record the garak version and configuration hash with each run. Confirm retention requirements with your assessor rather than assuming them.
Number of reviewers of findings	Each failed probe needs a triage decision: confirmed issue, false positive, accepted risk or fix required. Detector results can be noisy, so more reviewers mean more calibration time and more disagreement to resolve.	Name one triage owner and a small review group. Agree a written triage rule set and calibrate it on the first baseline run before scaling to more targets.

Project phases and outputs

The sequence moves from an inventory of existing manual tests to a repeatable garak baseline, a documented comparison method and a handover the client's team can run without us.

Inventory existing manual tests
We review the prompts, scripts, notes and tickets your team uses for jailbreak and prompt injection testing, record which models and applications each one targets, and note the outcome each test was meant to show. The closing output is a test inventory signed off by your security lead, with each item tagged as portable, needs custom handling or retire.
Map tests to garak probes and detectors
Each inventoried test is matched to the closest garak probes and the detectors that judge their output, using the probe documentation in the garak user guide. Gaps where no probe fits are listed separately so they can be covered by custom work or kept as manual tests. The output is a mapping table showing the probe, detector and recommended settings for each test, reviewed with your team.
Connect the target and run smoke tests
We configure garak against the target, whether a supported hosted model API or your own application through the REST generator with a YAML endpoint description, and handle authentication in a way that keeps keys out of scan files. A small set of probes confirms that requests reach the target, responses parse correctly and the JSONL report and garak.log are written. The step closes when a smoke run completes end to end and its report has been checked line by line.
Port prompts that need custom handling
Manual tests that do not map cleanly are rewritten as garak-compatible prompts or custom probe modules, keeping the original intent and expected failure condition. Each ported item is tested against the target to confirm it produces the same kind of evidence as the manual version. The output is a reviewed set of custom probes with notes on what each one checks and why it exists.
Run the baseline and review results
A full baseline run covers the agreed probe set against the target, using the default of 10 generations per prompt unless the scope changes that setting. Results are reviewed with your team to classify false positives, where a detector flags a safe response, and false negatives, where harmful output passes unflagged, and detector or probe adjustments are recorded. The closing output is a reviewed baseline report with a documented list of accepted exceptions.
Define the comparison method and report retention
We write the procedure for comparing a new run against the baseline, covering which probe versions, target configuration and settings must match for results to be comparable, and how failure rates are read. Retention rules for JSONL reports and logs are agreed with your compliance and data owners, including where reports are stored and who can read them. The output is a written comparison procedure and retention schedule approved by your team.
Hand over and transfer knowledge
Your engineers run the scan pipeline with us under observation, then run it themselves while we review their outputs. We deliver the configuration, custom probes, runbook and comparison procedure as documentation in your repository. The step is complete when your team has executed a full scan and comparison without assistance, and the handover checklist in the written scope is signed by both sides.

What stays in-house and what goes to a partner

This table compares the migration work an organization usually keeps internally with the mechanical work often outsourced, and lists the skills each part needs so you can match tasks to people.

Part of the migration	Usually kept in-house	Often outsourced	Skills it needs
Unacceptable output definition	Yes: business owners state which outputs count as harm for each product, such as leaked customer data or policy-breaking advice	Partner drafts a worksheet from the manual test library for owners to edit	Product knowledge, risk ownership, security testing judgment
Finding approval	Yes: a named owner accepts, rejects or waives each FAIL result and records the reason	No, except to prepare evidence summaries for review	Risk acceptance, familiarity with the application's intended behavior
Remediation ownership	Yes: engineering teams change prompts, filters, retrieval or model settings and retest	Partner may suggest fix patterns, but does not own product changes	Application engineering, prompt and guardrail design
Inventory of manual tests	Shared: owners confirm which jailbreak and injection prompts still reflect real attacks	Partner collects, deduplicates and tags existing prompt libraries and notebooks	Security testing, reading prompt logs, spreadsheet or script handling
Probe mapping	Review only: owners confirm that mapped probes cover the risks they care about	Partner maps each manual test to garak probes and detectors and notes gaps	Working knowledge of garak probe and detector categories, LLM attack techniques
Endpoint configuration	Owners provide access rules, test accounts and approved hours for scans	Partner writes the REST generator YAML and tests authentication headers and response parsing	Python and CLI operation, API authentication, HTTP debugging
Baseline setup and comparison scripting	Owners decide which run becomes the baseline and what counts as a regression	Partner runs baseline scans, pins the garak version and scripts JSONL comparisons	Python scripting, JSONL handling, version pinning, scan pipeline operation

What usually goes wrong

Teams moving from manual jailbreak and prompt injection testing to garak tend to hit the same few problems. Each one undermines whether scan results can be trusted or compared.

Treating FAIL as a verdict

A garak FAIL shows that at least one attempt produced undesirable output, not how often or how badly the model failed. Review the detector output and the JSONL attempts behind each flagged probe before deciding whether a finding is real, a false positive, or a detector mismatch.

Baselines on unreproducible models

A baseline is only useful if you can rerun it. Record the exact model name, provider version, endpoint configuration and garak version used, because a baseline set on a model version nobody can reproduce cannot show drift later. Confirm with the provider how it labels and retires model versions.

Comparing runs with different settings

Runs that use different probe sets or generation counts are not directly comparable. The default is 10 generations per prompt, so a run with a lower count can look cleaner simply because it sampled less. Pin the probe list, generator options and generation count in a saved configuration and reuse it for every comparison.

Losing manual test intent

Copying manual prompts into a scan without their success criteria keeps the text but drops the reason it was written. Each ported prompt should keep a note on what behavior counts as a failure, which detector checks it, and which manual tester owned it, so the automated result can be judged against the original intent.

Scanning unpinned endpoints

If the target application, gateway routing or system prompt changes between runs, a difference in results may come from the target rather than the model. Pin the endpoint configuration, record changes to it, and schedule scans so that production changes do not silently alter what is being tested.

Reports without a retention owner

JSONL reports and garak.log files can contain harmful generated content and prompts that reflect sensitive systems. Name an owner for storage, access, retention and deletion before the first scheduled run, and confirm with your security and legal teams how long these records may be kept.

How the work changes by starting point

Teams whose jailbreak and prompt injection tests live in spreadsheets or wiki pages usually start with the largest translation job. The prompts exist, but they have no fixed structure, expected outcome or record of past results, so the first step is turning them into something a scan can use.

Inventory and deduplicate: Group existing prompts by the weakness they target and remove near-duplicates before deciding what to port.
Define pass and fail criteria: Most manual tests judge output by eye. Write the criterion for each group so a detector or evaluator can apply it consistently.
Map prompts to probe families: Decide which existing prompts fit a garak probe and which need a custom probe or a manual review step that stays outside automation.
Confirm who owns the results: Identify who reviews failures, since spreadsheets often hide a single tester's judgment.
Confirm data handling: Wiki pages may contain sensitive examples or internal system prompts. Confirm where the prompt library may be stored before it is loaded into scan configuration.

What you need to provide

Before the migration starts, your team needs to hand over four things: an inventory or export of the manual tests you run today, with a named owner for each test; read access to the target application or model endpoint; a test account with permissions that match the user role you want exercised; and a written list of the intended behaviors and policies that outputs are judged against. Without the last item, a garak run can flag output, but nobody can say whether it breaks a rule you care about. The inventory also tells us which manual prompts can be ported directly and which depend on context that only the owner knows.

You also need to name one owner who approves findings before they go into a remediation queue. This person decides whether a flagged output is a real weakness, a false positive from a detector, or an accepted risk. Decide early where JSONL reports and garak.log files will be stored, who can read them, and how long they are kept, because both files can contain prompts and generated text that may be sensitive. Confirm that a host running Python 3.11 or newer is available to the migration team, since the garak PyPI package requires that version or later.

API keys and model access credentials should be provisioned by your team, not shared through the engagement channel. Confirm in writing which environments are in bounds for test traffic and whether any production endpoint may be probed, at what request rate, and within which time windows. Ask your own platform or operations team to confirm that rate limits and content filters on the target will not be triggered in ways that disrupt real users. These rules should be agreed before the first scan, not after an alert fires.

Frequently asked questions

How much effort does moving from manual LLM testing to garak take?

The effort depends mainly on how many models and applications you test, how many manual test prompts you already have, and how well those prompts are documented. garak is free and Apache 2.0 licensed, so the work is in people and compute: configuring each target, porting prompts into a repeatable form, setting a baseline, and reviewing the JSONL reports the scans produce. Ask any partner to break the estimate into target setup, prompt porting, baseline runs, and handover, with each item tied to a written scope.

How long does a garak migration take?

Duration is driven by target count and inventory quality more than by the tool itself. Each model or endpoint adds configuration and access work, and an incomplete list of targets or undocumented manual prompts adds discovery time before any scan runs. Teams with a current inventory of models, applications and endpoints, plus a set of prompts with known outcomes, move faster. Confirm the timeline against your own target list in the written scope rather than relying on a general figure.

Can our internal team do the migration without a partner?

Yes, if you have engineers who can install a Python package, write a YAML file for an endpoint, and interpret scan output. garak installs from PyPI and requires Python 3.11 or newer, and its documentation at docs.garak.ai covers the common generators. The usual gaps are authentication handling for internal gateways, agreeing which probes matter for your use case, and turning results into remediation tickets. A partner can cover those gaps or hand them over with documentation.

What does our team need to provide for the migration?

Plan to provide a list of target models and applications, test accounts or API access with the scope your security policy allows, network paths to any internal endpoints, and the existing manual test prompts with their expected outcomes. You also need a named owner who can approve which probes run and a place to store reports and API keys. Missing access or unclear ownership is the most common cause of delay, so confirm these items before work starts.

Does garak cost anything to run?

garak itself has no license fee. The maintainers describe it as a free tool, and it is published under the Apache 2.0 license. Your costs sit in three places: people who configure and review scans, model API charges or GPU time for generations, and ongoing scan operations such as storing JSONL reports. Because each probe makes multiple generations per prompt by default, API or compute spend scales with the number of probes and targets, so ask for a per-run estimate based on your chosen scope.

Can our existing manual prompts be reused in garak?

Many can be reused, but they usually need restructuring before they become repeatable. garak works with probes and detectors rather than free-form chat sessions, so a manual jailbreak attempt must be mapped to a probe category, a fixed prompt, and a way to judge the output. Prompts that depend on a human reading a response or on conversation state need more rework. Review each prompt during discovery and record which ones map cleanly, which need adaptation, and which should be retired.

How do baselines help when a model or prompt changes?

A baseline is a recorded scan of a known configuration, so later runs can be compared against it. When a model version, system prompt or guardrail changes, you run the same probes and compare failure rates against the baseline to see what improved or regressed. garak marks a probe as FAIL when any attempt produces undesirable output, and each run writes a JSONL report, which makes these comparisons reproducible. Pin the garak version as well, since probe behavior can change between releases.

How DeployOpen delivers this work

DeployOpen moves existing manual jailbreak and prompt injection testing onto garak in infrastructure you control, starting from the prompts and checks your team already uses. The work is scoped in writing first, then built, documented and handed over so your team can run repeatable scans and compare results over time.

How we scope

Inventory the manual test prompts, attack categories and pass or fail rules your team uses today
Identify which models and applications are in scope and how each is reachable from your network
Agree in writing which garak probes, detectors and generators the engagement covers before work starts
Define what a baseline run is, which garak version it pins to, and how results will be compared
Engagements are fixed-scope, with deliverables and timelines agreed in writing; ongoing support is available as a retainer

How we build

Install garak from PyPI on a dedicated scan host running Python 3.11 or newer, with the version pinned
Configure generators for each target, including a REST generator described in YAML where the endpoint is custom
Port existing manual prompts into garak plugin configuration and check that results match your team's prior judgments
Set up scan runs that write JSONL reports and continue the garak.log file, stored where your team can review them
Test the pipeline with a baseline run, then record the expected failure rates that later runs are compared against

How we hand over

Deliver the scan configuration, probe selections and pinned version documented in your own repository
Provide a runbook covering how to launch scans, read JSONL reports and interpret FAIL results with failure rates
Walk your engineers through the baseline, the comparison method and how to record changes between runs
Document the decisions made during the build, including which probes were excluded and why
Confirm with your team the owners for scan schedules, report review and remediation follow-up

How we support

Review garak release notes before upgrades, since probe sets change across frequent releases
Re-run the baseline after any garak or target model change so results stay comparable
Offer retainer support for pipeline maintenance, new targets and periodic scan reviews, scoped in writing
Confirm with your team how API keys, scan hosts and report storage will be maintained after handover
Ask any support provider how they will handle upgrades that change probe outputs or failure rates

garak migration from manual LLM testing to automated scans