garak vs PyRIT: which one fits a one-off assessment and which fits an in-house red team
If your team needs answers about model weaknesses before a release or an audit and must pick a tool this quarter, the short answer is this: garak suits a one-off assessment run by a small team, and PyRIT suits an in-house red team that scripts its own attacks. garak is a command-line scanner. You choose a target through a generator, pick probes for hallucination, data leakage, prompt injection, toxicity or jailbreaks, and let its detectors judge the results. It writes a JSONL report of every attempt and a hit log of the attempts that found a weakness, which is usually enough to hand to a release reviewer or auditor. PyRIT is a Python framework. Its targets, prompt converters, scorers and multi-turn attacks such as Crescendo, TAP and Skeleton Key are building blocks that your engineers assemble into attack logic, with a memory layer that records conversations and scores in SQLite or Azure SQL.
The difference matters most in three decisions. First, who writes the attacks. With garak, the probe library is the attack catalog, and a small team can run it without writing much code. With PyRIT, the attack logic is code your red team owns, which is a better fit when your threats are specific to your application, such as a particular agent workflow or internal data store. Second, whether multi-turn attacks matter. If your risk depends on a model being walked gradually toward a failure over several messages, PyRIT's built-in multi-turn strategies are the more direct path. If single-prompt coverage is enough for the assessment, garak's catalog may be sufficient. Confirm which probes and strategies each version includes against the current documentation before you commit.
Third, who will maintain the harness. A one-off scan needs a runner, a target configuration and someone to read the report, which is modest ongoing work. An in-house PyRIT harness needs Python maintenance, dependency upgrades as releases such as v1.0.0 and v1.1.0 arrive, and scorers that someone checks for false positives. Ask yourself who will own that code after the assessment ends. If the answer is nobody, the scanner is the safer choice. Both projects are open source under permissive licenses, so the real cost is staff time and the effort to keep results defensible, which an implementation partner can scope in writing.
Which scenario favors garak and which favors PyRIT
This table maps common AI security testing scenarios to the tool that fits them better, based on how each one is built. Read each row as a question about your own team: who runs the tests, how often, and where the results need to live.
| Scenario | Better fit | Reason |
|---|---|---|
| One-off vulnerability scan before launch | garak | A command-line scanner with built-in probes for prompt injection, leakage, toxicity and jailbreaks, plus a JSONL report and hit log, gives a broad first pass without building a harness. |
| Scheduled regression scans | garak | Runs from a CLI with a fixed probe set and generator config, so the same command can repeat on a schedule and produce comparable JSONL output each time. Confirm the scheduling setup in your own pipeline. |
| Scripted multi-turn jailbreak research | PyRIT | Ships multi-turn attack strategies such as Crescendo, TAP and Skeleton Key, which are built around orchestrated conversations rather than single-shot probes. |
| Custom converters and scoring for a specific application | PyRIT | Prompt converters and scorers are Python components you compose and extend, which suits application-specific transformations and judging logic. |
| Teams with Python skills but no existing harness | PyRIT | Scripts are the primary interface, so a Python team can build targets, attacks and scorers directly. Expect more engineering effort than running a scanner with a config. |
| Teams that need results in a database | PyRIT | Its memory layer records conversations, scores and results in SQLite or Azure SQL. Confirm whether garak's JSONL output can meet your storage and query needs, or whether you need to load it yourself. |
How garak and PyRIT are built and what that means for operations
garak is a scanner you run against a target, while PyRIT is a framework you script attacks with, so the two tools put configuration, state and maintenance in different places. The cards below cover the parallel components on each side and the checks an operator should make before relying on either one.
garak probes and detectors
Probes generate the attack prompts and detectors judge whether the model failed each one. Operators choose which probe families and detectors to run from the command line, so the scan scope is set per run rather than in a script. Check which detectors match your failure definitions, because a detector that misjudges a response will skew the hit log. Review the probe list again after each upgrade, since the set of attacks changes between releases.
garak generators and harnesses
Generators are the connectors to the target, including Hugging Face, OpenAI, AWS Bedrock, llama.cpp models and custom REST endpoints. A harness runs the chosen probes against the chosen generator. Operators maintain the generator configuration and any credentials for each endpoint, and must update them when a target's API changes. Confirm how your REST endpoint is wrapped before trusting results, because a mismatched request format can produce misleading scores.
garak reports as state
garak writes a JSONL report with every attempt and a hit log with only the attempts that found a vulnerability. The files are local output, so retention, access control and deletion are the operator's job. Decide where reports are stored before the first run, and treat the hit log as sensitive because it records working attack prompts against your system.
PyRIT targets and converters
PyRIT targets wrap the system under test, and prompt converters transform attack text before it is sent, for example to encode or rephrase it. Operators write these components in Python, so the attack logic lives in code that must be reviewed and versioned like any other test suite. Plan for target adapters to change when your application's interface changes.
PyRIT scorers and attack strategies
PyRIT scorers judge responses, and multi-turn attacks such as Crescendo, TAP and Skeleton Key carry context across turns. Operators decide which scorer governs each attack and must test those scorers against known outputs, since a multi-turn result is only as reliable as the judgment at each step. Expect scorer and attack logic to need review when you change models.
PyRIT memory and repository source
PyRIT's memory layer records conversations, scores and results in SQLite or Azure SQL, so run data persists in a database you choose and secure. The original Azure/PyRIT repository is archived, and development continues at microsoft/PyRIT. Confirm your team pulls from the active repository and pins a release, including v1.1.0, before building internal tooling on it.
How garak and PyRIT compare on capability and operations
This table sets garak's command-line scanner against PyRIT's Python framework across license, release, attack style, scoring, storage and CI fit. Read each row as a question about how you would run the tool, then confirm the rows marked for confirmation with the project before you commit.
| Capability | garak | PyRIT |
|---|---|---|
| License | Apache-2.0 | MIT |
| Maintainer | NVIDIA, in the NVIDIA GitHub organization | Microsoft, at microsoft/PyRIT (the original Azure/PyRIT repository is archived) |
| Latest supplied release | v0.17.1, October 16, 2026 | v1.1.0, September 4, 2026 |
| Interface | Command-line scanner driven by probes, detectors, generators and harnesses | Python framework of targets, prompt converters, scorers and attack orchestrators |
| Attack style | Single-prompt probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks | Multi-turn attacks such as Crescendo, TAP and Skeleton Key, plus single-turn prompts |
| Scoring | Detectors judge each attempt for a failure | Scorers evaluate responses within the framework, confirm the scoring options for your attack |
| Reporting | JSONL report of every attempt plus a hit log of successful attempts | Results and scores recorded in memory, confirm export formats with the project |
| Result storage | Local output files written by the scan | SQLite or Azure SQL memory layer for conversations, scores and results |
| Generator or target support | Hugging Face, OpenAI, AWS Bedrock, llama.cpp and custom REST endpoints | Custom targets written in Python, confirm the list of built-in targets for your model |
| CI fit | Runs as a command, so it can be placed in a pipeline step, confirm pass/fail conventions | Scripted in Python, so it fits a test suite, confirm how runs are scheduled in CI |
What each platform costs to license and what the license allows
garak is licensed under Apache-2.0 and is maintained in NVIDIA's GitHub organization, at the garak repository. Apache-2.0 permits commercial use, modification and internal redistribution, subject to its notice and attribution terms, so an in-house red team can change probes or detectors and keep those changes private. The license does not come with a support contract, a hosted service or a managed scanner. Your team carries the work of installing the CLI, choosing generators for each target, running scans and reviewing the JSONL report and hit log.
- License terms: Apache-2.0 governs use, modification and redistribution of the code.
- Maintainer: NVIDIA's GitHub organization publishes releases, including v0.17.1 from October 16, 2026.
- Paid editions: No paid tier was supplied, so confirm with the project whether any commercial edition or support offering exists.
- Your costs: Compute for scan runs, model API or endpoint usage charged by your target provider, and engineering time to triage findings.
- Upgrades: Pin the version you validate and test new releases before adopting them in a shared pipeline.
Before you commit, confirm the license file in the repository and the release notes for the version you plan to deploy.
PyRIT is licensed under MIT. The original Azure/PyRIT repository is archived on GitHub, and development continues at the microsoft/PyRIT repository. If you have pinned a source reference to the archived location, move it to the new repository and check the release notes for the version you adopt. MIT is a permissive license that allows commercial use, modification and internal use with a copyright and license notice retained, so an in-house team can extend targets, converters and scorers. The license does not add hosted infrastructure or vendor support.
- License terms: MIT governs use, modification and redistribution, with the notice retained.
- Repository move: The archived Azure/PyRIT repository no longer receives development; use microsoft/PyRIT.
- Recent releases: v1.0.0 was published on July 24, 2026 and v1.1.0 on September 4, 2026.
- Paid editions: No paid tier was supplied, so confirm with Microsoft or the project whether any commercial offering exists.
- Your costs: Compute, storage for the SQLite or Azure SQL memory layer, target model usage and the engineering time to script multi-turn attacks.
Confirm the license and the current repository location before you copy code into an internal fork.
The license price of both tools is zero, so the budget question is what it takes to run them. Cost falls into one-time items such as setup, target configuration and writing the first test plans, and recurring items such as scan compute, target model usage, storage for results and the time people spend reviewing findings. Both projects are used with models you already operate or buy, so your usage bill for those models often exceeds the tool cost itself. The sizes of these items depend on how many targets you test, how often scans run and how many attack variants you script.
- One-time: Environment setup, integration with your identity and secret handling, and initial attack design.
- Recurring: Target model usage, compute for each run, storage for reports and the people who review hits.
- Cost drivers: Number of targets, scan frequency, and whether multi-turn attacks run against paid endpoints.
- Quote items: Ask for the hours to install, configure and validate each tool in your environment.
- Hidden work: Keeping a pinned version current and retesting after target model updates.
Ask any implementation partner to price installation, pipeline integration and triage as separate line items so you can compare the two tools on effort.
Neither project was supplied with a paid edition, a managed service or a support tier, so this tab does not state that one exists or that none does. The practical step is to check each project's repository, documentation and release notes, and to ask the maintainers directly before you rely on a commercial option. Ask for written confirmation, because a vendor's terms can change between your evaluation and your purchase. This also matters for any hosted offering built on top of an open-source core, since that offering may carry different license or support terms from the code you download.
- Ask the project: Whether a commercial edition, support plan or hosted service exists for each tool.
- Read the license: Confirm the license file in each repository matches what your legal team has reviewed.
- Check the source: Verify the repository you install from is the current maintainer location.
- Record the answer: Keep the dated written answer with your procurement file.
- Plan for change: Note that a tool's ownership or terms may change, and set a review date.
Bring these answers to your security and legal reviewers before you choose one tool for a shared pipeline.
What drives the running cost of each tool
The running cost of garak and PyRIT comes from four places: model usage for every attack, the host that runs the harness, the people who write and maintain attacks, and the people who read the results. Model usage is the most visible line. Each garak probe run and each PyRIT multi-turn attack sends prompts to the target and, where a scorer is an LLM, to a judge model as well, so a multi-turn Crescendo, TAP or Skeleton Key run typically sends far more requests than a single-shot garak probe. Ask the vendor or your own team to estimate requests per run for the probe families and attack strategies you plan to use, and confirm which of them call a paid endpoint for scoring.
Engineering time is the cost that differs most between the two tools. garak is configured from the command line: you choose probes, detectors and a generator, then review the JSONL report and hit log. PyRIT is a Python framework, so an in-house team writes and maintains attack scripts, converters, scorers and memory handling, and must update them when the PyRIT API changes between releases such as v1.0.0 and v1.1.0. Both tools have upgrade work when new releases land, but PyRIT's scripts carry more of that work. Review effort is also recurring: someone must triage hit logs or scorer output, decide what counts as a real failure, and track false positives over time. Persistent storage in SQLite or Azure SQL adds a small operational duty for backups and retention. Sort costs into one-time and recurring before you ask for a quote. One-time items include installing and connecting the generators to your targets, designing the first red team plan, choosing detectors or scorers, and agreeing on what a failure means for your risk owners. Recurring items include scan or attack runs on each model change, model and judge usage, hosting, maintenance of scripts or configs, and reviewing results. Ask any provider to split the fixed-scope fee from ongoing retainer work, to name who runs the tools and who reviews findings, to state which upgrades are included, and to list the model and infrastructure costs you will pay directly. Confirm each item against your own environment, since DeployOpen does not publish a price list and quotes follow the written scope.
Moving between garak and PyRIT without losing test history
This sequence lets a team switch harnesses while keeping its prior results, target definitions and findings readable, so the new tool is judged against the same baseline the old one produced. The output is a documented migration record and a re-run comparison that shows where the two tools agree and where they differ.
Archive existing garak reports and hit logs
The engineer who owns the current garak runs copies every JSONL report and hit log into a versioned evidence location, keeping the file names, run timestamps and the garak version used for each run. garak writes a JSONL record of every attempt and a hit log of attempts that found a vulnerability, so both files are the primary history to preserve. Done means each run can be opened and read without the original harness, and a checksum list exists for the archive.
Export PyRIT memory records if PyRIT is already in use
If the team already runs PyRIT, the engineer exports conversations, scores and results from the memory layer, which stores them in SQLite or Azure SQL, to a format the assessment team can read. Teams moving in the opposite direction can skip this step. Confirm the export method with the PyRIT version in use, because the memory schema is tied to the release. Done means a sample conversation and its scores can be reproduced from the exported copy.
Map probes and attacks to test intent
A security engineer builds a table that links each garak probe category, such as prompt injection, data leakage or jailbreaks, to the equivalent PyRIT attack or scorer, and records the risk each test was meant to cover. Some garak probes will have no direct PyRIT counterpart, and some PyRIT multi-turn attacks such as Crescendo, TAP or Skeleton Key will have no garak equivalent, so those gaps should be written down. Done means every existing test has a stated intent and a mapped target, or a documented reason it is retired.
Rebuild generators and targets against the same endpoints
The platform engineer recreates each model connection in the new harness, pointing at the same endpoints, model versions and authentication method used in the original runs. In garak these are generators; in PyRIT these are targets. Target definitions transfer conceptually, but the configuration code is rewritten for each tool. Done means a single benign prompt returns a response from each rebuilt target, and the endpoint, model identifier and version are recorded next to the archived runs.
Re-run a baseline on the unchanged model
Before any new test is added, the team runs the rebuilt harness against a model version that already has archived garak results, using the mapped probes or attacks. Scoring logic will differ between the tools, so the comparison is about whether the same weaknesses surface, not whether counts match exactly. Done means a written comparison lists agreements, disagreements and any finding that appears in only one tool, with the reason for each where known.
Rebuild scoring and scheduling in the new harness
Detectors in garak and scorers in PyRIT are not interchangeable, so the team reviews each scoring rule and decides whether to keep the new tool's default or write a custom scorer. The CI job or scheduled run that triggers tests is also rebuilt in the new tool's pipeline. Confirm with the vendor or your own environment how each tool's scoring is documented before relying on it for a decision. Done means the pipeline runs on a schedule and a scored sample matches the reviewer's expectation.
Retire the old harness and keep the archive
Once the baseline comparison is signed off, the team removes the old harness from scheduled jobs and marks its configuration as read-only, while keeping the archived reports and the mapping table under the same retention policy as other security evidence. Retiring the code does not delete the history. Done means no scheduled job calls the old tool, the archive is still readable, and the owner has recorded the date and approver of the switch.
What implementing each tool involves and where projects run long
An in-house garak install is mostly configuration work. The team sets up a Python environment, chooses a generator for the target (Hugging Face, OpenAI, AWS Bedrock, llama.cpp or a custom REST endpoint), and then selects probes and detectors that match the risks in scope, such as prompt injection, data leakage or jailbreaks. Credentials for the generator belong in the secrets store, not in shell history. After a run, the reader works through the JSONL report of every attempt and the hit log of attempts that found a vulnerability. Confirm the exact command-line options and the generator configuration keys against the version you deploy, using the garak repository as the reference.
PyRIT asks for more engineering. Your team writes the targets that call your application, the prompt converters that transform attacks (encoding, translation or rephrasing), and the scorers that judge responses. Multi-turn strategies such as Crescendo, TAP and Skeleton Key are scripted against those components, and the memory layer must be provisioned, either SQLite for a local workspace or Azure SQL for shared use. Someone on the team owns that storage, its retention rules and access to the stored conversations, which often contain sensitive test data. Check the current component APIs in the PyRIT documentation before you commit to a design.
Projects usually run long in three places. The first is agreeing what counts as a failure. Detector and scorer output is only useful once the business has defined which responses are unacceptable, and that decision often needs legal, product and security sign-off. The second is the target's access controls: test accounts need the same permissions as real users, no more and no less, or results will overstate or understate exposure. The third is upkeep. Attack scripts and detector configurations break when prompts, tools, retrieval sources or model versions change, so the team must budget for regular maintenance, not a single build. DeployOpen can plan and run this work under a written, fixed scope with agreed deliverables. Effort and duration depend on the number of targets, the depth of multi-turn testing and the state of your environment, so ask for a quote tied to a written scope.
Questions buyers ask before choosing between garak and PyRIT
What does it cost to run garak or PyRIT?
Both tools are open source, so the license fee is zero for either one. The cost is the people time to set up targets, configure runs, interpret results and keep scripts working, plus compute for the model calls the tests generate and any hosted model API charges. garak is licensed under Apache-2.0 and PyRIT under MIT, so neither license adds a recurring charge. Drivers that raise cost include the number of targets, how many probes you run per release, and whether a specialist writes multi-turn attack logic in PyRIT. Ask any implementation partner for a quote that separates one-time setup from recurring runs, and confirm which model API charges your team will carry directly.
How long does a first assessment take to set up?
The answer depends mostly on whether you are scanning one endpoint with a standard probe set or building attacks against a custom application. garak is a command-line scanner, so a first run against a supported generator can start quickly once credentials and the target are configured. PyRIT is a Python framework, so a first useful assessment usually includes writing target code, choosing converters and scorers, and configuring memory storage in SQLite or Azure SQL. No duration is established here. Ask for a written scope with named targets, the probe or attack families in scope, and the deliverables before agreeing on a timeline.
What does the internal team need to provide?
Your team needs access to the target model or application, a named owner for the test accounts and API keys, and someone who can read the results and decide what counts as a failure. For garak, the main skill is operating a CLI and reviewing the JSONL report and hit log. For PyRIT, the team also needs Python engineering to write and maintain attack scripts, and someone who understands multi-turn attacks such as Crescendo, TAP and Skeleton Key well enough to judge their output. Confirm in writing who owns environment changes, data handling for stored conversations, and sign-off on findings.
Can one internal team run either tool without outside help?
A team with Python skills and access to the target can run garak with little more than a CLI and configuration, since it is built around probes, detectors, generators and harnesses that already exist. Running PyRIT without outside help is realistic for a team that already builds security tooling, because it asks for scripting and design choices. The harder part is judgment: deciding which attack paths matter for your application and whether a score reflects a real weakness. If that judgment is not in-house yet, a fixed-scope assessment with a clear handover is a common route. Ask for the handover plan, including documentation and a walkthrough of the scripts.
Which tool should a regulated team pick for a release gate?
For a repeatable gate that must run on every release, garak is the simpler fit because it runs as a scanner with a predictable report format and an Apache-2.0 license your legal team can review directly. PyRIT fits better when the risk is a conversation that unfolds over several turns and your team can maintain the attack code. Neither tool issues an audit report or certification. Compliance readiness work prepares the controls and evidence that an independent assessor reviews. Confirm the release criteria, evidence retention period and approval path with your compliance lead before selecting either tool.
Can garak and PyRIT results be shared with each other?
Results do not move between the two tools automatically, because garak writes a JSONL report of every attempt plus a hit log, while PyRIT stores conversations, scores and results in its memory layer backed by SQLite or Azure SQL. Sharing is possible when your team maps both outputs to a common schema, such as a shared finding ID, probe or attack name, target version, score and reviewer decision. That mapping is a small integration task but it needs an owner. If you also plan to use promptfoo, which runs red team plugins aligned with the OWASP LLM Top 10 and NIST AI Risk Management Framework, decide the common schema before the first run.
How DeployOpen helps teams choose and run garak or PyRIT
DeployOpen helps teams decide whether garak, PyRIT or both fit their assessment or in-house red team work, then installs and documents the chosen tool inside the client's own infrastructure. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- We confirm whether the work is a one-off assessment or an ongoing in-house red team program.
- We list the target model endpoints, generators and authentication paths the tool must reach.
- We agree in writing which attack categories, detectors or multi-turn strategies are in and out of scope.
How we build
- We install and configure the chosen tool on infrastructure the client controls, including its dependencies.
- We wire probes or attack scripts to the client's own targets and set up result storage for reviewable evidence.
- We test the setup against a known target first so findings can be trusted before wider runs.
How we hand over
- We deliver run instructions, configuration files and a record of how each test was set up.
- We document how to read JSONL reports, hit logs or the memory store, and how to triage a finding.
- We train the people who will run the tests, so the team can repeat assessments without us.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover tool version upgrades, so the team tracks each release and its configuration changes.
- We confirm with the client which support level and response expectations apply before a retainer starts.
