Which of garak, PyRIT and promptfoo fits your LLM assessment
If your team has shortlisted garak, PyRIT and promptfoo and needs to test an LLM application before release, use garak for model-level vulnerability scans, PyRIT for scripted multi-turn attacks run by an in-house team, and promptfoo for config-driven application tests in CI. Most typical assessments combine two of these tools rather than relying on one, because each one answers a different question about the system. Context to weigh, not a verdict: promptfoo announced on March 9, 2026 that it is being acquired by OpenAI and remains MIT licensed, and the original Azure/PyRIT repository was archived on March 27, 2026, with development continuing at microsoft/PyRIT.
Start by deciding who will run the tests and what evidence the release process needs. If the question is whether the underlying model is susceptible to prompt injection, jailbreaks or leakage, garak's probes and detectors give a direct answer, and its JSONL report records every attempt. If your red team needs to script a Crescendo or Skeleton Key style conversation against a custom endpoint and keep scored results in SQLite, PyRIT's memory layer and Python framework are the better fit. If developers need to keep prompts, providers and assertions in version-controlled config and fail a pipeline on regression, promptfoo is designed for that workflow.
Before committing, confirm three things with the vendor or your own environment: the Python and Node.js versions your build agents can run, since garak needs Python 3.11 or newer, PyRIT supports Python 3.10 through 3.14, and promptfoo needs Node.js 22.22.0 or newer; whether your target can be reached by the built-in generators or needs a custom REST configuration; and whether your compliance evidence must come from a hosted or self-hosted promptfoo deployment, since the self-hosted app stores data in a local SQLite database and lacks built-in authentication and SSO.
Which tool fits which assessment scenario
This table matches four common assessment scenarios to the tool whose design fits them best, and the reason rests on a supplied fact about how each tool works. Read the best fit first, then the why column to confirm the condition under which that choice holds in your environment.
| Scenario | Best fit | Why |
|---|---|---|
| One-off model scan | garak | garak is a command-line scanner that runs probes against a target and judges output with detectors, writing a JSONL report of every attempt and a hit log. A single run against a Hugging Face, OpenAI, Bedrock or REST endpoint produces a result without building a test project first. |
| In-house multi-turn red team | PyRIT | PyRIT is a Python framework of targets, converters, scorers and multi-turn attacks such as Crescendo, TAP and Skeleton Key, with memory in SQLite or Azure SQL. Teams that write custom attack logic and keep results across sessions get the most from its scripting model. |
| Release gate in CI | promptfoo | promptfoo runs locally from declarative configs and can run checks in CI/CD pipelines, so a pull request can fail on a regression in the application's prompts or tools. Confirm that the self-hosted version fits your pipeline, because its local SQLite store is not recommended for production. |
| Team that already runs a Python test suite | PyRIT | PyRIT is a Python framework, so its attacks and scorers can live beside existing pytest code and share its dependencies. Teams without Python test habits may find promptfoo's YAML-style configuration quicker to adopt, and should confirm this with their own engineers. |
How garak, PyRIT and promptfoo each test an LLM
garak tests the model itself. It is a command-line scanner built from five plugin types: probes that generate attacks, detectors that judge whether the output failed, evaluators, generators that connect to the target, and harnesses that run them together. The default probewise harness runs each probe with its recommended detectors. By default garak makes 5 generations per prompt and marks a probe as FAIL when any attempt produces undesirable output. Each run writes a JSONL report of every attempt and its evaluation, plus a hit log of the attempts that found a weakness. A red team reads these files as evidence, so confirm early that your target can be reached through one of garak's generators or a REST endpoint described in its YAML configuration.
PyRIT tests a scripted attack. It is a Python framework, not a scanner. You compose targets (the system under test), prompt converters that transform payloads, scorers that judge responses, and attacks such as Crescendo, TAP and Skeleton Key that can run several turns. Its memory layer records conversations, scores and results in SQLite or Azure SQL, so a multi-turn finding can be replayed and audited. Because you write the attack logic, PyRIT suits a team that wants to model a specific adversary path. The cost is that someone on your side must write, maintain and review that code, and you should confirm which Python version your environment uses against the PyRIT documentation before you commit.
promptfoo tests the application. It is a local CLI and library that runs evaluations and red team checks from declarative configuration files, and it can run the same checks in CI/CD pipelines. Its tests exercise your prompts, retrieval, tools and provider settings as the application wires them together, not the bare model. Evals run on your machine and talk directly to the LLM, so the configuration file becomes the reviewable artifact. Note that the self-hosted web app uses a local SQLite database without RBAC or built-in SSO, so a shared server deployment needs separate design work. Use the three tools together by layer: garak for the model, PyRIT for scripted multi-turn attacks, and promptfoo for the application in the release pipeline.
How the three tools compare on coverage, reporting and CI fit
This table sets garak, PyRIT and promptfoo side by side on license, maintainer, release, install, attack coverage, scoring, reporting and CI use. Read each row as a question to confirm against your own target and pipeline, since the supplied facts do not cover every capability. The main takeaway is that the three tools overlap less than their names suggest: garak scans a model, PyRIT scripts attacks, and promptfoo tests an application.
| Capability | garak | PyRIT | promptfoo |
|---|---|---|---|
| License | Apache-2.0 | MIT | MIT |
| Maintainer | NVIDIA, in the NVIDIA GitHub organization | Microsoft, at microsoft/PyRIT (original Azure/PyRIT repository is archived) | promptfoo, now part of OpenAI, remains open source |
| Latest supplied release | v0.17.1 (October 16, 2026) | v1.1.0 (September 4, 2026) | 0.124.1 (October 8, 2026) |
| Language and install | Command-line tool installed from PyPI with pip, Python 3.11 or newer | Python framework installed with pip, Python 3.10 up to 3.15 (not included), 3.13 recommended | CLI and library distributed through npm, Node.js 22.22.0 or newer |
| What is tested | Probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks against a model endpoint | Content harms, psychosocial risks and data leakage datasets, with custom attacks scripted in Python | Application-level evals and red team plugins aligned with OWASP LLM Top 10, NIST AI RMF and EU AI Act references |
| Multi-turn attacks | Not a focus of the supplied facts; confirm adaptive probe behavior in the user guide | Supported, including Crescendo, TAP and Skeleton Key | Confirm strategy coverage for multi-turn runs in the red teaming documentation |
| Scorers and judging | Detectors judge each output; a probe is marked FAIL when any attempt is undesirable | LLM scorers, Azure AI Content Safety or custom logic | Assertions and graders defined in declarative configs; confirm grader options for your checks |
| Reporting output | JSONL report of every attempt and evaluation, plus a hit log and a garak.log file | Results and conversations stored in SQLite or Azure SQL memory | Local results from declarative configs; the self-hosted app adds a web UI backed by SQLite |
| CI/CD use | Run as a command in a pipeline; confirm exit behavior and report handling for your gate | Scriptable from Python; pyrit_scan command is available for scanner-style runs | Designed to run checks in CI/CD pipelines from declarative configs |
| Local or self-hosted web UI | No web UI in the supplied facts; runs locally from the command line | CoPyRIT web GUI for human-led red teaming and team collaboration | Runs locally; self-hosted app serves a web UI, but SQLite storage is not recommended for production |
What is free and what is paid in each tool
garak is licensed under Apache 2.0 and is developed in NVIDIA's GitHub organization. The maintainers describe it as a free tool, and the supplied facts do not list a paid edition, hosted tier or enterprise license for it. The cost of running garak is therefore the cost of the infrastructure, model access and staff time you bring to it, not a license fee.
- License: Apache 2.0, which permits commercial use and internal modification under its terms. Confirm the terms against the LICENSE file in the repository before redistributing modified builds.
- Maintainer: NVIDIA maintains the project in its GitHub organization, and v0.17.1 was released on 16 October 2026.
- Paid edition: None appears in the supplied facts. Ask NVIDIA or the garak maintainers directly before assuming one will never exist.
- Recurring costs: Target model API usage, compute and the time spent reviewing JSONL reports and hit logs. Probes generate many attempts, so provider usage can grow with scope.
- Support: Community support through the project. Commercial support, if needed, would come from a separate services agreement.
Before budgeting, check the garak user guide at docs.garak.ai for current terms and confirm your target provider's per-token pricing for the probe volume you plan to run.
PyRIT is an MIT-licensed framework from Microsoft. The original Azure/PyRIT repository was archived on 27 March 2026 and is read-only, so development now happens at microsoft/PyRIT. Teams that cloned or pinned the old repository should move their references and confirm which release line they depend on, since v1.0.0 and v1.1.0 were published under the new location.
- License: MIT, which is permissive for commercial and internal use. The license covers the framework code, not the model providers you connect to.
- Repository move: The archived Azure/PyRIT repository is read-only. Current releases, including v1.1.0 published on 4 September 2026, come from microsoft/PyRIT.
- CoPyRIT: Described in the PyRIT documentation as a web GUI for human-led red teaming and team collaboration. The supplied facts do not state whether CoPyRIT carries a separate license or cost, so check its terms in the documentation.
- Azure service: The PyRIT group context mentions an Azure service built on the framework. Its pricing is separate from the open-source package and should be quoted on its own.
- Recurring costs: Model and scorer API calls, storage for SQLite or Azure SQL memory, and engineering time to write and maintain attack scripts.
Ask whether any paid support or hosted option exists, and confirm the license of CoPyRIT and any Azure-hosted offering before you assume they are included.
promptfoo is an MIT-licensed CLI and library. Its Community edition is free and includes all LLM evaluation features, all model providers and integrations, and red teaming with 10k probes per month, with community support. Paid editions sit above that. Promptfoo announced on 9 March 2026 that OpenAI agreed to acquire it, and the project states it remains open source and MIT licensed.
- Community edition: Free, with red teaming capped at 10k probes per month according to the promptfoo pricing page.
- Enterprise: Adds team sharing, teams-based access control and SSO, custom roles and permissions, continuous monitoring, a centralized guardrail dashboard, remediation recommendations and priority support with SLAs. Pricing is custom.
- On-Premise: Runs the Enterprise functionality on your own infrastructure with complete data isolation and a dedicated deployment engineer. Pricing is custom.
- Self-hosted open source: The self-hosted app uses local SQLite, has no RBAC, built-in authentication or SSO, and is not recommended for production use, so access controls you need may require the paid tiers.
- Recurring costs: Enterprise and On-Premise are subscriptions quoted by promptfoo, plus model usage for every test run.
Ask promptfoo for a written quote that states what is covered, the probe allowance, the SLA terms and whether the deployment engineer is included.
What drives the running cost of each tool
Running these three tools costs less in licenses than in people and model calls. The cards below cover the cost drivers that differ most between garak, PyRIT and promptfoo, so you can ask the right questions before you budget.
Engineering time for attack scripts
PyRIT is a Python framework, so most of its running cost is the engineer who writes and maintains the attack scripts, converters and scorers. Multi-turn strategies such as Crescendo, TAP and Skeleton Key need someone who understands the target well enough to tune them. Ask who will own those scripts after the first assessment, and whether the team has the Python skills to update them when targets change. Garak and promptfoo need less scripting for a first pass, but custom probes or configs still take engineering time.
Model API spend per garak run
Garak sends each prompt to the target model several times. Its default is 5 generations per prompt, so a full scan multiplies your API calls by that factor before any custom settings. Every generation is billed by the model provider if the target is a hosted API. Lower the generations for smoke tests, then raise them for the release gate. Confirm the provider's per-token rate and your rate limits before you schedule scans in CI.
Promptfoo probe volume limits
The Community edition of promptfoo is free and includes red teaming, but it caps red team probes at a monthly allowance. Each CI run that generates adversarial inputs draws from that allowance, so a pipeline that runs on every pull request can exhaust it early in the month. Map your expected run frequency to the limit before you wire promptfoo into a pipeline. If you need more volume or team features, the paid tiers are the next step, and their pricing is set by quote.
Self-hosted promptfoo storage
Self-hosted promptfoo keeps its data in a local SQLite database that cannot be shared across replicas. It has no RBAC, horizontal scaling or built-in authentication or SSO, and the documentation says it is not recommended for production use. If several teams or pipelines need shared results, you must add a shared database and an access layer yourself, which is engineering work with its own upkeep. Ask the vendor or your platform team to confirm which parts they would build and support.
Enterprise pricing set by quote
Promptfoo Enterprise adds team sharing, SSO, custom roles, continuous monitoring and priority support with SLAs. Its pricing is custom, so the number you pay depends on your scope and the features you need. The On-Premise option runs the Enterprise functionality on your infrastructure with a dedicated deployment engineer, which changes the cost mix toward services. Request a written quote that lists the seats, support level and deployment model separately so you can compare it with the engineering cost of building those features in-house.
Upgrades across Python and Node.js
Garak and PyRIT run on Python, while promptfoo runs on Node.js and needs version 22.22.0 or newer. Garak's PyPI package requires Python 3.11 or newer, and PyRIT supports Python 3.10 up to but not including 3.15. Each runtime upgrade can break dependencies, and the tools release on their own schedules, including garak v0.17.1 and PyRIT v1.1.0. Budget recurring time for pinning versions, testing upgrades in a staging pipeline and rerunning baseline scans after each change.
How to roll out garak, PyRIT and promptfoo together
This sequence moves a team from a one-time model scan to repeatable, CI-gated checks for its LLM application, with reports kept for review. Each step names the owner and the artifact that proves it finished.
Fix the target and generator configuration
Your team records which model or application endpoint is under test and how each tool reaches it. For garak, that means choosing a generator such as Hugging Face, OpenAI, AWS Bedrock, llama.cpp or a REST endpoint described in a short YAML file. For PyRIT, it means a target class for the same endpoint, and for promptfoo, a declarative provider entry. Confirm with the model owner which endpoints are in scope and which credentials each tool may hold. The step is done when one written inventory lists the target, the generator or provider settings and the owner of each secret.
Run a garak baseline scan
Run garak from the command line against the fixed target using its default probewise harness, which runs each probe with its recommended detectors. garak makes 5 generations per prompt by default and marks a probe as FAIL when any attempt produces undesirable output, so the baseline is a broad, model-level view of hallucination, data leakage, prompt injection, toxicity and jailbreaks. Keep the JSONL report of every attempt and the garak.log file. The step is done when the baseline report is stored with the run date, the garak version and the probe list, so later runs can be compared against it.
Script multi-turn cases in PyRIT
The in-house red team turns the findings that matter for this application into scripted multi-turn attacks in PyRIT. PyRIT supports strategies such as Crescendo, TAP and Skeleton Key, uses converters and scorers, and records conversations and results in its memory layer, which is SQLite by default or Azure SQL. Your team chooses the scorer, either an LLM, Azure AI Content Safety or custom logic, and writes down why it judges a response as a failure. The step is done when the scripted attacks run end to end and their results are readable from memory, with the scorer choice documented.
Encode regression checks in a promptfoo config
Turn the cases that failed in the baseline and in PyRIT into declarative test cases in a promptfoo configuration, and add the red team plugins and strategies that match your risk framework, such as the OWASP LLM Top 10 or the NIST AI Risk Management Framework. promptfoo runs locally and talks directly to the model, so the config lives in your repository next to the application code. The step is done when the config runs on a developer machine and produces a pass or fail result for every encoded case, with the expected behavior written in each test.
Wire the checks into CI
Add the promptfoo run to your CI/CD pipeline so each change to the prompt, model or retrieval logic runs the regression set before release. Keep the garak and PyRIT runs on a schedule or on demand, because full scans take longer and are better suited to periodic review than to every commit. Your platform team confirms the runner has the Node.js version promptfoo requires and the Python version PyRIT and garak require, and that secrets are injected from your vault rather than stored in the repository. The step is done when a deliberate test failure blocks a merge in a test branch.
Set retention for reports and logs
Decide how long garak JSONL reports, garak.log files, PyRIT memory databases and promptfoo results are kept, and where. Because garak appends its log across runs and PyRIT memory is a database, the retention job needs to know both file locations and storage limits. If you self-host promptfoo, note that its self-hosted app stores data in a local SQLite database that cannot be shared across replicas and has no built-in RBAC or SSO, so results stay on one instance and access is controlled outside the tool. Ask your compliance lead which evidence the assessment needs. The step is done when the retention policy names each artifact, its storage location, its owner and its deletion date.
What implementing each tool takes in your environment
Start with the runtime each tool needs on the machine that will run it. garak installs from PyPI with pip and requires Python 3.11 or newer, and its maintainers develop it on Linux and macOS, so confirm that your build hosts match before you commit. PyRIT also installs with pip, supports Python 3.10 up to but not including 3.15, and its documentation recommends Python 3.13, which makes it the easiest of the three to pin to a single interpreter across a team. promptfoo installs from npm and requires Node.js 22.22.0 or newer. Ask your platform team whether Node.js and Python are already approved, supported and patched in your build images, because a version mismatch is the most common reason a first run fails before any attack is sent.
The self-hosted promptfoo path needs the most decisions. The app is an Express server with a web UI and API, deployable with Docker, Docker Compose or Kubernetes with Helm, but the self-hosting documentation says it stores data in a local SQLite database that cannot be shared across replicas, has no RBAC, no horizontal scaling and no built-in authentication or SSO, and is not recommended for production use. For a regulated team, that means a shared promptfoo server needs an access layer you supply, such as a reverse proxy with your identity provider in front of it, and a single-replica deployment with backups of the database file. Confirm with promptfoo whether On-Premise, which the pricing page describes as running Enterprise functionality on your infrastructure with a dedicated deployment engineer, is the better fit if several teams need shared access and SSO. Treat that as a commercial question, since its cost is set by promptfoo rather than published here.
The team effort is mostly in the parts the tools do not provide. Someone has to own the model endpoint credentials for each target, store API keys in your secrets tooling rather than in config files, decide where JSONL reports and garak.log files are kept, and define who reviews a FAIL before it reaches a release decision. PyRIT adds a memory layer in SQLite or Azure SQL that will hold conversation data, so its location and retention need the same review as any dataset that contains prompts and outputs. Plan for a named owner for each tool and a schedule to update it, since each project ships new releases frequently. Where you want a vendor-backed route to support, ask DeployOpen to scope the installation, hardening, CI wiring and report handling in writing before work starts, and ask each vendor which support channel covers your version.
Common questions about garak, PyRIT and promptfoo
How much effort does it take to run garak, PyRIT and promptfoo?
Each tool installs from a standard package manager, so the first run is quick, but the sustained effort depends on how often you test and how much you customize. garak installs from PyPI with pip and needs Python 3.11 or newer, and its probes and detectors run from the command line. PyRIT installs with pip, supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13. promptfoo runs from declarative configs and needs Node.js 22.22.0 or newer. The recurring effort comes from writing target configs, tuning attacks for your application, reviewing findings, and keeping each tool current with its releases. Ask any provider to estimate effort against your target count, your model access and your review process.
Can our team run these tools without outside help?
Yes, for the basics. All three tools are open source and run on your own machines, and the documentation for each covers installation and a first scan. Teams with Python or Node skills and a clear test target usually get a first pass running on their own. Outside help tends to add value in three places: writing custom generators or targets for internal APIs, interpreting findings against your risk model, and building a repeatable pipeline that your engineers can own. DeployOpen plans, implements and documents this kind of work as a fixed-scope engagement, with scope agreed in writing before work starts.
How long does a first assessment take, and what drives that time?
No fixed duration can be given here, because the time depends on the application. The main drivers are the number of model endpoints and tools under test, how many conversational or agent flows exist, how hard it is to connect each target (a plain REST endpoint is simpler than a web app driven through a browser), and how many findings need human review. garak reports a failure rate per probe with 5 generations per prompt by default, so runs against many probes take longer. Ask for a written scope listing targets, attack categories, the reviewers involved and the report format before agreeing a timeline.
What does the work need from our side?
Plan for four inputs. First, runtime: Python 3.11 or newer for garak, a supported Python version for PyRIT, and Node.js 22.22.0 or newer for promptfoo. Second, model access: API keys or endpoints for each target, or a local model for garak through llama.cpp. Third, a named reviewer who knows the application's intended behavior, because each tool flags possible failures and a person must judge them. Fourth, storage for reports and logs, since garak writes a JSONL report and a garak.log file, and PyRIT keeps results in SQLite or Azure SQL. Confirm network paths and data handling rules with your security team before sending prompts to any external provider.
Does promptfoo's ownership by OpenAI change its license?
No. promptfoo is MIT licensed, and the project's README states that promptfoo is part of OpenAI and remains open source and MIT licensed. The founders' announcement dated March 9, 2026 says it will remain open source and continue to support a diverse range of providers and models. The ownership question still matters for procurement, though. Review how OpenAI's roadmap could shape future editions, and confirm the terms of any paid Enterprise or On-Premise contract directly with the vendor, since those are priced separately and are not covered by the MIT license.
Should we still use LLM Guard given that it was archived?
Not for new work. LLM Guard, Protect AI's MIT-licensed scanner, was archived on GitHub on July 9, 2026 and is no longer maintained, so it receives no security fixes. Existing deployments can keep running, but they will not get patches for newly found issues. If you use it today, plan a migration to a maintained option and check which scanners your application depends on. This comparison does not cover LLM Guard as a testing tool, and a separate review of runtime guardrail libraries should guide the replacement.
How should we combine the three tools in one assessment?
A common pattern is to split the work by what each tool does best. Use garak to scan the model endpoint for known weakness categories such as prompt injection, jailbreaks, data leakage and hallucination. Use PyRIT for scripted multi-turn attacks such as Crescendo, TAP or Skeleton Key against the full application, with scoring stored in its memory layer. Use promptfoo to keep application-level evals and red team checks in CI so that regressions are caught on each release. The tools write different outputs, so agree on one findings register and map each result to the same risk categories before reporting. This split is illustrative and should be adjusted to your application.
What should we ask for in a quote?
Ask for a written scope that names the targets, the tools and versions to be used, the attack categories covered, and what is excluded. Request the list of deliverables, such as the JSONL reports, the findings register, a remediation summary and a rerun of checks after fixes. Ask how the work is split between setup, custom connectors, test execution, review and reporting, and who is responsible for each. Confirm the timeline, the named reviewers, and whether ongoing support, upgrades or continuous testing are covered by a separate retainer. DeployOpen does not publish a price list, and a quote follows from the written scope.
How DeployOpen helps choose and implement these tools
DeployOpen plans, implements, customizes, integrates, hardens and documents self-hosted open-source platforms inside your own infrastructure, including security testing of AI systems. For garak vs PyRIT vs promptfoo, we start from your target application, your CI pipeline and your hosting constraints, then agree the tooling plan in writing before any work begins.
How we scope
- We agree in writing which models, endpoints and application flows each tool will test.
- We confirm where each tool runs, including your cloud accounts, data centers or air-gapped networks.
- We record which attack categories, frameworks and report formats your assessment must produce.
How we build
- We install and configure garak, PyRIT and promptfoo in your environment, pinned to the versions you approve.
- We connect generators, targets and endpoints so each tool reaches the system under test.
- We wire promptfoo or the chosen scanner into your CI pipeline and set the pass and fail criteria.
How we hand over
- We document configurations, run procedures, report locations and the triage steps for findings.
- We walk your security and platform engineers through running each tool and reading its output.
- We deliver the written record of decisions, so your team can rerun or change the setup later.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- A retainer can cover re-running assessments when models, prompts or endpoints change.
- We can review tool upgrades against your written scope before your team adopts new versions.
