Open-source LLM red teaming tools compared

garak is a command-line scanner assembled from five kinds of plugin. Probes generate attacks, detectors judge whether the model produced undesirable output, generators connect to the target (Hugging Face Hub models, OpenAI, AWS Bedrock, llama.cpp models, or any REST endpoint described in a short YAML file), evaluators and harnesses run the combination. The default probewise harness runs each probe with its recommended detectors, and garak makes 5 generations per prompt by default, so a probe is marked FAIL with a failure rate when any attempt produces undesirable output. Each run writes a JSONL report of every attempt and its evaluation, plus a garak.log file that continues across runs. Your team should check which probes and detectors apply to your target before trusting a clean run, and confirm the generation count you want in the configuration.

PyRIT is a Python framework rather than a finished scanner. Its building blocks are targets, converters, scorers, attacks, memory and datasets. Targets include OpenAI, Azure, Anthropic, Google and Hugging Face models, custom HTTP or WebSocket endpoints, and web applications driven through Playwright. Scorers can use an LLM, Azure AI Content Safety or custom logic, and memory stores conversations and results in SQLite or Azure SQL. You can run it through the pyrit_scan command, through CoPyRIT for human-led testing, or by writing custom attacks in Python. That flexibility suits a security engineering team that wants to script multi-turn attacks, but it also means someone must own the code, the scoring logic and the storage decisions. Confirm which Python version your environment uses, since the documentation recommends Python 3.13.

promptfoo takes a configuration-first approach. You describe the application, providers, test cases and red team settings in declarative config files and run them locally as a CLI, from a Node.js library, or as a step in a CI/CD pipeline. The documentation states that evals run on your machine and talk directly with the model. The self-hosted web app is an Express server with a database, but its own documentation says the bundled SQLite store cannot be shared across replicas and lacks RBAC and SSO, so it is not recommended for production use. If you plan to share results across a team, plan for the Enterprise or On-Premise tier or a different storage design, and confirm those terms with the vendor.

All three core tools carry permissive open-source licenses. garak is Apache-2.0, PyRIT is MIT and promptfoo is MIT, so none of them requires a commercial agreement to run probes against your own applications. The license question that matters more here is who maintains the project. PyRIT development moved from the archived Azure/PyRIT repository to microsoft/PyRIT, and promptfoo is now part of OpenAI, although its README states it remains open source and MIT licensed. Ask your security team to confirm that the version you pin is the one the maintainers still ship, and to record the release you tested against in your change log.

promptfoo is the only one of the three with a paid tier on the same product. Its Community edition is free, includes all evaluation features and all model providers, and allows red teaming at 10k probes per month with community support. Enterprise adds team sharing, access control through teams and SSO, custom roles and permissions, continuous monitoring, a centralized guardrail dashboard, remediation recommendations and priority support with SLAs, priced on request. On-Premise runs that Enterprise functionality on your own infrastructure, with complete data isolation and a dedicated deployment engineer. Pricing for both is custom, so the quote should state the seat model, probe volume, support tier and whether On-Premise installation and upgrades are included.

Be careful with a shared self-hosted deployment on the free path. The self-hosting documentation says the promptfoo app stores data in a local SQLite database that cannot be shared across replicas, and that it has no RBAC, no horizontal scaling and no built-in authentication or SSO, and is not recommended for production. If several red teamers or a product team will share results, treat that setup as a review item: either accept single-user operation, put access control in front of it under your own controls, or confirm with promptfoo whether the Enterprise or On-Premise tier covers your requirement. Check the current details on the promptfoo pricing page before you decide.

Before you commit to garak, PyRIT or promptfoo, confirm that the probes you run match the risks in your application. garak ships probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks, and PyRIT ships multi-turn strategies such as Crescendo, TAP and Skeleton Key, but neither list is a statement that your application's specific workflows, tools or data paths are covered. Map each risk your team has agreed to test (for example, whether a retrieval step can leak documents across users) to a named probe, converter or plugin, and mark any risk with no match as a gap to test manually. Then confirm that the generator or connector you need actually reaches your target. garak's REST generator depends on a YAML description of your endpoint, PyRIT needs a custom HTTP or WebSocket target or a Playwright-driven web app target, and promptfoo needs a configured provider. Check authentication, network paths and response formats in a staging environment before you plan a test schedule around any of them.

Next, decide where results will live and who may read them. garak writes a JSONL report of every attempt and a hit log, and PyRIT stores conversations and scores in SQLite or Azure SQL, so both produce sensitive records that may contain the prompts and outputs that broke your application. Ask who owns those files, how long they are kept, and how they reach a ticketing or risk system. For promptfoo, the self-hosted app stores data in a local SQLite database that cannot be shared across replicas and lacks RBAC and SSO, and its documentation says it is not recommended for production. If several teams must review results from one deployment, confirm with promptfoo whether Enterprise, with team sharing, access control and SSO, is required, and get that answer in writing. Finally, ask each maintainer how security fixes are published and how quickly they are handled. Check release notes and repository security policies, and record the current versions you have tested, since the maintainers' own release pages are the authoritative source for what changed.