promptfoo vs PyRIT: which to choose

promptfoo vs PyRIT: which to choose for release testing or deep red team work

If you need a repeatable check that runs on every release and fails the build when an LLM application regresses, choose promptfoo. If your main goal is a security engineer scripting multi-turn attacks against a model or application and studying how it breaks over a long conversation, choose PyRIT. Most teams with an LLM application on their shortlist are deciding between these two because they want one of the two jobs done well, and the answer depends on which job comes first. Both are open source under the MIT license, so license alone will not separate them.

promptfoo fits when the test suite should live in version control as declarative configs and run in CI/CD. It runs locally from config files, and its red team plugins and strategies are aligned with the OWASP LLM Top 10 and the NIST AI Risk Management Framework, which makes it easier to show coverage to a risk committee. This matters most when several engineers change prompts, models or retrieval settings and you need the same checks to run on each change. Confirm in the promptfoo documentation that the CI integration and failure behavior match your pipeline before you commit to it as a gate.

PyRIT fits when the work is a Python-driven exercise rather than a pass or fail check. Its targets, prompt converters, scorers and multi-turn attacks such as Crescendo, TAP and Skeleton Key let a red teamer chain techniques and adapt the next turn based on the last response, and its memory layer keeps the full conversation for review. This matters when the risk is an attacker who adapts over many turns and when you have Python skills on the security team. Note that the original Azure/PyRIT repository is archived, so confirm you are tracking microsoft/PyRIT. The decision turns on whether the goal is gating releases or running exercises by security engineers.

Which tool fits which testing scenario

This table maps common testing scenarios to the tool that fits them better, with a short reason for each. Read it by matching your team's day-to-day work to a row, then check the reason against your own pipeline and staffing before choosing.

Scenario	Better fit	Why
Gating every pull request with a pass or fail check	promptfoo	Declarative configs run from the CLI in CI/CD, and its documented failure flags suit a release gate.
Nightly regression runs on a prompt library	promptfoo	Fixed config files and repeatable eval runs make it simple to compare prompt versions over time.
Red team exercise led by a security engineer over many turns	PyRIT	Multi-turn strategies such as Crescendo, TAP and Skeleton Key are built in, and the engineer can script and adjust each turn.
Testing a web app through a browser	PyRIT	PyRIT drives web app targets through Playwright, which covers interfaces that have no clean API.
Team that already writes Python test suites	PyRIT	Attacks, scorers and memory are Python objects, so they fit existing pytest-style code and custom logic.
Team that needs a shared web interface for human testers	PyRIT	CoPyRIT provides a web GUI for human-led red teaming and collaboration, while the free self-hosted promptfoo server has no built-in authentication or RBAC.
Team that needs team sharing, SSO and RBAC	promptfoo Enterprise	Team sharing, SSO and role-based access are Enterprise features, while self-hosted promptfoo lacks RBAC and SSO, so confirm the edition you need before choosing.

How promptfoo and PyRIT are built and what that means for operations

promptfoo and PyRIT both test LLM applications, but they ask for different operating models. One is a Node CLI driven by config files, the other a Python framework you script and host yourself, and each design choice shapes who runs it and where the results live.

promptfoo as a local CLI

promptfoo is a Node CLI and library driven by declarative configs. Evals run on the machine that starts them and talk directly to the model under test, so no intermediate service sits between your test data and the provider. For a platform team, this means the pipeline runner needs Node.js 22.22.0 or newer, network access to each model endpoint, and provider credentials stored as CI secrets. Confirm which providers your runners can reach before choosing this path.

promptfoo self-hosted server limits

The self-hosted promptfoo app is an Express server with a web UI and API, deployable with Docker, Docker Compose or Helm on Kubernetes. Its data sits in a local SQLite database that cannot be shared across replicas, and the self-hosted build has no RBAC, horizontal scaling, built-in authentication or SSO. The vendor does not recommend it for production. Plan for a single-instance service if you need a shared dashboard, or ask about the Enterprise or On-Premise tiers for multi-user access control.

PyRIT as a Python framework

PyRIT is built from targets, converters, scorers, attacks, memory and datasets, and it installs with pip. Each attack is code you write or adapt, so the team owns the attack logic as well as the test harness. The datasets cover content harms, psychosocial risks and data leakage. Expect to maintain Python test code the way you maintain any internal tool, and check that the attack strategies you rely on match the version you pin.

PyRIT memory and storage

PyRIT memory records conversations, scores and results in SQLite or Azure SQL. For a single analyst running local exercises, SQLite keeps everything on one machine and needs no database service. For shared or long-running campaigns, Azure SQL is the documented alternative, which means a database to provision, back up and grant access to. Decide early where red team transcripts are allowed to live, because they often contain sensitive prompts and model outputs.

PyRIT's three entry points

PyRIT runs as a scanner through the pyrit_scan command, as CoPyRIT, a web GUI for human-led red teaming and team collaboration, or as a Python framework for custom attacks. Each entry point suits a different operator: pyrit_scan fits scheduled runs, CoPyRIT fits a tester working through a multi-turn exercise with colleagues, and the framework fits engineers building new attack flows. Pick the entry point your operators will actually use, then confirm it meets your access control needs in your own environment.

PyRIT Python version constraint

The PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13. Pinning to the recommended version keeps your CI images and your developer machines aligned. If your organization standardizes on an older or newer interpreter, check that interpreter is inside the supported range before you commit to PyRIT, since a mismatch will surface as an install or runtime failure rather than a warning.

promptfoo vs PyRIT on attack coverage, CI fit and extensibility

This table compares promptfoo and PyRIT on licensing, ownership, configuration style, attack coverage, CI fit and extensibility. Read each row against your own release process and red team workload, and treat rows marked for confirmation as questions to put to the maintainers before you commit.

Capability	promptfoo	PyRIT
License	MIT licensed	MIT licensed
Maintainer and ownership	Part of OpenAI after the March 9, 2026 announcement; README states it remains open source and MIT licensed	Development continues at microsoft/PyRIT after the Azure/PyRIT repository was archived on March 27, 2026
Latest version cited	0.124.1, published October 8, 2026	v1.1.0, published September 4, 2026
Language and install	Node.js 22.22.0 or newer, distributed through npm	Python with pip; PyPI package supports Python 3.10 up to but not including 3.15, documentation recommends 3.13
Test definition	Declarative configuration files run from the CLI or library	Python code that composes targets, converters, scorers and attacks
OWASP LLM Top 10 and NIST AI RMF alignment	Red team plugins and strategies documented against both frameworks and the EU AI Act	Datasets cover content harms, psychosocial risks and data leakage; confirm framework mapping in the documentation
Multi-turn strategies	Confirm the current list of multi-turn strategies in the red teaming documentation	Crescendo, TAP and Skeleton Key documented as multi-turn attack strategies
Scorer options	Confirm which graders and custom checks each plugin uses	LLM-based scorers, Azure AI Content Safety or custom logic
CI/CD use	Documented for pipelines; runs checks from declarative configs	Scanning through the pyrit_scan command; confirm pipeline gating patterns before adopting
Targets	Wide provider coverage per the project's documentation; confirm your specific model and endpoint	OpenAI, Azure, Anthropic, Google, Hugging Face, custom HTTP or WebSocket endpoints, and web apps through Playwright
Memory and results storage	Local evaluation output; self-hosted app uses a local SQLite database not suited to production	Conversations and results stored in SQLite or Azure SQL
Web interface	Self-hosted web UI and API served by an Express app; Enterprise adds team features	CoPyRIT web GUI for human-led red teaming and team collaboration
Identity and access features	Self-hosted server has no RBAC, built-in authentication or SSO; SSO and teams access control are Enterprise features	Confirm authentication and access control for CoPyRIT in your deployment

What drives the running cost of promptfoo and PyRIT

Neither core tool carries a license fee. promptfoo is MIT licensed and PyRIT is MIT licensed, so the money you spend goes to the people who run them, the model calls they make, the compute that executes them and, for promptfoo, any paid tier you choose. The largest recurring line is usually the target and attacker model calls. Red team runs send many generated prompts to your application and to an LLM that judges the answers, and each of those calls is billed by your model provider. Probe volume and the number of attack strategies per run set that bill directly, so cap it in the config or script before you schedule it in CI.

The one-time work is the engineering effort. For promptfoo, a platform or AppSec engineer writes the declarative config, selects the OWASP LLM Top 10 plugins and strategies that match your application, and wires the pipeline job. For PyRIT, an engineer writes Python attack scripts that choose targets, converters, scorers and multi-turn strategies such as Crescendo, TAP or Skeleton Key, and sets up the SQLite or Azure SQL memory layer. Recurring effort follows each upgrade, since both projects ship new releases and a changed target or dependency can break a working setup. If you run promptfoo on your own servers, the platform team also carries the self-hosted app, which the promptfoo documentation says uses a local SQLite database and lacks RBAC and SSO, so it is not recommended for production. Confirm that before budgeting a shared server.

The Community edition of promptfoo includes red teaming with 10k probes per month, so a small team may never see a licensing cost. Enterprise adds team sharing, SSO, RBAC, continuous monitoring and priority support at custom pricing, and On-Premise runs that functionality in your infrastructure with a dedicated deployment engineer. When you request a quote, ask for the annual subscription, whether it is priced per seat, per application or per probe volume, what the probe allowance is above the Community limit, what support and SLA terms apply, and whether On-Premise includes deployment and upgrade work or only license access. For PyRIT, there is no vendor quote. Ask your model provider and any Azure services you plan to use for scorers or memory about per-token and per-request rates, and set a probe budget per pipeline run.

What promptfoo Community, Enterprise and On-Prem include

The Community edition is the free tier of promptfoo and is the starting point for most teams that want declarative tests in a pipeline. It includes all LLM evaluation features, all model providers and integrations, and red teaming with a monthly allowance of 10k probes. Support is community based, so there is no contracted response commitment. Because promptfoo runs locally and talks directly to the model, the work runs on hardware you control, and your team pays for that compute and the engineering time to maintain configs.

Evaluation features: all LLM evaluation features are included, so prompt and model comparisons do not require a paid plan.
Provider coverage: all model providers and integrations are included in the free tier.
Red team allowance: red teaming includes 10k probes per month, which bounds how much adversarial testing a single month can cover.
Support model: support is through the community, which suits teams that can resolve issues internally.
Local execution: evals run on your machine and call the LLM directly, so the free tier does not require a vendor-hosted control plane.

Confirm with promptfoo how the 10k monthly probe allowance is counted across CI runs and developer machines, and whether it changes with usage.

How implementing each tool involves the team and where projects run long

Implementing promptfoo starts with a declarative config that names the application under test, the test cases, and the red team plugins and strategies to run. Someone on the team must map those plugins to the OWASP LLM Top 10 and the NIST AI Risk Management Framework, so each failing test traces to a risk your risk owners already track. The CI runner needs Node.js 22.22.0 or newer, and the model credentials belong in the pipeline's secret store. The job is then wired into GitHub Actions, GitLab CI or Jenkins. The hardest decision is the pass or fail threshold: which failure classes block a merge, which only report, and who approves a change when a new model version shifts the baseline. Expect the first runs to need triage before thresholds settle. Confirm the exact CI steps in the promptfoo red teaming documentation.

PyRIT is a Python project, so the team needs an engineer who can build and maintain the environment. The PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13, so pin that version in the build image and do not leave it to the runner default. The main design work sits in code: choosing targets (OpenAI, Azure, Anthropic, Google, Hugging Face, custom HTTP or WebSocket endpoints, or web apps driven through Playwright), writing prompt converters that transform attack payloads, and building scorers that use an LLM, Azure AI Content Safety or custom logic. Then select a multi-turn strategy such as Crescendo, TAP or Skeleton Key and tune it against your application. Decide early whether memory lives in SQLite or Azure SQL, since that choice sets where conversations, scores and results are kept. See the PyRIT documentation for the current component list.

Projects usually run long in four places. First, the attack scope: agree in writing which applications, data, harm categories and attack depth are in bounds before anyone runs a multi-turn attack against a live system. Second, scorer tuning: a judge that is too strict floods the team with false positives, and one that is too lenient hides real failures, so budget time to calibrate scorers against labeled examples. Third, shared access: self-hosted promptfoo stores data in a local SQLite database that cannot be shared across replicas, has no RBAC or built-in SSO, and is not recommended for production. PyRIT's default memory is also local, so confirm how CoPyRIT handles accounts before a team shares it. Fourth, these limits decide whether you need Enterprise tiers or a custom data store. DeployOpen's fixed-scope engagements agree deliverables and timelines in writing before work starts, so these choices get made up front.

Moving from promptfoo to PyRIT or back, step by step

This sequence moves an LLM test program between promptfoo and PyRIT without losing attack scope or the history needed to judge whether a release got safer. It ends with one CI gate running on the new tool and an archived record of the old one.

Inventory existing tests, plugins and attack goals
The engineer who owns the promptfoo configs lists every test case, red team plugin, strategy and custom assertion in use, along with the business risk each one covers. Record which OWASP LLM Top 10 categories and NIST AI RMF functions the current suite maps to, and note any custom datasets referenced from the config. The step is done when the inventory is a single table that a security lead has reviewed and signed off, with each row tied to a release gate or a periodic scan.
Map each plugin and strategy to PyRIT components
Work through the inventory row by row and match each promptfoo plugin or strategy to a PyRIT attack, prompt converter, scorer or dataset. Where the match is partial, such as a single-turn plugin with no multi-turn equivalent, or a promptfoo assertion with no clear scorer, write the gap down rather than forcing a match. The output is a mapping document with three columns: covered, partially covered and not covered, reviewed by the engineer who will run the new suite. Confirm each PyRIT component against the PyRIT documentation for the version you will pin.
Rebuild targets in the new tool
Recreate each system under test as a PyRIT target, using the built-in OpenAI, Azure, Anthropic, Google or Hugging Face target types, a custom HTTP or WebSocket endpoint, or a Playwright-driven web app target where the application has no API. Keep the endpoint, authentication method and system prompt details in version control, not in notebook state. The step closes when a single benign prompt reaches each target and returns a response that is stored in PyRIT memory. Confirm that credentials are injected from your secrets store, not hard-coded.
Re-baseline results so findings can be compared
Run the rebuilt attacks against the current production build and the last release candidate, then record the results as the new baseline. Because promptfoo and PyRIT write different result formats, export the promptfoo findings history and build a mapping table from promptfoo result fields to PyRIT memory records, keyed by target, attack goal and date. Without this mapping, a before and after comparison cannot be made across tools. The step is complete when a reviewer can trace any historical finding to its equivalent in the new store.
Run both tools in parallel for one release cycle
Keep the promptfoo CI job active while the PyRIT job runs on the same builds, and compare the two verdicts for each release candidate. Log every disagreement, where one tool flags a failure and the other passes, and resolve it by reviewing the prompt, the response and the scorer logic. The team should agree in advance on what counts as a match. The step ends when one full release cycle has passed with no unexplained disagreements, and the owner signs off on the mapping.
Move the CI gate last
Only after parallel runs agree, switch the release gate to the new tool. Rebuild the pipeline job in your CI system, define the pass or fail threshold, and confirm that a failing run blocks the merge or deployment as intended. Test the gate with a deliberately vulnerable build in a non-production environment. The step is done when a failing test blocks a pipeline run in a test branch and the job log shows which attack and scorer caused the failure.
Archive the old configs and results
Once the new gate has run successfully for the agreed period, freeze the promptfoo configs, plugin lists and exported results in a read-only location with a note on the tool version used. Keep the mapping document and the baseline alongside them so auditors and future teams can reconstruct why a finding was classified as it was. Confirm the retention period with your compliance or records owner, because that period is set by your organization, not by either tool. The step closes when the archive can be opened and read without the original tool installed.

Questions teams ask before choosing or hiring help for promptfoo or PyRIT

Can our team run promptfoo or PyRIT without outside help?

Yes, if you have an engineer who can own the tool and the test suite. promptfoo runs locally from declarative configs and can run checks in CI/CD pipelines, so a team with Node.js experience can usually start with a small config and a single pipeline job. PyRIT installs with pip and is a Python framework, so it suits a team that writes Python and is comfortable building custom attacks and scorers. The harder work is ongoing: choosing which attacks matter for your application, interpreting results, keeping model and dependency versions current, and deciding what blocks a release. Ask whether those responsibilities sit with a named person before you decide who does the setup.

What does promptfoo or PyRIT work cost, and what drives it?

The software is open source under the MIT license for both tools, so the license itself is not the main cost. The spend comes from people time, model API usage during tests, and infrastructure to run scans and store results. Engineering time to write configs, wire CI, and tune attacks is typically the largest one-time item, while model calls and scan volume recur with every run. promptfoo Community includes red teaming with 10k probes per month, and promptfoo Enterprise is priced by custom quote according to its pricing page. If you hire help, the quote should follow a written scope that lists the applications covered, the attack classes, the CI integration, and the reports you expect. DeployOpen does not publish a price list, and fixed-scope work is agreed in writing before it starts.

How long does setup take for promptfoo or PyRIT?

No setup duration is established in the sources for this comparison, so confirm it against your own environment. The main drivers are how many LLM applications you test, how many providers and models sit behind them, whether your application needs custom HTTP or browser-driven targets, and how much of your existing test data you can reuse. A promptfoo gate for a single application with a standard provider is usually a smaller job than a PyRIT harness that drives a multi-turn web application through Playwright. Ask any vendor or partner for a timeline tied to a written scope, and ask what they will need from your team before the work begins.

What does our team need to provide for these tools?

The requirements differ by tool, and you should confirm the current versions before you plan. For promptfoo, the npm release 0.124.1 requires Node.js 22.22.0 or newer. For PyRIT, the PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13. Both tools need model API keys for the providers you test, and both need access to the CI system where scans run. Plan for a named owner who manages keys, approves scan targets, reviews findings, and keeps the suite aligned with release cycles. Without that owner, the suite tends to drift and results stop being acted on.

Can the self-hosted promptfoo server serve a production team?

Not on its own, according to the self-hosting documentation. The self-hosted app is an Express server with a web UI and API that can be deployed with Docker, Docker Compose or Kubernetes with Helm, but it stores data in a local SQLite database that cannot be shared across replicas. It has no RBAC, horizontal scaling, built-in authentication or SSO, and the documentation says it is not recommended for production use. The Enterprise tiers add team sharing, access control, SSO, continuous monitoring and a centralized guardrail dashboard, and On-Premise runs that functionality on your own infrastructure with complete data isolation. Confirm the current feature set with promptfoo before you plan a shared deployment.

Does PyRIT fit a team that already has Python tests?

Yes, when your team wants attack logic to live next to its existing code. PyRIT is built from targets, prompt converters, scorers, multi-turn attacks such as Crescendo, TAP and Skeleton Key, and a memory layer backed by SQLite or Azure SQL. A team with pytest-style suites can call PyRIT from Python and reuse its fixtures and reporting conventions. The cost is that scorers, attack strategies and memory need to be understood before results can be trusted. Also check whether your gate needs declarative configs that non-Python staff can edit, because that is where promptfoo is usually easier to hand over.

Are archived AI guardrail tools like LLM Guard still relevant to this choice?

LLM Guard is relevant only as a warning. Its repository was archived on GitHub on July 9, 2026, and it is no longer maintained, so it receives no security fixes. If your current stack depends on it, plan a migration to an actively maintained tool and keep it out of new designs. Neither promptfoo nor PyRIT is a runtime guardrail library, so this choice is about testing, not production filtering. Confirm the maintenance status of any tool you add to your pipeline before you rely on it.

Can promptfoo and PyRIT be used together?

Yes, they can be used together, although you should first decide which one owns each decision. A common split is to run promptfoo as a pipeline gate on every release, using its declarative configs and OWASP LLM Top 10 plugins, and to run PyRIT on a schedule for deeper multi-turn exercises with custom converters and scorers. Two tools mean two sets of results, two dependency trees and two owners, so document which findings block a release and which go to a review queue. Start with the gate, then add the deeper suite once the team has a stable process for triage.

How DeployOpen helps choose, implement or migrate between these tools

DeployOpen plans, implements, integrates and documents self-hosted open-source platforms inside your own infrastructure, so the choice between promptfoo and PyRIT can be tested against your release process and your red team workload. Engagements are fixed-scope, with scope and deliverables agreed in writing before work starts.

How we scope

Confirm whether your need is release gating in CI, periodic deep red team exercises, or both before choosing a tool
Define which LLM applications, model providers and endpoints the written scope will cover
Agree in writing which OWASP LLM Top 10 areas and internal test categories each tool must exercise
Document where test runs, reports and any stored conversation data will live in your environment

How we build

Set up promptfoo declarative configs and pipeline checks in your CI system, with failure thresholds you approve
Configure PyRIT targets, converters and scorers against your endpoints, with memory storage set up for your team
Confirm the Node.js and Python runtimes, package versions and network access each tool needs on your systems
Test the setup against agreed sample targets before any production application is added to the scope

How we hand over

Deliver runbooks covering how to add test cases, read reports and rerun scans after a model or prompt change
Document the attack coverage each tool was configured to run, and the gaps the written scope leaves open
Provide a record of versions, licenses and configuration decisions so your security and compliance leads can review them
Walk your engineers through pipeline results and PyRIT scan or CoPyRIT workflows before the engagement closes

How we support

Ongoing support, upgrades and continuous testing are available as retainers, agreed separately from the initial scope
Plan upgrades when promptfoo or PyRIT publish new releases, testing changes in a staging environment first
Review red team coverage periodically as your applications, models and OWASP-aligned test needs change
Ask any quote to itemize setup, pipeline integration, handover and retainer work, so the scope stays comparable

promptfoo vs PyRIT