PyRIT implementation plan and cost

What a PyRIT implementation involves

A PyRIT implementation turns the open-source framework into a repeatable red teaming capability for your own AI systems. PyRIT is an MIT-licensed Microsoft framework built from targets, converters, scorers, attacks, memory and datasets. Most of the effort goes into engineering those parts for your environment and designing test scenarios that reflect your product's risks. Installing the package is the smallest step. Note that the original Azure/PyRIT repository is archived, and development continues at microsoft/PyRIT, so every reference in your plan should point there.

Three triggers usually start the work. The first is a team's first AI red teaming program, which needs a defined test catalog, an owner and a review routine. The second is a new generative AI product, where tests should be tied to release gates. The third is a regulator or customer asking for evidence of adversarial testing, which requires documented methods, scorer configuration and retained results. Identify which trigger applies before you agree on scope, because it determines how much evidence handling and repeatability the build must include.

Teams with strong Python and security testing skills can often build a first setup in-house, starting with one target and a small set of attacks and scorers. Partners add the most value when the program covers many targets, custom endpoints or evidence requirements under a fixed scope. Ask any partner for a written list of targets, attack strategies, scorers, memory storage and deliverables before agreeing on price. Ask your own team to estimate the same list so the two options can be compared directly.

Confirm the Python version in your build image before writing any scenarios. The PyRIT documentation recommends Python 3.13, and the PyPI package supports Python 3.10 up to but not including 3.15. Check this against your approved runtimes and your security team's patch cadence, since a mismatch will surface as a dependency problem during the first run.

What the work includes

Each card is a deliverable your team can review before the build starts and again at handover. Together they form the basis of the written scope and quote.

Target inventory and connectors

List every system the red team may test, including model APIs from OpenAI, Azure, Anthropic, Google and Hugging Face, custom HTTP or WebSocket endpoints, and web apps driven through Playwright. Confirm each target's authentication method, rate limits and whether it is a test or production system before any connector is configured. The inventory becomes the scope record for the engagement.

Attack strategies and scenario library

Select single-turn and multi-turn strategies such as Crescendo, TAP and Skeleton Key for each target, then build reusable scenarios on PyRIT datasets for content harms, psychosocial risks and data leakage. Each scenario should name its objective, dataset, strategy and expected evidence so results stay comparable across test cycles.

Scorer configuration

Choose how responses are judged: LLM-based scorers, Azure AI Content Safety, or custom logic for checks specific to your application. Document each scorer's prompt or rule, the threshold that counts as a finding, and a sample of human-labeled responses used to measure agreement. Scorer choice drives both the review effort on false positives and model call spend.

Memory storage on SQLite or Azure SQL

PyRIT memory keeps conversations and results in SQLite or Azure SQL. SQLite fits a single analyst's workstation, while Azure SQL suits shared team use. Decide where the database is hosted, who can read stored prompts and harmful outputs, and how the data is backed up and retained.

Runner entry points and configuration baseline

Decide how tests are launched: the pyrit_scan command for scheduled scans, CoPyRIT for human-led sessions, or the Python framework for custom attacks. Set a shared baseline in ~/.pyrit/.env for credentials and ~/.pyrit/.pyrit_conf for settings, and confirm the same baseline is applied on every test host.

Findings report template and handover notes

Agree a findings template that records target, strategy, scorer result, reproduction steps, severity and remediation owner. At handover, document the scenario library, scorer settings, memory location and open questions so your team can rerun tests after upgrades. Ask the partner to confirm the template before testing begins.

What drives cost and duration

This table lists the factors that change PyRIT implementation effort and spend, why each one matters and how to keep it under control.

Driver	Why it matters	How to keep it down
Number and type of AI targets	Each model provider, custom HTTP or WebSocket endpoint and Playwright-driven web app needs its own target configuration, authentication handling and response parsing.	Start with the targets that carry the highest risk, and reuse one target pattern across similar endpoints.
Complexity of custom endpoints and web app targets	Custom request formats, session handling, multi-step logins and dynamic page elements take more build and test time than standard model APIs.	Document the endpoint contract early and ask whether a stable test interface can be exposed for red teaming.
Number of attack strategies and scorers	Multi-turn strategies such as Crescendo, TAP and Skeleton Key, and scorers using LLMs, Azure AI Content Safety or custom logic, each add configuration, tuning and validation work.	Agree a short list of strategies and scorers for the first phase, and add others only when a test objective needs them.
Amount of custom Python for converters and attacks	Custom converters and attack logic are code that must be written, reviewed, tested and retested after each PyRIT release.	Prefer built-in components where they meet the objective, and keep custom code in one reviewed package with tests.
Number of datasets and harm categories in scope	Content harms, psychosocial risks and data leakage datasets each need scope decisions, tuning for your use case and documented coverage.	Limit the first scope to the harm categories tied to the system's intended use and record why others are excluded.
Memory and team access requirements	Choosing SQLite or Azure SQL, sharing results across a team and controlling access to stored prompts and harmful outputs all add setup and security review work.	Decide the storage and access model before build, and confirm who may read stored results.
Environments (lab, staging, production-like)	Each environment needs its own configuration, credentials, target endpoints and test data handling, and production-like testing needs change approval.	Test in a lab or staging environment first, and limit production-like runs to a short approved window.
Review workload for human assessors	Scored outputs and flagged responses need people to confirm findings, and the volume of review grows with the number of runs and strategies.	Set review criteria and sampling rules in advance, and ask for an estimate of reviewer hours tied to the planned run volume.
Model API and scoring costs	Attack and scoring runs make paid model calls, and spend depends on run volume, multi-turn length and the scorer models chosen. These costs are billed by your model providers, not DeployOpen.	Confirm pricing with each provider before planning, cap run volumes during tuning and track spend per scenario.

Project phases and outputs

The sequence moves from a written threat model to a tested, documented PyRIT setup that your team can run and extend without the partner.

Scope the threat model and targets
Agree the AI systems to be tested, the harm categories in scope (content harms, psychosocial risks, data leakage and any domain-specific categories), and the evidence your reviewers and auditors will need. DeployOpen leads the workshops with your security, product and compliance owners. Output: an approved target inventory and threat model that records what each test must prove and who signs off on results.
Set up the environment and dependencies
Install PyRIT from PyPI on a Python version the package supports, pinned to the release agreed in scope, and document the dependency lock. The documentation recommends Python 3.13, which we confirm against your hosts and approved base images. Output: a reproducible test environment with a tested install and a version record for the team.
Integrate targets and credentials
Connect each approved target (model APIs, custom HTTP or WebSocket endpoints, or web apps driven through Playwright) and load credentials through your secrets process, not hand-edited files. Verify that each target returns responses and that failures are logged. Output: a target connection register and a credential handling record that your security team has reviewed.
Calibrate scorers and strategies
Run scorers and attack strategies against samples with known outcomes, both known-bad responses and known-good responses, and record where scorers agree or disagree with your reviewers. Adjust thresholds or custom logic until results match agreed expectations. Output: a calibration log with sample sets, scorer settings, results and the decisions made.
Build reusable scenarios
Convert the calibrated strategies into scenario configurations that your team can rerun by target and harm category, with settings kept in version control. Each scenario is tested against its target so the expected output is known. Output: a tested scenario set with a short description, owner and expected results for each entry.
Run a pilot with human review
Execute the scenario set against a pilot target and have named reviewers check flagged and unflagged results, storing conversations in the memory backend agreed in scope. Track false positives and misses, then adjust scorers or scenarios and rerun the affected cases. Output: a pilot report with review outcomes and the changes made as a result.
Hand over with a runbook
Transfer the environment, scenario set, calibration log and credentials process to your team, and walk through the runbook covering routine runs, version updates, result handling and access control. Your engineers run one scenario set themselves while DeployOpen observes. Output: a signed handover with a runbook and a list of open items.

What teams keep in-house and what they outsource

This table lists each PyRIT work area with the team that usually owns it, the skills it needs and whether a partner is most useful, so you can see where to keep control and where outside help shortens the first build.

Work area	Usually owned by	Skills needed	Partner role
Threat modeling and harm definition	Internal product owner and security lead	LLM application knowledge, risk writing, security testing methodology	Keep in-house; a partner may facilitate the workshop but should not set the harm categories
Target integration	Internal platform or application engineers	Python, HTTP or WebSocket APIs, authentication for your model endpoints	Shared; partner can build the first connectors if your team reviews the code
Scorer calibration	Shared between red team and partner	Python, LLM evaluation, content safety judgment, labeled sample data	Partner is most useful here: calibrating scorers against reviewed examples and documenting thresholds
Scenario engineering	Partner for the first library, then internal red team	Python, PyRIT attacks and converters, security testing methodology	Partner is most useful for building the first scenario library and handing over its structure
Memory and infrastructure	Internal platform or cloud team	SQLite or Azure SQL administration, Azure or host administration, backups	Partner documents the setup; your team operates it
Credential handling	Internal security and platform team	Secrets management, .env file controls, access review	Keep in-house; partner reviews configuration but should not hold live keys
Findings review and final judgment	Internal red team lead and risk owner	Domain expertise, risk acceptance, severity judgment	Keep in-house; partners may propose findings but the final call stays with you
Reporting and documentation	Shared; partner drafts, internal owners approve	Risk writing, technical writing, audience-specific summaries	Partner is most useful for documenting the setup; approval stays internal
Training the red team	Internal red team lead with partner support	Security testing methodology, PyRIT operation, CoPyRIT use	Partner can deliver initial sessions; ongoing practice stays with your team

What usually goes wrong

Most PyRIT programs fail on setup decisions made early, not on the framework itself. The items below are the patterns we see most often, with the check that prevents each one.

Untuned scorers

Scorers that flag too much bury reviewers in false positives, while scorers that flag too little hide real failures. Before trusting a scorer, run it against a labeled sample of known-good and known-bad responses, record its disagreement rate with reviewers, and adjust thresholds or criteria. Repeat this check whenever the scorer model or its prompt changes.

Targets without a test boundary

Adding a target without defining what it is allowed to reach leads to tests that touch production data, third-party APIs or internal tools. Write down, for each target, the endpoints in scope, the accounts it uses, the data it may see and the systems it must not call. Review that boundary before the first attack run.

Credentials in plain files

PyRIT reads credentials from ~/.pyrit/.env, so keys copied into shared notebooks, chat threads or repository history are a common leak path. Give each tester separate keys with spending limits, keep the .env file out of version control, and rotate keys after handover. Confirm your secrets approach with your security team before the pilot starts.

Memory without an access plan

PyRIT memory stores full conversations and scored results in SQLite or Azure SQL, and those records can contain harmful outputs and sensitive prompts. Decide who may read the database, where it lives, how long it is kept and how it is deleted. Budget for this access control work, because it recurs with every new tester.

Unsupported Python version

The PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends 3.13. Pinning a host image or CI runner outside that range causes installation failures that can look like tool bugs. Set the interpreter version explicitly in your build, and test the install on the same image your team will use.

Scenarios tied to one model

Attack scenarios tuned against one model version often stop finding issues after the provider updates the model. Parameterize targets and keep a regression set of prompts that you rerun on each model change. Record the exact model version in every result so findings can be compared across time.

Scorer output used without review

Treating scorer verdicts as final findings produces both missed issues and reported issues that are not real. Assign a human reviewer to confirm every high-severity result and to sample lower-severity ones. Record reviewer decisions alongside scorer output so the team can measure how often the scorer agrees.

PyRIT results as an audit report

PyRIT produces test evidence, such as attack logs, scores and reviewer notes. It does not produce certifications or compliance determinations, and independent assessors make those decisions against the standards that apply to you. Present PyRIT output as supporting evidence for an assessor, and ask them which records they need before testing begins.

How the work changes by starting point

A first PyRIT install starts with no harness, so most of the effort goes into choosing what to test and proving the setup works. The work is mostly configuration and a small set of targets.

Confirm your Python version against the PyPI package range, which supports Python 3.10 up to but excluding 3.15. The documentation recommends Python 3.13.
Choose SQLite or Azure SQL for memory before the first run, because results accumulate from the start.
Store credentials in ~/.pyrit/.env under a controlled service account, and keep settings in ~/.pyrit/.pyrit_conf.
Size the effort by the number of target types and by how many attack strategies and scorers the team must validate before sign-off.
Risk: a first run against a live model can generate model call charges before anyone has set a spending limit.

What you need to provide

A PyRIT engagement moves fastest when the client names its people and approves its boundaries before any adversarial prompt is sent. Assign a named owner for each AI system under test, a separate owner for red teaming results, and a reviewer for each finding category. Each owner should be able to approve a change to scope, accept a risk, or pause testing without waiting for a committee.

Provide a written test boundary and rules of engagement. These should list each target, the permitted attack types, the hours when testing may run, the environments in scope, and what happens if a tested system degrades or produces content that must be escalated. Supply API access to each target, along with approved model accounts for attack generation and scoring, each with a spending limit the client sets. Confirm the runtime the team will use: PyRIT's documentation recommends Python 3.13, and the package supports Python 3.10 up to but not including 3.15. Also provide a host or cloud account for test runners, and decide whether PyRIT memory will use SQLite or Azure SQL.

Confirm early: which models may receive adversarial prompts, and where prompts, responses and scoring logs may be stored. Many organizations restrict harmful content to specific regions, tenants or retention periods, and some providers' terms affect what can be sent. Get written approval from the security and legal owners before the first run. Also supply an approved sample of harm categories and datasets, plus sign-off from the teams that own each tested system so that results can be acted on.

Frequently asked questions

What drives the cost of a PyRIT implementation?

The written scope drives cost more than the software does, because PyRIT is open source and carries no license fee. The main cost components are engineering time to connect your targets, build custom attacks, converters and scorers, and set up memory storage, plus model API spend for attacks and scoring, plus the time your own staff spend reviewing findings. Scope grows with the number of targets, the attack strategies you want, and how many reusable scenarios your team needs after handover. Ask any quote to itemize these components separately, state which ones are one-time and which recur, and name who pays for each model API account.

How long does a PyRIT implementation take?

No duration is fixed for this work, so the written scope sets the timeline. Timing depends on how many targets need connectors, whether they are model APIs, custom HTTP or WebSocket endpoints, or web apps driven through Playwright, and how many custom attacks and scorers must be built and tested. Access to the systems under test and approval to run tests against them also shape the schedule. Ask for a milestone plan that lists each target, scenario and handover step with the dependencies you must clear.

How much model API spend should we expect, and why does it vary?

Model API spend has no fixed figure because it depends on your test design. The largest drivers are attack volume, meaning how many prompts and objectives run against each target and how often you repeat them. Multi-turn strategies such as Crescendo, TAP and Skeleton Key send several requests per objective, so they cost more than single-turn tests. LLM-based scorers add a further call for each response they evaluate, and the choice of scorer model changes the price per call. Confirm the expected test volume, the model providers and the account owner before budgeting, and ask the scope to estimate spend per test run.

Can our team run PyRIT alone using its documentation?

Yes, a capable engineering team can install PyRIT from PyPI and work through the official documentation, which covers targets, converters, scorers, attacks and memory. The harder work is operational: connecting your own endpoints, writing scorers that match your harm definitions, storing results securely and making scenarios repeatable for other testers. Check whether your team has Python experience, time to maintain custom code after each release, and people who can judge the findings. A partner is useful when those skills or capacity are missing, and the scope can cover only the gaps.

What must we provide before the implementation starts?

You provide access, decisions and accounts. Typical inputs include a list of target systems with endpoint details and test accounts, the model providers and API keys your organization approves, a Python 3.13 environment or another supported version in the 3.10 to 3.14 range, a host to run tests from, and a decision on SQLite or Azure SQL for memory. You also name the people who approve test scope, review harmful outputs and receive findings. Confirm these inputs with your security and procurement teams before work begins, because missing approvals are a common cause of delay.

Does PyRIT replace a compliance assessment?

No. PyRIT is a testing tool that produces evidence about how an AI system responds to adversarial inputs, and that evidence can support a compliance assessment. Independent auditors and certification bodies decide whether controls meet a standard, and DeployOpen does not issue audit reports, attestations or certifications. Ask your assessor which evidence they need, such as test plans, scope records, run logs and reviewed results, and design the PyRIT setup to produce them.

What happens after go-live?

After handover, your team owns the setup, including keeping PyRIT current, rerunning custom targets, converters and scorers after each release, and reviewing new findings. Releases change behavior, so plan regression runs before upgrades. Ongoing support, upgrades and continuous testing are available as retainers, and the handover scope should state which of these tasks your team keeps and which a retainer would cover. Confirm the release you will pin and the owner of each recurring task.

How DeployOpen delivers PyRIT implementations

DeployOpen plans, builds, documents and supports PyRIT red teaming setups inside infrastructure you control, starting from a written scope that names the targets, attack strategies, scorers and storage in play. Engagements are fixed-scope, and the work moves from agreed scope to a tested setup your team can run.

How we scope

The written scope names each target to test, such as model APIs, custom HTTP or WebSocket endpoints, or web apps driven through Playwright.
It lists the attack strategies, including multi-turn options such as Crescendo, TAP and Skeleton Key, and the scorers that judge responses.
It sets the memory choice between SQLite and Azure SQL, the model API accounts used, and the deliverables, agreed in writing before work starts.

How we build

We install PyRIT on a supported Python version in your environment and configure credentials and settings without hardcoding secrets.
We build reusable scenarios from custom targets, converters, scorers and attacks, then test each one against the agreed targets before handover.
We set up result storage and review workflows so findings from each run can be traced back to the scenario and configuration that produced them.

How we hand over

We document the scenarios, scorer logic, configuration and run procedures so your red team can repeat and extend them without us.
We walk your engineers and red team through running scans, reviewing stored results and changing targets or scorers in your environment.
We provide the written record of what was built, tested and handed over, matched against the scope agreed at the start.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, covering PyRIT releases and changes to your custom components.
Before any upgrade, we revalidate custom targets, converters and scorers against the new release so existing scenarios keep producing comparable results.
Confirm with DeployOpen during scoping which support hours, response paths and release-testing scope the retainer includes.