What PyRIT upgrades and support cover
Upgrading PyRIT touches more than the package. PyRIT is an MIT-licensed Python framework from Microsoft, built from targets, converters, scorers, attacks, memory and datasets. Most red teaming setups add their own targets for internal chat endpoints, custom converters, scorers tuned to an organization's harm categories, and attack scenarios that combine them. Any new release can change the interfaces those components depend on, so the upgrade scope should cover your custom code as well as the upstream library.
Four triggers usually start an upgrade. The first is a new release, such as PyRIT v1.1.0, which was released on 4 September 2026. The second is a change to the Python version, which must stay within the range the PyPI package supports, from Python 3.10 up to but not including 3.15. The third is a changed model API or endpoint on a tested system, which can break a custom HTTP or WebSocket target without any PyRIT change. The fourth is handover of a red teaming setup to an internal team, which often happens after the initial build.
In short, teams with Python maintainers can often handle routine version bumps in-house, provided they have a test suite that exercises each custom target, converter and scorer. Partners are commonly asked to do the first upgrade and establish a baseline: recorded expected results for a fixed set of prompts, a dependency lock, and a list of every custom component with its owner. Ask any partner to confirm which of these they will deliver and which your team must keep running afterward.
What the work includes
Upgrading PyRIT is more than a version bump, because custom targets, converters and scorers can change behavior between releases. The work below covers each component from release review to ongoing support, with the deliverable each one produces.
Release review and upgrade plan
We read the release notes for the target PyRIT version and compare them with the version you run today, noting changed APIs, removed features and dependency shifts. The output is an upgrade plan that lists which custom components are affected and in what order they will be tested. Deliverable: upgrade record.
Environment rebuild on Python 3.13
We rebuild the test environment on Python 3.13, the version the PyRIT documentation recommends, and confirm the pinned PyPI package installs cleanly alongside your internal tooling. Installs are checked against the supported Python range, which you should confirm against the current package metadata before work starts. Deliverable: upgrade record.
Regression of custom targets
Custom HTTP, WebSocket and Playwright-driven web app targets are re-run against the upgraded framework to confirm requests, response parsing and session handling still behave as before. Playwright targets need particular attention because browser and selector changes can break them without any PyRIT error. Deliverable: test results.
Converter and scorer validation
Each custom converter and scorer is run against a fixed set of reference inputs with known expected outputs, so changes in transformation or scoring logic show up as diffs. Scorers that call LLMs or Azure AI Content Safety should be checked for changed verdicts on the same samples. Deliverable: test results.
Attack strategy reruns
Multi-turn strategies in your library, such as Crescendo, TAP and Skeleton Key where you use them, are rerun against approved test targets to confirm they still run to completion and produce comparable results. Differences in attack success are reviewed before anyone treats them as a finding or a regression. Deliverable: test results.
Configuration review of local settings
We review the credentials file at ~/.pyrit/.env and the settings file at ~/.pyrit/.pyrit_conf on each test host, checking that new releases read them as expected and that no secrets sit in shared or logged locations. Changes to defaults are recorded so the team knows what was altered. Deliverable: runbook.
Ongoing support after handover
After handover, support covers triage of questions and failures, planning for later releases and reruns of the regression suite when targets or model providers change. Requests and resolutions are logged so recurring issues can be traced to a component or a release. Retainer terms, response expectations and scope should be agreed in writing. Deliverable: support log.
What drives effort and duration
This table shows which factors change the work in a PyRIT upgrade or support engagement, why each one matters, and how to reduce it before you ask a partner to quote against a written scope.
| Driver | Why it matters | How to keep it down |
|---|---|---|
| Number and type of custom targets | Each HTTP, WebSocket or Playwright web app target has its own connection and response parsing code to recheck after each release. | Share one adapter pattern across endpoints and retest only the targets a release touches. |
| Number of custom converters and scorers | Each custom converter or scorer is code that must be rerun against known inputs after every upgrade. | Keep a small set with fixtures and expected outputs so regression runs stay quick. |
| Dependence on PyRIT internal APIs | Code that imports internal modules may break between releases and takes longer to repair than code using public interfaces. | Use documented public interfaces and wrap any internal call in one module that is easy to patch. |
| Memory backend: SQLite or Azure SQL | SQLite suits a single test host, while Azure SQL adds connection setup, access control and database migration checks. | Choose the backend before building scenarios and confirm how each release handles existing memory data. |
| Size of the test scenario library | More attacks, datasets and multi-turn strategies mean more stored results to compare after each release. | Tag scenarios by risk and target, then run a core set on every upgrade. |
| Number of environments | Each environment needs its own credentials in ~/.pyrit/.env and matching configuration, so drift adds checks. | Keep one configuration template and promote the same pinned version through each environment. |
| Scorer dependence on LLM judges or Azure AI Content Safety | Changes to a judge model or service can shift scores, and each LLM judge adds model call cost and its own upgrade path. | Pin judge model versions where possible and keep a fixed set of labeled responses to compare scores. |
| Documented baseline | Without a recorded baseline, a difference after an upgrade is hard to attribute to PyRIT, the scorer or the target. | Record versions, configuration and reference results before the first upgrade, and store them with the code. |
How an upgrade or support engagement runs
The sequence takes a PyRIT installation from its current version to a tested, documented and supported state, with each phase closing on a written output.
Inventory the current installation
We record the installed PyRIT version, the Python runtime, custom targets, converters, scorers and attacks, pinned dependencies, memory configuration and the ~/.pyrit settings in use. The output is a written inventory that you review and approve before any change is planned. Ask for this inventory format in your scope.
Assess the target release
We compare the current version with the target release against the PyRIT documentation and the GitHub release notes, then list breaking changes, deprecated interfaces and dependency conflicts that touch your custom components. The output is a release assessment that names each affected component and the proposed fix. Confirm the target version with your team before this step closes.
Check the Python runtime and upgrade the test environment
We confirm the Python version against the range the PyPI package supports and the version the documentation recommends, then upgrade PyRIT and pinned dependencies in an isolated test environment. The output is a working test environment with a lockfile or pinned requirements, plus a record of any dependency changes. If your current runtime falls outside the supported range, the runtime change becomes a separate work item in the scope.
Regress targets, converters, scorers and attacks
We run each custom target, converter, scorer and attack against the upgraded environment and compare results with the baseline recorded before the upgrade. Differences in scores, converter output or attack behavior are logged and triaged. The output is a regression report with recorded results for every component in the inventory, which your team signs off.
Cut over with a rollback path
We schedule the cut-over window with you, deploy the validated version to the production test host, and keep the previous version, environment files and memory database backup available for rollback. The output is a cut-over record showing the deployed versions, the backup location and the rollback steps, tested before the window closes. The schedule is agreed in the written scope.
Hand over and run the support period
We deliver the updated documentation, runbook, component inventory and regression suite to your team, and walk through how to rerun the tests on the next release. A support period then covers questions and fixes under the terms in the written scope. The output is an accepted handover record that names the support contacts and the end date of the support period.
What teams keep in-house and what they outsource
This table shows which PyRIT upgrade and support tasks an internal team usually keeps, which parts a partner commonly leads, and the skill each part needs.
| Work area | Usually in-house | Usually partner-led | Skill it needs |
|---|---|---|---|
| Tracking releases and changelogs | Watching the microsoft/PyRIT releases page, deciding which versions the team adopts, and approving upgrade timing | Reading each changelog against your custom converters, scorers and targets and writing an impact note for each release | Python packaging, reading changelogs against your own code |
| Python environment management | Owning the pinned environment, the Python version used in production, and where it runs | Setting up a reproducible environment and checking it against the supported Python range before cutover | Python tooling, dependency pinning, container or virtualenv practice |
| Owning test targets and credentials | Owning the target systems, model endpoints, API keys and the ~/.pyrit/.env file, and deciding who may use them | Advising on how targets and secrets are wired into the setup and reviewing that secrets are not hard-coded in custom code | Access management, secrets handling, knowledge of the tested systems |
| Reviewing findings and acceptance criteria | Judging whether a finding is a real weakness, setting severity, and deciding what counts as passing | Helping define measurable acceptance tests for each custom component so results can be compared between releases | Red team judgment, threat modeling, risk ownership |
| Writing custom converters and scorers | Defining what each converter and scorer should detect and the harm categories they cover | Writing or reviewing the code, adding unit tests, and running a security review of custom code | Python, LLM evaluation methods, application security review |
| Regression reruns after upgrades | Deciding which attack scenarios must rerun and signing off on the results | Running the regression suite after each upgrade, comparing outputs with the baseline, and reporting differences | Test automation, PyRIT attack and scorer configuration |
| Initial baseline | Supplying the current scenarios, datasets and expected outcomes for the baseline | Capturing the first known-good baseline of results and scorer behavior that later upgrades are compared against | PyRIT internals, data handling, test design |
| Documentation and handover | Maintaining runbooks, owner lists and the decision log for upgrades and findings | Documenting custom code, configuration, upgrade steps and the regression procedure for your team to operate | Technical writing, knowledge of the codebase |
What usually goes wrong and how to prevent it
Most PyRIT upgrade problems follow a few predictable patterns. Plan for each one in the upgrade runbook and in the regression set that runs before any live campaign.
Unpinned dependencies change silently
Installs that float to the latest package versions can change PyRIT behavior between test runs without any code change on your side. Pin the PyRIT version and its transitive dependencies in a lock file, record the versions used for each campaign, and rerun a fixed smoke set after every environment change.
Custom attacks break after API changes
Custom targets, converters and attacks often call internal services that your platform team changes on its own schedule. Keep a contract test for each custom target that sends a known request and checks the response shape, and run it before every PyRIT upgrade so breaks appear in a controlled test before a live campaign starts.
Scorers drift when the judge changes
A scorer that uses an LLM judge can rate the same response differently after the judge model, its version or its prompt changes. Treat the judge as part of the test configuration, store the scorer prompt and judge identifier with each result, and rescore a labeled sample of past responses to measure any shift before accepting the change.
Python version outside the supported range
The PyPI package supports Python 3.10 up to but not including 3.15, and the PyRIT documentation recommends Python 3.13. Hosts running an interpreter outside that range may fail to install the package or behave unexpectedly. Confirm the interpreter version in the runbook and build the test environment from the same image used for production.
Results stored without a baseline
Findings from a new release are only meaningful next to results from the previous one. If memory is not tied to a fixed prompt set, target configuration and scorer version, comparisons become guesswork. Before upgrading, run the current baseline set and save its results with version identifiers in SQLite or Azure SQL memory.
Credentials left in the environment
PyRIT reads credentials from ~/.pyrit/.env, and upgrade scripts or shared shells can expose the same keys to other packages or to logs. Remove keys the upgrade does not need, restrict file permissions, rotate keys that were shared during testing, and check that no key appears in console output or stored results.
How the work differs by starting point
Teams already on a recent 1.x release usually need the smallest change, but the release notes still need a line-by-line review against your code. Confirm the target version in the microsoft/PyRIT releases page before you plan the move, since PyRIT v1.1.0 is the current release line in the facts we are working from.
Main extra checks:
- Release notes diff: Read every changed API, default and attack or scorer behavior between your current version and the target.
- Dependency pins: Confirm the Python version your environment runs (the PyPI package supports Python 3.10 up to but not including 3.15) and the version the documentation recommends.
- Regression run: Rerun your saved attack scenarios and compare scores against the previous baseline.
- Memory schema: Confirm your SQLite or Azure SQL memory still reads and writes results after the change.
- Configuration files: Check ~/.pyrit/.env and ~/.pyrit/.pyrit_conf for settings the new release renames or ignores.
Teams with custom targets and scorers built on an earlier release face the most rework, because their own code depends on framework interfaces that may have changed. Treat each custom component as a product that needs its own test pass, not as a configuration detail.
Main extra checks:
- Interface contracts: Compare the base classes your targets, converters and scorers extend against the target release and list every signature that moved.
- HTTP or WebSocket targets: Replay recorded requests and responses against the custom endpoint to confirm parsing still works.
- Scorer calibration: Run labeled sample responses through custom scorers and compare results with the pre-upgrade output.
- Converter behavior: Confirm that each custom converter still transforms prompts the same way across the set of test inputs.
- Documented expectations: Record which results changed and whether the change is acceptable to the red team lead.
Teams that started from the archived Azure/PyRIT repository must move to the maintained project before any upgrade work begins. The archived repository was made read-only on 27 March 2026, so it will not receive fixes. The full move is described on the migration page, and this tab covers only the checks that apply after you have moved.
Main extra checks:
- Source of truth: Confirm which repository your builds, forks and internal tooling now pull from, and remove any remaining references to the archived project.
- Pinned packages: Check every requirements file and lock file for old package names or versions that the maintained release no longer provides.
- Internal forks: Compare any internal fork with the maintained code to decide which local patches still apply.
- Custom attack retest: Rerun every custom attack after the move, since behavior may differ from the archived code you relied on.
- Result continuity: Confirm that stored results from the archived setup remain readable and comparable.
Teams running PyRIT in an air-gapped or restricted network depend on mirrored packages, so every upgrade starts with an inventory of what the environment can actually download. Ask your security team which package indexes and model endpoints are approved before you schedule any change.
Main extra checks:
- Package mirror: Mirror the PyPI package and all its dependencies for the target version, and verify the mirror contents before import.
- Python runtime: Confirm the Python interpreter available inside the network matches a version the package supports.
- Offline validation: Run the test suite and sample scenarios without internet access to confirm nothing reaches out at runtime.
- Model endpoints: Confirm which approved model or Azure endpoints PyRIT can reach, and test custom internal targets within the boundary.
- Change approval: Record the mirror source, checksums and approval trail so reviewers can trace each upgrade.
Teams handing over to internal staff after an initial build need the upgrade process to be written down and rehearsed, not held by the partner. The handover should leave your staff able to run an upgrade, retest custom components and support red team users without outside help.
Main extra checks:
- Runbook: Document the upgrade steps, rollback path and the owner for each step, written so an engineer who did not build the setup can follow it.
- Dry run: Have internal staff perform one upgrade in a non-production environment while the partner observes.
- Test suite ownership: Name who maintains custom regression tests and how new failures are triaged.
- Vulnerability intake: Confirm internal staff know that PyRIT vulnerabilities are reported privately through the Microsoft Security Response Center under the project's security policy.
- Support boundaries: Agree in writing which questions internal staff handle and which return to a retainer or a fixed-scope change.
What you need to provide
Before we start any PyRIT upgrade or support work, we need a small set of inputs from your side. The first is an inventory of every PyRIT installation in use, with its version, the Python interpreter it runs on, and the pinned dependencies in its requirements or lock file. Without this, we cannot tell which releases are safe to move to, and an upgrade can silently change scorer or converter behavior that your team relies on. The Python requirement should be checked against the supported range listed for the PyPI package, and the documentation's recommended Python version should be noted as a reference point.
Second, we need the source for everything you have built on top of PyRIT: custom targets, converters, scorers and attacks, along with any local datasets and the configuration in ~/.pyrit/.pyrit_conf that drives them. Custom code is where most upgrade regressions appear, so we need it in a repository we can branch and test. We also need the owner of each custom component to confirm what it is expected to do, because a passing test does not always mean the scorer still judges responses the way your red team expects.
Third, provide a non-production test environment with the target systems or authorized test endpoints that PyRIT will exercise, plus API keys and model access scoped for testing only. Keys should be separate from production credentials and have spend limits you control, since attack and scoring runs consume model calls. If test endpoints are unavailable, release validation will be limited to mocked targets, which will not show how a real system responds to multi-turn attacks.
Finally, name two accountable people. The first owns release decisions: approving which PyRIT version is adopted and when. The second owns red team findings: deciding how results are triaged, retained and reported. We also need written authorization that defines the systems, endpoints, time windows and attack types in scope. Without named owners, upgrade work stalls at the first decision, and without written authorization we will not run testing against any system.
Frequently asked questions
Are PyRIT upgrades free, and what do they actually cost?
The software is free. PyRIT is released under the MIT license and has no paid edition, so there is no license fee for upgrading. The cost comes from three places: engineering time to read release notes, update dependencies and retest your custom targets, converters and scorers; model API calls made during regression runs; and reviewer time to check whether any findings changed. Engineering time is usually the largest item. For a line-by-line breakdown of one-time and recurring costs, see the cost of ownership page for PyRIT.
How long does a PyRIT upgrade take?
The duration depends on how much custom code sits on top of PyRIT and how well that code is tested. The main drivers are the number of custom targets, converters and scorers, whether any custom attacks rely on APIs that changed between releases, how many model providers you test against, and whether you already have a regression baseline to compare against. DeployOpen states the timeline in the written scope before work starts, so ask for the drivers to be listed for your environment rather than relying on a general estimate.
Can our own team upgrade PyRIT without an outside partner?
Yes, if your team has Python maintainers who can read the release notes and change custom code, and if you have a test baseline that shows how your scenarios behaved on the current version. Without a baseline, it is hard to tell whether a changed result came from PyRIT, from a model update or from your own code. Teams without those two things often bring in a partner for the first upgrade, then keep the baseline and runbook in-house for later releases.
How often should we upgrade PyRIT?
Review every release, then decide on a planned cadence rather than upgrading immediately. Check each release on the microsoft/PyRIT releases page for breaking changes, and confirm that your Python version falls inside the supported range (the PyPI package supports Python 3.10 up to, but not including, 3.15, and the documentation recommends Python 3.13). Pin your version, schedule regression runs on a staging copy, and move production only after the results match your baseline.
What does a PyRIT support retainer cover?
A retainer covers ongoing work agreed in writing after handover, such as reviewing new releases, planning and running upgrades, validating custom targets, converters and scorers after each change, and answering operational questions about red teaming workflows. Hours, response expectations and included activities are set in the retainer scope, not assumed. Ask for a list of what is included, what is billed separately, and how upgrades are handled if a release changes an interface your custom code depends on.
How do we report a vulnerability in PyRIT?
Report it privately, not in a public GitHub issue. The PyRIT project directs vulnerability reports to the Microsoft Security Response Center under the project's security policy, which is described in the microsoft/PyRIT README. Follow that policy for the project itself. If the issue came from your own deployment, such as a credential stored in the .env file or an exposed test host, handle it through your internal incident process as well.
How DeployOpen scopes and supports PyRIT work
DeployOpen plans and delivers PyRIT upgrades and support as a written, fixed-scope engagement, with the work carried out in infrastructure you control. Ongoing support after handover is available as a retainer.
How we scope
- The written scope lists which PyRIT release line you move to, including the 1.x line and its current 1.1.0 release.
- It names each custom target, converter, scorer and attack that must be revalidated after every release.
- It defines the Python version, dependency pins, memory backend and environment files covered by the work.
How we build
- We upgrade PyRIT in a staging environment that mirrors your test hosts and pinned packages before any production change.
- We rerun custom attacks and scorers against agreed test cases and compare results with the prior baseline.
- We document every configuration change, including settings in the .pyrit directory, so the next upgrade starts from a known state.
How we hand over
- Handover includes a written upgrade runbook covering dependency pins, rollback steps and the validation checklist used.
- Your team receives a record of test results per custom component, showing what passed, changed or needs follow-up.
- We walk your engineers through the runbook so they can repeat the upgrade process on their own schedule.
How we support
- Ongoing support can be set up as a retainer covering review of new PyRIT releases and revalidation of custom components.
- Support scope, response expectations and the release cadence are agreed in writing before the retainer starts.
- Questions to confirm with us include which upstream release changes affect your targets and whether your memory backend needs migration.
