Self-host PyRIT or use the Foundry red teaming agent?
Choose self-hosted PyRIT when the targets, prompts or results cannot leave your network or a cloud region you have not approved, or when the system under test falls outside what the Foundry agent supports. The Foundry AI Red Teaming Agent is built on PyRIT's attack strategies, but the managed service is limited to Foundry hosted prompt and container agents that use Azure tool calls. It does not support non-Foundry agents, non-Azure tools or function tool calls, and it supports only text-based scenarios. If your application calls a custom HTTP or WebSocket endpoint, a model served outside Foundry, or a function tool, the managed agent does not cover it, and you will need PyRIT running in your own environment. PyRIT is MIT licensed. Its original Azure/PyRIT repository was archived on March 27, 2026, and development continues at the microsoft/PyRIT repository, so confirm you are pinning to the active source.
The managed agent fits a narrower case: a team testing Foundry hosted agents with Azure tool calls in one of the five cloud regions Microsoft lists for cloud red teaming: East US 2, France Central, Sweden Central, Switzerland West and US North Central. Cloud red teaming is also the only mode that runs the agent-only risk categories of prohibited actions, sensitive data leakage and task adherence. If you need those categories but your data may not enter a listed region, you have a conflict to resolve before choosing a tool. Ask Microsoft which regions and data residency terms apply to your tenant, and confirm whether local runs can cover the categories you need.
Self-hosting moves cost and effort onto your team. You pay for the compute that calls your targets, the scorer models (PyRIT scorers can use LLMs, Azure AI Content Safety or custom logic), the storage for its SQLite or Azure SQL memory, and the engineers who write attack configurations and keep dependencies current on supported Python versions, which the PyPI package limits to 3.10 up to but not including 3.15. The managed agent shifts the runtime cost to Azure but still produces Attack Success Rate figures that Microsoft Learn says are scored by generative models and can include false positives. Before committing, have your team review a sample of flagged attempts, and confirm with Microsoft how the ASR scoring model is chosen and whether you can see its judgments.
How the two models compare on control, scope and results
This table sets self-hosted PyRIT against the Foundry AI Red Teaming Agent across the factors that shape control, scope and reporting. Read each row as a question of who runs the engine, what it can reach and where results live, and treat any cell marked as a question to confirm as an open point for your vendor conversation.
| Factor | Self-hosted PyRIT | Foundry AI Red Teaming Agent |
|---|---|---|
| Licensing | MIT licensed, open source on microsoft/PyRIT | Managed Azure service; confirm commercial terms and metering with Microsoft |
| Where the attack engine runs | Your own machines, CI runners or cloud accounts | Local runs or cloud runs; agent-only categories need Azure cloud regions |
| Supported targets | OpenAI, Azure, Anthropic, Google, Hugging Face, custom HTTP or WebSocket endpoints, and Playwright-driven web apps | Foundry-hosted prompt and container agents using Azure tool calls; confirm other model targets with Microsoft |
| Agent support | Custom agent targets through your own endpoints, built and maintained by your team | Foundry hosted agents only; no non-Foundry agents, non-Azure tools or function tool calls |
| Risk categories | Content harms, psychosocial risks and data leakage datasets, plus custom objectives you write | Includes prohibited actions, sensitive data leakage and task adherence, which run only in cloud red teaming |
| Modalities | Text and other modalities depend on your converters and targets; confirm the scope you need | Text-based scenarios only, according to Microsoft Learn |
| ASR reporting | Scores come from your chosen scorers (LLMs, Azure AI Content Safety or custom logic); you define how results are reported | Reports Attack Success Rate and logs findings in Foundry for tracking over time; ASR is model-scored and can yield false positives |
| Memory storage | Conversations and results in SQLite or Azure SQL, under your control | Findings stored in Foundry; confirm retention, export and access controls with Microsoft |
| Python versions | PyPI package supports Python 3.10 up to but not including 3.15; 3.13 recommended | Not applicable; the service runs in Azure and needs no local Python version |
Which data leaves your boundary in a cloud red teaming run
A cloud red teaming run sends attack prompts and model responses to Foundry, while a self-hosted PyRIT run keeps them inside your environment. The cards below separate where data sits, who can read findings, and what residency and audit questions follow from each choice.
Where cloud run data is processed
Cloud red teaming in the AI Red Teaming Agent is available in East US 2, France Central, Sweden Central, Switzerland West and US North Central. If your residency rules require processing in a specific country or region, check whether your chosen region is on that list before you plan a cloud run. Ask Microsoft which regions hold the attack prompts, model outputs and scoring calls, and whether any of them leave that region.
Who can read Foundry findings
The agent logs findings in Foundry so they can be tracked over time. Anyone with access to the Foundry project can therefore see attack results, including examples of prompts that produced harmful output. Confirm which roles hold project access, how Foundry role assignments are reviewed, and whether findings about your application should be treated as confidential security records.
Memory location in self-hosted PyRIT
PyRIT's memory layer stores conversations, scores and results in SQLite or Azure SQL. A local SQLite file stays on the machine running the scan, so you control its backup, encryption and deletion. If you choose Azure SQL, the database sits in your Azure subscription under your access controls. Decide which option matches your data handling policy before the first run.
Residency and sector rule effects
A cloud run creates a new data flow that your records of processing, data protection impact assessments and vendor register may need to cover. Sector regulators in banking, healthcare and the public sector often ask where security testing data is held and who processes it. Involve your privacy and legal teams before choosing a cloud run, and document the answer for each regulator your organization reports to.
Independent auditor judgment
Running a red teaming tool in a given region does not make your system compliant with any framework. Independent auditors or certification bodies decide whether your controls for testing, data handling and vendor oversight meet the standard. Ask your auditor early which evidence they expect for AI testing, and whether a managed service's own assurance reports cover the data flow you intend to use.
Evidence DeployOpen prepares
DeployOpen can plan a self-hosted PyRIT deployment in your own infrastructure and prepare the supporting evidence: data flow diagrams, configuration records, memory storage settings, access lists and test run logs. DeployOpen does not issue audit reports, attestations or certifications. Agree the evidence list and its format in the written scope before work starts.
What the Foundry agent can test in your stack and what it cannot
The agent-only risk categories decide whether the Foundry agent covers your application at all. Prohibited actions, sensitive data leakage and task adherence run only in cloud red teaming, so a local run will not exercise them. Before you plan around them, confirm in the AI Red Teaming Agent documentation on Microsoft Learn which cloud regions currently host these runs. The documentation lists East US 2, France Central, Sweden Central, Switzerland West and US North Central, and a data residency requirement outside those regions rules the cloud path out.
The agent's scope for agents is narrow. It supports Foundry hosted prompt and container agents that call Azure tools. It does not support agents hosted outside Foundry, non-Azure tools or function tool calls. If your agent uses function calling, or runs on another host and calls a third-party API, the managed agent cannot attack the tool path that matters most. Also note that the agent supports only text-based scenarios, so multimodal inputs and outputs are outside its coverage. Map each agent in your stack against these conditions before you commit to the managed route.
Treat ASR as a triage signal, not a verdict. Attack Success Rate is scored by generative models, and Microsoft Learn notes that this scoring can produce false positives. Have a named reviewer read the transcripts behind every flagged result before a finding goes into a backlog or a risk register, and record the reviewer's decision alongside the Foundry entry. Where self-hosted PyRIT reaches further is in its scorer choice: its scorers can use LLMs, Azure AI Content Safety or custom logic, and its memory keeps the full conversation history in SQLite or Azure SQL. That lets you test non-Foundry agents, function calls and custom tools, and it works where the cloud regions do not. Confirm each agent pattern against current Microsoft Learn documentation before you decide that a self-hosted setup is the only option.
Which targets self-hosted PyRIT reaches that the Foundry agent does not
Self-hosted PyRIT includes target classes for OpenAI, Azure, Anthropic, Google and Hugging Face models, so one attack script can run against several providers and compare results. The setup burden sits in your environment: you install the package with pip on a supported Python version, store provider credentials in your own secrets store, and budget for the models that act as judges. The Foundry agent's documented scope is Foundry hosted agents, so confirm before assuming it covers direct model endpoints. The PyRIT documentation lists the targets for the version you deploy.
- Provider coverage: Confirm the exact model names and API versions each target class supports in the release you install, since provider lists change between releases.
- Multi-turn strategies: Crescendo, TAP and Skeleton Key are multi-turn attack strategies that you configure yourself, including the objectives and converters they use.
- Scoring dependencies: Scorers can use LLMs, Azure AI Content Safety or custom logic, so each judge model is another dependency with its own access and cost.
- Python baseline: The PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13.
- Credential handling: Provider keys and any Azure SQL memory connection stay in your environment, so rotation and access reviews fall to your team.
Check each provider target, the judge model and the memory backend against the documentation for the version you deploy.
PyRIT can target a custom HTTP or WebSocket endpoint, and it can drive a web application through Playwright. This path covers chatbots, internal assistants and agent front ends that expose no model API you control. The effort moves to the target adapter: you map the request and response format, handle session state, and decide how browser automation authenticates. The Foundry agent's documented scope is Foundry hosted agents, so an application built outside Foundry falls to the self-hosted path.
- HTTP and WebSocket adapters: You write the adapter for the request schema, headers, authentication and streaming format, and that adapter becomes code your team maintains.
- Playwright upkeep: Browser-driven tests depend on page structure, so interface changes can break scripts and need a scheduled review.
- Conversation state: Crescendo, TAP and Skeleton Key depend on the target keeping context across turns, so confirm your endpoint preserves session state.
- Modalities: The Foundry agent supports only text-based scenarios. Confirm which modalities and converters your PyRIT version supports for your application.
- Network placement: Targets inside a private network need a runner with route access, which you must provision and document.
Run one non-production multi-turn test against a staging copy of the endpoint and confirm session handling before scoping the full suite.
The Foundry AI Red Teaming Agent covers Foundry hosted prompt and container agents that make Azure tool calls. Its attack strategies come from PyRIT, combined with Foundry Risk and Safety Evaluations. The agent-only risk categories of prohibited actions, sensitive data leakage and task adherence run only in cloud red teaming, and cloud runs are offered in a limited set of regions. Your agent's hosting type, tool types and region decide whether the managed run applies at all, and the AI Red Teaming Agent documentation sets out those conditions.
- Supported agent types: Foundry hosted prompt and container agents with Azure tool calls are in scope. Non-Foundry agents, non-Azure tools and function tool calls are out of scope.
- Cloud-only categories: Prohibited actions, sensitive data leakage and task adherence require cloud red teaming, so a local run cannot produce those results.
- Region list: Cloud red teaming is available in East US 2, France Central, Sweden Central, Switzerland West and US North Central. Check that your data handling requirements fit one of these.
- Scoring limits: ASR is scored by generative models and can produce false positives, so reviewers must check flagged results before reporting them.
- Modality: The agent supports only text-based scenarios.
Confirm the hosting type, tool types and region against the Microsoft Learn page before you rely on a managed run as audit evidence.
Many organizations run some agents in Foundry and others outside it. A practical pattern is to use the managed agent where its scope fits, self-host PyRIT for everything else, and keep results in one reporting structure. Both paths report attack success, but the managed agent logs findings in Foundry for tracking over time, while PyRIT memory writes to SQLite or Azure SQL in an environment you control. Agree on one definition of ASR and one finding format before comparing numbers across the two paths.
- Scope split: Route Foundry hosted agents with Azure tool calls to the managed agent and send other targets to self-hosted PyRIT.
- Scoring calibration: The managed agent and PyRIT use different scoring approaches, so review a sample of results from each before treating the numbers as comparable.
- Evidence storage: Foundry keeps tracked findings, while your PyRIT memory database needs backup, retention and access controls that you define.
- Overlap control: Text tests that both paths can run will duplicate effort unless you assign each risk category to one path.
- Permissions: Two tools mean two sets of keys, roles and reviewers to maintain.
Write a one-page ownership map of targets and risk categories, and have your security lead approve which path owns each result.
What your team runs in each model day to day
This table compares the routine work of keeping PyRIT running on your own systems against using the managed AI Red Teaming Agent in Foundry. Read each row as a question to answer for your team: who installs, patches, stores data and handles cloud resources. The main takeaway is that the self-hosted model puts every operational task on your team, while the managed agent shifts some of them to Azure but ties you to Azure regions and resource management.
| Operating task | PyRIT (self-hosted) | AI Red Teaming Agent (Foundry) |
|---|---|---|
| Installation | Install the PyRIT package from PyPI with pip into an environment you control | No package to install; runs are created through Foundry, using PyRIT attack strategies behind the service |
| Python version management | Your team pins a supported Python version; the documentation recommends Python 3.13 | Python runtime is managed by the service; confirm any SDK or client version requirements with Microsoft |
| Memory database operation | You choose SQLite for local use or Azure SQL, then own backups, access and retention | Findings are logged in Foundry for tracking over time; confirm retention and export options in Microsoft Learn |
| Scorer configuration | You configure LLM scorers, Azure AI Content Safety or custom logic, and manage their keys and costs | Scoring uses generative models; confirm which scorers apply and how false positives are handled |
| Report handling | You generate, store and distribute reports from memory output and your own templates | ASR results and findings appear in Foundry; confirm export formats for your audit evidence |
| Access control | Access to the host, database and API keys is governed by your identity and secrets controls | Access is governed by Azure role assignments in the Foundry project; confirm roles with Microsoft |
| Upgrades across releases | You adopt releases such as v1.0.0 (July 24, 2026) and v1.1.0 (September 4, 2026) and retest your own scripts | Upgrades are handled by Microsoft; confirm how changes to attack strategies are announced |
| Cloud resource management | Limited to whatever infrastructure you run; no Azure regional constraints for local runs | Cloud red teaming runs only in listed Azure regions; confirm region fit for your data residency needs |
Moving red teaming work between self-hosted PyRIT and the Foundry agent
This sequence shows how a team can move its attack scope, scoring and history from one red teaming setup to the other without losing the ability to compare results. It ends with a documented cutover that both sides can audit.
Inventory attack strategies, datasets and scoring logic
The team lead and the security engineer who owns the current setup list every attack strategy, prompt dataset, converter and scorer in use today, including any custom scoring logic. For PyRIT, this means the Python scripts, the attack classes and the scorer configuration. For the Foundry agent, it means the strategies and risk categories selected in the run configuration. Export the prompt datasets as files you control, not as references to a local path on one laptop. The step is done when every item has an owner, a location in your repository and a note on whether the other platform can reproduce it, including the cloud-only agent risk categories that cannot run in a self-hosted setup.
Map each target to the destination model
The engineer who owns the application under test writes a target map that shows how each system under test is reached in the new setup. A PyRIT target might be an OpenAI, Azure, Anthropic, Google or Hugging Face model, a custom HTTP or WebSocket endpoint, or a web app driven through Playwright. The Foundry agent supports Foundry hosted prompt and container agents with Azure tool calls, but it does not support non-Foundry agents, non-Azure tools or function tool calls. Mark every target that falls outside the destination's support as a blocker. The step is done when each target has either a confirmed mapping or a documented reason it stays on the current platform.
Confirm text-only scope and region requirements
The compliance lead and the platform owner check the destination's constraints before any run is scheduled. Microsoft Learn states that the AI Red Teaming Agent supports only text-based scenarios, so any multimodal test cases stay in PyRIT or move to another tool. Cloud red teaming is available only in East US 2, France Central, Sweden Central, Switzerland West and US North Central, so confirm that the data residency rules for your test prompts and responses allow those regions. Record the decision in the migration log. The step is done when the scope and region decision is signed off and any excluded test cases are listed by name.
Re-run a baseline on a shared prompt set
The security engineer runs the same fixed prompt set against the same target in both the old and the new setup. Use identical attack strategies where the two platforms offer them, and record the model version, configuration and run date for each run. Because the Foundry agent scores ASR with generative models, expect scorer differences even when prompts match. Store the raw outputs from both runs in your own repository. The step is done when two complete result sets exist for the same prompts and target, and the team can trace each result to its configuration.
Review ASR differences for false positives
A reviewer who did not build either setup compares the two ASR results item by item. Microsoft Learn notes that ASR is scored by generative models and can produce false positives, so every attack that differs between runs needs a human verdict. Record each reviewed item as a true finding, a false positive or an unresolved case, with a short reason. Calculate ASR only after the review, and report the reviewed figure alongside the raw score. The step is done when every disagreement has a recorded verdict and the reviewed ASR for each run is documented.
Confirm how finding history carries over
The engineer responsible for records checks where history lives on each side before cutover. Foundry logs findings so they can be tracked over time, while PyRIT memory stores conversations, scores and results in SQLite or Azure SQL. These are separate stores, so migrating a scan does not automatically move its history. Ask the vendor or your own platform team what export or API path exists, and test it with one finding before relying on it. Decide which history is kept as the audit record and where it is archived. The step is done when a test export has been read back successfully and the archive location is named in the runbook.
Cut over and retire the previous run
The platform owner schedules the cutover only after the baseline review is closed and the history export is verified. Point the scheduled red teaming jobs at the new setup, keep the old configuration in read-only storage for the agreed retention period, and disable the old schedules rather than deleting them. Notify the teams that consume the reports and update the runbook with the new owner, target map and archive location. The step is done when the old jobs are disabled, the new jobs have completed one scheduled run, and the archive of the previous run is confirmed as readable.
Questions teams ask before self-hosting PyRIT or using the Foundry agent
Can we run PyRIT on our own without a managed service?
Yes. PyRIT is an MIT-licensed Python framework that installs with pip, so your team can run it from its own infrastructure without buying a Microsoft service. The PyPI package supports Python 3.10 up to but not including 3.15, and the documentation recommends Python 3.13. What you take on is everything the managed agent would otherwise handle: choosing and wiring targets, selecting attack strategies and scorers, configuring memory, and keeping results in a form your reviewers can use. Engineers must write or adapt the attack scripts, and the team must decide where the SQLite or Azure SQL memory lives and who can read it. The trade-off is that you own every run from start to finish, including the work of tracking results over time.
What skills does a self-hosted PyRIT program need from our staff?
Plan for Python engineers who can read and modify attack code, plus security staff who can judge whether a flagged response is a real failure. PyRIT is built from targets, converters, scorers, attacks, memory and datasets, so an engineer needs to understand how these pieces fit before running a multi-turn attack such as Crescendo, TAP or Skeleton Key. Security staff must review scorer output, because PyRIT scorers can be LLMs, Azure AI Content Safety or custom logic, and each has different failure modes. Ask your team to confirm who will maintain scripts after the initial build, since the original Azure/PyRIT repository is archived and development now continues at microsoft/PyRIT.
How long does a PyRIT setup take for our number of targets?
The supplied facts do not give a duration, so treat any timeline you hear as something to verify against a written scope. Setup effort scales mainly with the number of targets and their types. A single OpenAI or Azure model is a different job from several custom HTTP or WebSocket endpoints, or a web application driven through Playwright, because each target needs its own connection, authentication and response handling. Scorer choice and the number of attack strategies also add work. Ask any implementation partner to break the estimate down by target, by attack family and by reporting need, and to state which assumptions would change the timeline.
What drives cost in a self-hosted PyRIT setup versus the Foundry agent?
In a self-hosted setup you pay for infrastructure, model API usage for both the target and any LLM scorer, and the people who build and run the tests. The Foundry agent shifts some of that into an Azure service model, but the supplied facts do not state its pricing, so check the current Azure terms directly. Costs that rise with scope include the number of targets, the number of attack runs, the use of LLM-based scoring, and the time spent reviewing false positives. Microsoft Learn notes that ASR is scored by generative models and can produce false positives, so budget for human review. When you request a quote from DeployOpen, ask for the written scope to list targets, attack families, scorers, memory location, report format, retained evidence and the handover of scripts. DeployOpen does not publish a price list, so the quote follows from that scope.
Can a cloud red teaming run test agents that use non-Azure tools?
No, not according to the supplied Microsoft Learn facts. The AI Red Teaming Agent supports Foundry hosted prompt and container agents with Azure tool calls. It does not support non-Foundry agents, non-Azure tools or function tool calls. If your agents call third-party APIs, internal services or custom functions, a self-hosted PyRIT run through a custom HTTP or WebSocket target is one path to test them. Confirm the exact target configuration with your engineers and with Microsoft before relying on either route, because support for specific tool patterns can change.
Does the Foundry agent cover every risk category we need?
Not necessarily. Microsoft Learn says the AI Red Teaming Agent supports only text-based scenarios, so multimodal risks fall outside its coverage. The agent-only risk categories of prohibited actions, sensitive data leakage and task adherence run only in cloud red teaming, and cloud runs are available in a limited set of regions: East US 2, France Central, Sweden Central, Switzerland West and US North Central. If your data residency rules or agent architecture do not match these conditions, a self-hosted PyRIT program may be the only option, with the caveat that you must design those risk tests yourself. Confirm region availability and category coverage against the current Microsoft Learn page before you commit.
Does DeployOpen issue compliance certificates for our AI red teaming?
No. DeployOpen plans, implements, runs and documents security testing, and it prepares systems, controls and evidence for compliance assessments. It does not issue audit reports, attestations or certifications. Independent auditors and certification bodies perform those assessments. A red teaming engagement can produce test records, scope documents and remediation evidence that an auditor may review, but whether that evidence satisfies a specific framework is the auditor's decision. Ask your compliance lead to confirm which assessor will review the evidence and what format that assessor expects before scoping any work.
How DeployOpen scopes and runs self-hosted PyRIT engagements
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside the client's own infrastructure. For PyRIT, that means running the framework on your systems, with your target endpoints, your memory store and your evidence trail, instead of relying on a managed service that only covers certain regions and agent types.
How we scope
- We confirm which targets, models and endpoints the written scope covers before any testing starts.
- We record which attack strategies, converters and scorers are in scope and why each one fits your system.
- We state whether agent risk categories are needed, since some only run in Azure cloud regions.
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work begins.
How we build
- We install PyRIT in your environment and confirm the Python version against the documented support range.
- We configure targets for your HTTP, WebSocket or web app endpoints and record each connection setup.
- We set up memory storage in SQLite or Azure SQL and define how conversations and scores are retained.
- We check scorer output for false positives before relying on any attack success rate figure.
How we hand over
- We deliver the configuration, run scripts and scoring logic so your team can repeat the test.
- We document each attack run, its scorer settings and the results stored in your memory database.
- We explain how to read attack success rate and where human review is still needed to confirm findings.
- We walk your security and engineering staff through the setup so they can run the next assessment.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after the initial handover.
- A retainer can cover PyRIT version upgrades, so your test suite keeps pace with new releases.
- We can extend the scope to new targets or agents when your application changes, under a written update.
- Ask any quote to list which support, upgrade and retest items are included and which are billed separately.
