What garak upgrades and support involve
garak is a free, Apache 2.0 licensed command-line tool installed from PyPI with pip, and it changes from release to release. The project's GitHub releases page lists v0.17.1, released on 16 October 2026, and each new version can add or revise probes, detectors and the details written into reports. An upgrade is therefore not only a package bump. It can change which attacks run, how a response is judged and what the JSONL report and garak.log contain, so results from before and after an update may not be directly comparable.
Most upgrade work is triggered by a small set of events: a new garak release, a new model or endpoint added to the scan scope, an audit or customer request for evidence, a failed pipeline run right after an update, or the departure of the engineer who owns the scans. Each trigger should lead to a decision: whether to adopt the new version, re-baseline the scan set, or leave the pinned version in place until the change is reviewed. Record that decision with the version number so a later reader can see why results moved.
Most teams keep this work in-house. A small team with stable pipelines and a pinned garak version can usually handle routine updates itself: read the release notes, run the scan set against a test target, compare the output with the existing baseline and then promote the change. A partner becomes more useful when several targets depend on the same scan set, when a pipeline breaks in ways nobody on the team can trace, or when the scans must be handed over to new owners. Ask any partner to confirm in writing which of these tasks sit inside a retainer and which are fixed-scope work.
What the work includes
The engagement covers the components below, from the baseline of your current installation through to a handover your team can run without us. Each one has a written output you keep.
Installation baseline and pinned environment
We record the installed garak version, the Python interpreter, dependencies, probe and detector set, generator configuration and the host each scan runs on. The result is a pinned environment definition, so any future run can be rebuilt and compared against the same setup. Confirm where the baseline is stored and who can change it.
Version policy and pinning approach
We agree which garak release your scans use, how often you review newer releases, and who approves a move to a new version. Pinning keeps results comparable between runs, while a written policy stops unplanned changes from reaching production scan pipelines. Ask how exceptions for urgent fixes are handled.
Tested upgrade procedure with regression comparison
Each upgrade is run in a staging copy of your environment against the same targets as the previous release. We compare the new JSONL reports with the earlier runs to find probes whose failure rates moved, then decide whether each change is a real shift or a tooling effect. The comparison and the decision are written into the upgrade test report.
Probe set and detector review
On each release we review changes to probes, detectors and their default harness settings, including the recommended detectors that the default probewise harness uses. This shows which tests were added, removed or altered, and whether your chosen probe selection still matches your risk scope. Changes are recorded in a changelog review record.
Python environment management
The garak PyPI package requires Python 3.11 or newer, so we confirm the interpreter version on every scan host and build an isolated environment for garak and its dependencies. We check that the environment is reproducible from the pinned definition. Ask who maintains the host OS and interpreter patches after handover.
Pipeline and scheduling support
We set up or review the job that runs garak on a schedule, including secrets for generator credentials, output storage for JSONL reports and garak.log, and alerting when a run fails. The support scope states which pipeline failures we handle and how quickly you can expect a response. Confirm the scheduler you already run before the build starts.
Handover runbook and support scope
The runbook covers how to upgrade, roll back, add a target, rerun a baseline and read a report, written for the engineers who will operate the scans. The support scope is a separate written document that defines what is covered after handover and what would need a new quote. Ask for both before signing.
What drives effort and cost
Each row names one driver of garak upgrade and support effort, explains why it grows the work, and gives the step that keeps it manageable.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Number of garak versions in use | Each team on a different version produces results that cannot be compared, and each version needs its own pinned environment to test and patch. | Pin one version for all scheduled scans and move teams together after a documented test cycle. |
| Targets and generator configurations in pipelines | Every model, endpoint and generator YAML file wired into a pipeline must be rechecked after a release that changes generators or probes. | Limit targets to those with a current business owner and retire configurations no pipeline uses. |
| Custom probes, detectors and REST generator YAML | Local plugins and endpoint descriptions can break when garak changes its plugin interfaces, and each one needs a regression run after every upgrade. | Keep custom code small, document each file's purpose and owner, and store them in version control with a test scan. |
| Default 10 generations per prompt and probe selection in scheduled jobs | Each attempt is a model call, so scheduled jobs with broad probe sets multiply API or GPU spend and runtime on every run. | Choose probes by threat model, review the generation count per job, and confirm the expected call volume with the model provider before scheduling. |
| Size of JSONL reports and garak.log history | Reports and the continued garak.log grow with every run, which slows review, comparison against baselines and retention work. | Set a retention period, archive completed runs with their version and config, and rotate logs per pipeline. |
| Air-gapped or private package mirror environment | Offline hosts need mirrored Python packages, model access paths and manual transfer of each release, adding steps to every upgrade. | Mirror the pinned release and its dependencies on a fixed schedule and record each mirror update in the change log. |
| Number of sign-off approvers | Each reviewer adds a wait for results review and a second round of changes when a probe change alters pass or fail outcomes. | Agree in advance which change types need which approver, and ask for a single named owner for routine version updates. |
| Support hours after handover | Without a defined support scope, questions about results, failed runs and new targets arrive ad hoc and consume engineering time. | Ask for a written support scope with response expectations and a list of supported targets before handover. |
How the engagement runs
The sequence moves garak from its current state to a pinned, tested version that your team can operate and compare run over run.
Discover current versions, pipelines and targets
We inventory the garak version in use, the Python environment, every scan pipeline, the models and endpoints each one targets, and the generator configuration. Output: a written inventory that you confirm before any change is made, so the baseline is built on what actually runs today.
Capture a baseline on a fixed reference set
We run the current garak version against a fixed reference set of probes and targets and save the results as JSONL reports, with the garak.log kept alongside. Output: archived baseline reports with the exact version, probe selection and generator settings recorded, so later runs have something to compare against.
Review the release and upgrade in a test environment
We read the release notes for the target garak version, note changes to probes, detectors and generators that affect your pipelines, and upgrade a copy of the environment. Output: a tested upgrade with a list of configuration changes and any pipeline fixes, confirmed to run end to end in the test environment.
Compare results against the baseline
We run the same reference set on the upgraded version and compare the JSONL reports with the baseline, probe by probe. Each difference in pass or fail status or failure rate is traced to a cause, such as a new probe, a changed detector or a generator change. Output: a comparison report in which every difference is explained or flagged for your decision.
Promote to production with version pinning
Once you approve the comparison, we promote the upgrade to production and pin garak and its dependencies to exact versions in your environment. Output: pinned production configuration and a record of the version change, so scans in the future produce results that are comparable with the baseline.
Hand over the runbook and agree support terms
We deliver a runbook covering the pinned versions, the upgrade procedure, baseline management and troubleshooting, then walk your team through it. Support terms follow the written scope. Output: a signed-off runbook and an agreed support arrangement, with the handover complete when your team has run a scan from the runbook without assistance.
What stays in-house and what a partner can take
This table separates the garak decisions that should stay with your security team from the upkeep work a partner can carry, with the skills each part needs.
| Work area | Usually kept in-house | Often outsourced | Skills involved |
|---|---|---|---|
| Target selection | Deciding which models and applications matter, and which deployments are in scope for scanning | Mapping the target list to garak generators and confirming each connection works | Application threat modeling, LLM application security knowledge, garak generator configuration |
| Probe coverage decisions | Approving which probes and detectors count as required coverage and any changes to that set | Checking how new or renamed probes change existing scan results before the set is changed | LLM attack taxonomy knowledge, reading probe and detector documentation |
| Release tracking | Deciding when to adopt a new garak release based on your risk tolerance | Watching the garak releases page, summarizing changes and flagging breaking items such as the Python 3.11 minimum | Reading release notes, Python packaging with pip and PyPI |
| Upgrade testing and environment pinning | Accepting the pinned version and the reason for any exception | Installing candidate versions in a test environment, pinning dependencies and recording the tested combination | Python virtual environments, dependency pinning, Linux and macOS packaging |
| Regression comparison | Judging whether a change in failure rate is a real security change or an acceptable difference | Comparing JSONL reports from the old and new versions on the same targets and generation settings | Reading JSONL reports, scripting comparisons, understanding garak failure rates and the default 10 generations per prompt |
| Pipeline repair and scheduling | Setting the scan schedule, owners and alert thresholds | Repairing broken CI jobs, rotating credentials used by scheduled runs and keeping garak.log output retained | CI scheduling, secrets handling, shell scripting, log management |
| Findings and remediation | Interpreting findings, assigning owners and approving fixes for the application or model | Producing summaries of which probes failed and where the evidence sits in each report | Application security judgment, reading garak JSONL reports, incident and ticket workflows |
| Documentation | Owning the risk decisions, accepted exceptions and the approved coverage list | Maintaining runbooks for installing, pinning, running and upgrading garak after handover | Technical writing, runbook design, familiarity with the garak user guide |
What usually goes wrong after an upgrade
garak changes over frequent releases, and most post-upgrade problems come from the scanner rather than the model under test. These are the failure patterns to plan for and the controls that catch them early.
Probe changes masquerading as model changes
A new release can add, retire or retune probes and detectors, so scan results move even when the target model is unchanged. Trend lines may then show a regression or improvement that is really a change in the test. Rerun a baseline on the old and new versions against the same target before comparing any trend.
Unpinned installs during scheduled runs
If the scan host installs garak without a fixed version, a scheduled job can pull a newer release without anyone approving it. Pin the exact version in a requirements file or lock file, and promote upgrades through a change process so every run in a series uses the same code.
Python version mismatch with the 3.11 requirement
The garak PyPI package requires Python 3.11 or newer, and a host image or container base that ships an older interpreter will fail at install or behave unpredictably. Check the interpreter version in the scan environment as part of every upgrade, and record it alongside the garak version in run metadata.
Generator configuration breaking silently
Changes in generator behavior, endpoint response formats or REST generator YAML handling can leave a run completing without meaningful output. A scan that finishes is not proof that it tested the target. Confirm that a known canary prompt still reaches the endpoint and returns parsed text after every upgrade.
Report and log format drift
Each run writes a JSONL report and a garak.log file that continues across runs, so both can grow large and their structure can change between releases. Downstream parsers, dashboards and ticketing imports may then fail or misread fields. Test the parser against reports from the new version and set a rotation policy for garak.log.
No owner for release notes
When no named person reviews release notes, probe additions, default changes and deprecations reach production scans unexamined. Assign one owner to read each release, decide whether to adopt it, schedule the baseline rerun and record the decision. Without that owner, the other failure patterns on this page tend to surface only after results are already in use.
How the work changes by starting point
Most early garak use happens on a single workstation, installed from PyPI with no recorded version. Upgrade work here is mostly discovery and baseline capture before anything changes.
- Inventory first: record the installed version, Python version (the PyPI package requires Python 3.11 or newer), and any local configuration files.
- Preserve existing reports: JSONL reports and the garak.log file continue across runs, so archive them before upgrading to keep comparable history.
- Pin before automating: agree on a version and install method so two people's results can be compared.
- Expect probe changes: newer releases can add or change probes, so the same target may produce different failure rates after an upgrade.
- Confirm ownership: a laptop setup usually has no named owner, so support scope must be assigned to a person or team.
Teams running scheduled jobs against multiple model providers have the most moving parts. Each provider has its own credentials, rate limits and model identifiers, and a garak upgrade can touch all of them at once.
- Pin the garak version in the pipeline definition: scheduled jobs should install a known release, with upgrades made through a reviewed change.
- Stage upgrades against one provider first: run the new version on a fixed set of targets and compare results with the last baseline before enabling it everywhere.
- Track generator configuration per provider: credentials and model names change independently of garak, so record them alongside each job.
- Budget for API spend during test runs: a probe set change can alter the number of requests a scan sends, which affects per-run cost.
- Define failure handling: decide whether a failed job blocks a release, pages an owner or only logs a result.
Teams that point garak at their own application or gateway through the REST generator maintain a YAML description of that endpoint. Upgrade work must check that this configuration still parses and still reaches the endpoint correctly.
- Version the YAML file: keep the endpoint description in source control so changes can be reviewed and rolled back with the garak version.
- Retest authentication headers after changes: token rotation or gateway updates can break a scan without any change in garak itself.
- Verify response parsing: if the application's response format changes, the parsing rules in the configuration need updating.
- Watch for load on production systems: confirm that the endpoint owner has approved scan volume and a schedule before each upgrade test.
- Assign endpoint owners: support depends on knowing who changes the application and who can approve configuration updates.
In an air-gapped or restricted environment, garak is installed from an internal package mirror rather than directly from PyPI. Every upgrade becomes a transfer and validation task before it reaches the scan host.
- Mirror the exact release: confirm the internal repository holds the version under test along with its dependencies, including the Python 3.11 or newer requirement.
- Ask how packages enter the network: confirm the approved transfer process, review steps and lead time with your security team before planning an upgrade.
- Check model access paths: local models, such as those run through llama.cpp, or internal gateways must remain reachable without external calls.
- Keep the probe set documented: record which release introduced each change so reviewers can assess it offline.
- Plan support access: confirm how a partner receives logs, configuration and reports without moving data outside the approved boundary.
Teams that have built their own wrapper around garak, for example to parse JSONL reports into dashboards or tickets, depend on the report format and command-line behavior. Upgrades must test the wrapper as well as garak.
- Test the parser against new report output: compare JSONL fields from the current and candidate versions before changing the wrapper.
- Map wrapper logic to garak behavior: identify where the wrapper relies on failure rates, probe names or the garak.log file.
- Keep a sample report set: archived reports from known runs let you check wrapper output after each upgrade.
- Separate wrapper and garak changes: make one change at a time so a regression can be traced to its source.
- Document ownership of the wrapper: support covers the code that reads garak output, which is often outside the scope of a garak-only engagement.
What you need to provide
DeployOpen works inside your infrastructure, so the first thing we need is written access to the scan hosts and your CI system. Specify which hosts run garak, which service account runs scheduled scans, where pipeline definitions live, and who can change them. Access should be granted before the upgrade work starts, and the grant should name the environments and permissions involved so that scope is clear to both sides.
Provide a list of targets and generator configurations. For each target, name the model or application, the garak generator type, and the YAML endpoint description where one applies. Credentials should stay in your secrets process, not in configuration files we receive. Tell us which secrets store holds each key and how rotation works, so we can reference secrets by name in pipelines without handling raw values.
Share the garak version you run today and your pinned requirements file, along with any local probe or detector modifications. Also provide your existing JSONL reports and garak.log history. These files form the baseline for comparing runs after an upgrade, so confirm that they are retained for the period you need and that they are stored in a location we can read without copying sensitive generated content elsewhere.
Agree a reference prompt set that your team considers representative of the risks you care about, and name owners who can approve upgrades and review findings. Decide in advance the acceptable change in failure rates between runs, since that threshold determines whether a new release is accepted, investigated or rolled back. Recording these decisions before the first upgrade keeps the process repeatable when the people involved change.
Frequently asked questions
How often should garak be upgraded, and does every release need a full regression run?
Upgrade on a planned cadence that you set, reviewing each release before it reaches scheduled scans. Not every release needs a full regression run. Scope the regression to the probes, detectors and generators your program uses, and run the full set only when a release changes how those plugins behave. The garak releases page on GitHub lists each version with its notes, which is the starting point for that review. For example, the project's current release at the time of writing was v0.17.1, released on 16 October 2026. Your team should confirm the release notes against your own target list before choosing a cadence.
Does garak itself cost money to run?
No. garak is licensed under Apache 2.0 and the maintainers describe it as a free tool, so the license cost is zero. Spend comes from the systems it tests and the people who run it. Each run consumes model API calls or GPU time, and by default garak makes 10 generations per prompt, so the volume of prompts and generations drives the bill. Staff time covers configuring targets, reviewing JSONL reports and triaging failures. Ask any quote to separate the one-time setup effort from the recurring run costs and the upgrade reviews.
What drives the effort of a garak upgrade?
The main driver is how much of the probe and detector set your program depends on, and how many targets you scan. Effort rises when a release changes detectors or evaluators that your baselines rely on, when your custom REST generator YAML or authentication handling needs edits, or when the Python environment must move to a newer runtime. The PyPI package requires Python 3.11 or newer, so older hosts need work first. Effort falls when scans are pinned to a known version, the environment is reproducible, and the regression set is small and documented.
Can our team upgrade and run garak without a partner?
Yes, if you have engineers comfortable with Python tooling, command-line scanning and reading JSONL output. garak is installed from PyPI and documented in the user guide at docs.garak.ai, so an internal team can run it. The questions to answer honestly are whether someone owns version pinning, who reviews each release, and who keeps the scan pipeline running when the original engineer moves on. A partner adds value mainly in building that ownership model and in handover. Quotes follow a written scope agreed before work starts.
What does DeployOpen need from us to start a garak upgrade or support engagement?
DeployOpen needs access to the environment where garak runs, the list of model endpoints and applications in scope, and the current scan configuration, including any REST generator YAML files and probe selections. It also needs the reports and baselines you already keep, and a named owner on your side who can approve changes. Confirm with your team which credentials can be shared and how they will be provided. Any work plan is fixed to a written scope that lists deliverables and timelines before work begins.
How do baselines stay comparable when garak probes change between releases?
Pin the garak version used for each baseline, record the probe, detector and generator settings, and keep the JSONL report from that run. When a new release changes a probe or detector, rerun the affected baseline on both versions so the difference is visible, rather than comparing scores across different versions without a bridge run. Results from a changed probe should be treated as a new baseline. Confirm with your team how the change is documented so auditors and reviewers can trace each result to its version.
What does support cover after handover, and does it include changes to our security test coverage?
Support after handover depends on the retainer or service agreed in writing. Ongoing support, upgrades and continuous testing are available as retainers, and the covered tasks, response expectations and review cadence should be listed in the scope. Changes to security test coverage, such as adding probes for a new application or retiring probes that no longer match your risk, are a scope decision. They are included only when the written scope says so. Ask for the coverage change process and its effort estimate in any quote.
How DeployOpen delivers garak upgrades and support
DeployOpen keeps an existing garak setup current and comparable across its frequent releases, working in infrastructure you control. Engagements are fixed-scope, and ongoing support is available as a retainer once the work is handed over.
How we scope
- Inventory the garak version, Python runtime and probe set currently in use, and record which pinned version your reports depend on.
- Agree in writing which target models, generators and REST endpoints the upgrade and support work will cover.
- Confirm the upgrade cadence, release-review responsibilities and the retainer terms, if any, before work starts.
How we build
- Pin garak to an agreed release, such as v0.17.1, and manage the Python 3.11 or newer environment reproducibly.
- Test probe and detector changes against a baseline scan so that changes in results can be traced to specific releases.
- Configure and verify each generator, including REST endpoint YAML files, so scan pipelines run unattended with logged output.
How we hand over
- Document the pinned versions, upgrade procedure, rollback steps and the location of JSONL reports and garak.log files.
- Walk your engineers through running a scan, reading FAIL results and comparing failure rates between runs.
- Deliver a runbook describing who approves a new garak release and how the probe set is reviewed before adoption.
How we support
- Review each new garak release against your pinned setup and propose upgrades as separate, tested changes.
- Keep scan pipelines running after handover, investigating failed runs, broken generators and dependency issues.
- Offer ongoing support, upgrades and continuous testing as a retainer, with scope confirmed in writing.
