garak vs Giskard: which to choose

garak or Giskard: which fits your AI security test

Choose garak when the target is a model endpoint and you want a command-line scanner that runs a defined set of probes and detectors and writes a reproducible report. Choose Giskard v3 when the target is a multi-step agent or pipeline and you want the adversarial suite built from a plain-language description of what the agent does. The decision turns on three questions: whether the target is a single model endpoint or a multi-step agent, who will run the tests, and whether your team is still on Giskard v2.

garak's generators reach Hugging Face, OpenAI, AWS Bedrock, LiteLLM, llama.cpp models and any REST endpoint described in a short YAML file. Each run writes a JSONL report of every attempt plus a hit log, which suits a security engineer who runs the scan from a pipeline and hands the output to a reviewer. Giskard's vulnerability scan suits a team whose agent calls tools or holds state across turns, because v3 can wrap a black-box agent and generate attacks from its description. Confirm that the description matches what the agent is allowed to do, since the suite is only as accurate as that text. Also check the default judge model, which the README lists as openai/gpt-4o-mini, and confirm where judge calls go before you run either tool against regulated data.

If you are on Giskard v2, plan the move to v3 as a migration. The giskard-oss README says v2 remains available but is no longer actively maintained. In v3, vulnerability_scan supersedes the v2 LLM scan and quality_scan supersedes RAGET, while the legacy tabular and ML scan, the v2 ML test suite and the Giskard Hub are not planned for v3. Inventory which v2 features your team depends on. Also check runtimes, because the garak package requires Python 3.11 or newer and Giskard 3.0.1 requires Python 3.12 or newer, so a shared test host may need an upgrade. Ask the maintainers to confirm any gap that affects your workflow.

Which scenario suits garak and which suits Giskard

This table maps common testing scenarios to the tool that fits them better, based on how each one is built: garak scans a model or endpoint with probes and detectors, while Giskard v3 builds suites from an agent description and can wrap agents and pipelines. Read each row as a starting point, then confirm the fit against your own endpoint and CI setup.

Scenario	Better fit	Reason
Scanning a bare model endpoint	garak	Its generators reach Hugging Face, OpenAI, AWS Bedrock, llama.cpp and REST endpoints, and probes run directly against the model.
Testing a black-box agent	Giskard v3	Its vulnerability scan generates adversarial suites from a plain-language description of the agent, so you do not need to write attack sets first.
Testing a multi-step pipeline	Giskard v3	It can wrap an LLM, a black-box agent or a multi-step pipeline, which garak's model-level probes do not address directly.
Gating a CI pipeline	Depends on the target	garak runs as a command-line job with JSONL reports, while Giskard checks are Python scenarios. Confirm how each fits your runner and pass criteria.
One-off assessment of a model	garak	A single pip-installed command with default probes and a failure rate per probe gives a quick, documented baseline.
Team already on Giskard v2	Giskard v3 (plan migration)	v2 remains available but is no longer actively maintained, and v3 replaces the LLM scan and RAGET. Confirm which v2 features you use before moving.

How garak and Giskard v3 are built and deployed

garak and Giskard v3 both run from Python and both put a judgment step between an attack and a verdict, but they package that work differently. The cards cover how each is installed, what runs where, what files each leaves behind, and which external dependencies your environment has to supply before the first scan.

garak's plugin model

garak is built from five plugin types: probes generate attacks, detectors judge whether a response shows a failure, evaluators summarize results, generators connect to the target, and harnesses run the probes. The default probewise harness runs each probe with its recommended detectors, so you can start with defaults and swap pieces only when a finding needs a different judge. Check which detectors a probe uses before you read its FAIL result as a verdict on the model.

garak install on Python 3.11

garak installs from PyPI with pip and the package requires Python 3.11 or newer. The maintainers document development on Linux and macOS, so teams running Windows hosts or locked-down build images should confirm the interpreter version and OS support in their own environment before they add garak to a pipeline image.

Giskard v3 split into two packages

Giskard v3 separates giskard-checks, which holds the scenario API, built-in checks and LLM-as-judge checks, from giskard-scan, which generates adversarial suites from a plain-language agent description. Both are marked stable in the giskard-oss README. Decide early whether you need the scan, the checks, or both, because the install extras determine what lands in your environment.

Giskard's Python 3.12 requirement

Giskard v3 requires Python 3.12 or newer, one minor version above garak's floor. A shared pipeline image that pins Python 3.11 cannot run both tools without a separate environment. Install with pip install giskard and add optional extras such as [scan] and [openai] only for the features you use, then confirm the resolved versions in a clean build.

Artifacts each tool leaves behind

A garak run writes a JSONL report of every attempt and its evaluation, and a garak.log file that continues across runs. Retain both, since the JSONL file is the evidence trail for any FAIL. Giskard scan output is a separate artifact, so confirm its format and where it is written in the version you pin, and plan storage and retention before the first scheduled run.

Judge model dependencies

Giskard's LLM-judge checks use openai/gpt-4o-mini as the default judge model, so a run that uses those checks sends content to a judge model unless you configure another one. garak's detectors can also rely on model-based judgment depending on the probe. Before running either tool on sensitive data, confirm which judge endpoint is called, whether it sits inside your boundary, and how much of each prompt and output it receives.

Capability, license and extensibility compared

This table compares garak and Giskard v3 on license, maintainer activity, attack coverage, target types, checks, reporting and identity integration. Read each row as a question to put to both projects, and treat unsupplied items as things to confirm before you commit.

Capability	garak (NVIDIA)	Giskard v3 (giskard-scan and giskard-checks)
License	Apache-2.0, developed in NVIDIA's GitHub organization	Apache-2.0, open-source library in the Giskard-AI/giskard-oss repository
Latest release cited	v0.17.1, published October 16, 2026	Giskard 3.0.1 on PyPI, published October 2, 2026; giskard-checks v1.0.6 on October 9, 2026
Attack coverage	Hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreak probes, combining static, dynamic and adaptive probes	Adversarial suites generated from a plain-language agent description, covering prompt injection, jailbreaks, harmful content and OWASP LLM Top 10 categories
Target types	Hugging Face Hub, Replicate, OpenAI, AWS Bedrock, LiteLLM, llama.cpp (gguf) and any REST endpoint configured in YAML	Wraps an LLM, a black-box agent or a multi-step pipeline; scenario API for application-level checks
Built-in checks and judging	Detectors judge each attempt; default harness runs each probe with its recommended detectors	String and regex matching, semantic similarity and LLM-judge checks such as Groundedness and Conformity; default judge is openai/gpt-4o-mini
Extensibility model	Plugin categories: probes, detectors, evaluators, generators and harnesses	Python library with scenario API, built-in checks and optional extras such as [scan] and [openai]
Reporting output	JSONL report of every attempt and its evaluation, a hit log of successful attacks, and a continuing garak.log	Scan results from the library; confirm the export formats and report structure for your pipeline
Runtime and Python requirement	Command-line tool installed from PyPI, Python 3.11 or newer, developed on Linux and macOS	Installed from PyPI with pip, Python 3.12 or newer
Identity integration (SSO, RBAC, audit logs)	Not stated in the supplied facts; confirm with NVIDIA through the project documentation before relying on it	Not stated in the supplied facts; the v2 Giskard Hub is not planned for v3, so confirm any multi-user controls directly with Giskard

What is free, what is maintained and what is retired

garak is licensed under Apache 2.0 and is developed in NVIDIA's GitHub organization. The maintainers describe it as a free tool, and the user guide is published at docs.garak.ai. The most recent release in the supplied facts is v0.17.1, published on 16 October 2026, so pin a version in your CI job and read the release notes before each upgrade.

License: Apache 2.0 covers the scanner itself, so you can run it inside your own cloud account or data center without a license fee for the tool.
Maintenance: Releases are tagged on the NVIDIA repository, which lets you pin and test a specific version.
Installation: garak installs from PyPI with pip, and the PyPI package requires Python 3.11 or newer.
Documentation: The user guide at docs.garak.ai is the reference for probes, detectors, generators and harnesses.
Costs that remain: Model API usage, compute for repeated runs, and staff time to write endpoint configurations and review JSONL reports are yours to budget.

Confirm with NVIDIA's repository and your legal team whether any commercial support or hosted offering exists, and which terms apply, before you assume one is available.

What Giskard's v3 rewrite means for teams on v2

Teams on Giskard v2 should plan for a rebuild of their test suites, not an upgrade of a config file. In v3, vulnerability_scan supersedes the v2 LLM scan, and quality_scan supersedes RAGET, the v2 RAG evaluation toolkit. The legacy tabular and ML scan and the v2 ML test suite are not planned for v3, and neither is the Giskard Hub. The v2 release remains available but is no longer actively maintained, so any ML tests you depend on need a longer-term home that is not Giskard v3.

The Python requirement is the first migration constraint. Giskard 3.0.1 requires Python 3.12 or newer, while garak needs only 3.11 or newer, so a shared runner or base image may need an interpreter upgrade before any test moves. Install with pip install giskard plus the [scan] and [openai] extras, and confirm the Python version your v2 suites run on today. The v3 structure also changes the code itself: the scenario API and checks live in giskard-checks, and the scan features live in giskard-scan, so existing v2 test definitions have to be rewritten against those interfaces. The giskard-oss repository documents the current split.

Before you decide, inventory every v2 test and tag it with its v3 destination. Map LLM scan tests to vulnerability_scan and RAG evaluation tests to quality_scan, then mark anything tabular, ML or Hub-dependent as having no v3 equivalent. Those items need a decision: keep them on a pinned v2 environment with a documented end date, or retire them and replace the coverage with another tool. Also record which judge model your checks rely on, since v3 defaults to openai/gpt-4o-mini, and confirm whether that model call is permitted under your data handling rules. Telemetry can be disabled with DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1.

What drives the running cost of each tool

Neither garak nor Giskard charges a license fee for the open-source code, so the cost of running either one sits in people, model calls, storage and upgrades. The cards below separate what you pay once, such as writing suites, from what recurs on every run, and they list what to ask for before you commit to a scope.

Engineer time for suite authoring

garak ships probes that generate attacks, so most setup work goes into choosing the probes that match your application and writing the REST generator YAML that describes your endpoint. Giskard v3 asks you for a plain-language description of the agent, and its vulnerability scan builds the adversarial suite from that text, so the authoring effort shifts toward describing the agent accurately and reviewing what the scan produces. Either way, this is a one-time cost that recurs whenever the target changes its tools, prompts or scope.

Judge model calls on every run

Giskard's built-in LLM-judge checks, including Groundedness and Conformity, use openai/gpt-4o-mini as the default judge model unless you change it, so each scan and check run sends judge requests that you pay for through your provider account. garak's detectors are the part to compare here: you should confirm which of its detectors call a model and which rely on local rules before you estimate recurring judge usage. Ask whether the judge can be pointed at a model inside your own infrastructure if data residency rules apply.

Target inference on repeated runs

Every attack a scanner sends becomes an inference call against your model or agent, and that cost recurs on each run. garak makes 5 generations per prompt by default, so a probe set multiplies quickly across an endpoint you bill per token or per request. Reduce the probe list to the categories that match your threat model for routine runs, and reserve the full sweep for release gates. Confirm with the target owner who pays for test traffic and whether it can run against a staging copy.

Storage and retention of run reports

garak writes a JSONL report of every attempt and its evaluation, plus a garak.log file that continues across runs, so report volume grows with each scheduled scan. Giskard's scan output is held in its own run artifacts and differs in format, so compare how each lands in your logging pipeline. Set a retention rule before the first scheduled run, decide whether prompts and outputs may be stored at all, and confirm who has read access to reports that may contain sensitive model output.

Upgrades and version pinning

Both tools release often, and garak's v0.17.1 release is dated October 16, 2026 according to its releases page, so probe and detector changes can alter results between versions. Giskard 3.0.1 requires Python 3.12 or newer and splits functionality into giskard-checks and giskard-scan, so a version jump may also mean a runtime upgrade. Budget recurring time to re-run baselines after each upgrade, and pin versions in your CI configuration so a scan result can be reproduced later.

Migration and external support

Teams on Giskard v2 face a one-time migration cost, because the README states v2 is no longer actively maintained and the legacy ML scan and Giskard Hub are not planned for v3. External support, whether from your own staff, a vendor or a services partner such as DeployOpen, is priced from a written scope and recurs only if you buy a retainer. Ask any provider to itemize setup, suite authoring, judge usage, upgrade testing and ongoing support separately so you can see which lines recur.

Moving between garak and Giskard without losing test history

This sequence keeps your past findings readable and comparable while you rebuild the active test suite on the other tool. Reports and target definitions move as documentation, and the new suite proves itself against the old one before anything is retired.

Export and archive existing reports
The team that owns the current scanner copies every run output into a dated archive outside the active working directory. For garak, that means the JSONL report of every attempt and its evaluation, the hit log of attempts that found a vulnerability, and the garak.log file, which carries across runs and is the record of how each run was configured. For Giskard v2 users, export the scan results and the test configuration in whatever form your installation produces. Done when you can open each archived file without the original tool and each run is labeled with its date, target and tool version.
Map threat categories across both tools
Build a table with one row per threat category your program tracks, such as prompt injection, jailbreaks, data leakage, hallucination, toxicity and the OWASP LLM Top 10 categories. For each row, record which garak probes and detectors cover it today and which Giskard vulnerability scan categories cover it. Mark gaps in both directions, since neither tool's coverage maps one to one onto the other. Have the security lead sign off on the table. Done when every category in your program has a named owner on each side or is recorded as an accepted gap.
Document targets as reusable definitions
Write down each target the current suite tests: the model or agent, its endpoint, authentication method, system prompt version, tools available to it and the owner. For garak, the REST generator is configured with a short YAML file that describes the endpoint, so keep those files in version control with the archive. For Giskard, record the agent description you used and the wrapper that exposes the target. These documents transfer as is, because they describe the system rather than the tool. Done when a colleague who has never used either tool can identify each target and how to reach it.
Rebuild probes or agent descriptions
This is the step that cannot be copied. For garak, select probes and confirm the detectors that the default probewise harness runs for each one, then decide where the default 5 generations per prompt suits your budget and where it does not. For Giskard v3, write a plain-language description of each agent that the vulnerability scan uses to generate its adversarial suite, and write the built-in checks, such as regex matching, semantic similarity or LLM-judge checks, for behavior you need to pin down. Confirm which judge model each check uses. Done when each mapped category has a runnable configuration and a reviewer has checked it against the category table.
Run both suites in parallel for one release
Keep the old suite running on the same targets through one full release cycle while the new suite runs alongside it. Compare findings by category, not by raw count, because the two tools generate attacks differently and their failure criteria differ. Investigate every case where one tool flags a failure and the other does not, and record whether the difference is a coverage gap, a detector difference or a real regression. Confirm the release cycle length with your own release calendar. Done when the team has a written list of discrepancies, each marked with a decision.
Verify the archive before retiring the old suite
Before anything is removed, check that each historical run can still be retrieved, that the archived JSONL reports parse and that the garak.log history for each target is complete. Store a copy of the category table, target definitions and parallel-run comparison with the archive so future auditors can see what was tested, when and against which configuration. Confirm that telemetry settings match your policy: Giskard can be set to send no telemetry with DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1. Done when a second person has restored one archived run from scratch and confirmed its contents.
Retire the old suite and record the change
Disable the old scheduled jobs, remove the old suite from CI and document the cutover date, the owner of the new suite and the location of the archive in your test inventory. If you are moving from Giskard v2 to v3, note that v2 remains available but is no longer actively maintained, and that the legacy tabular and ML scan, the v2 ML test suite and the Giskard Hub are not planned for v3. Keep the old environment readable for the archive period your policy requires. Done when the inventory shows one active suite per target and the retired suite is marked read only.

Where garak and Giskard projects usually run long

Most of the calendar time in a garak or Giskard project goes to decisions and wiring, not to running the tools. On the garak side, the first build step is describing the target. For a custom REST endpoint, the garak REST generator reads a short YAML file that describes the endpoint, so someone has to write that file against the real request and response shape, including authentication headers and how the reply is parsed from plaintext or JSON. For Hugging Face, OpenAI, AWS Bedrock or a llama.cpp model, the generator configuration is simpler, but credentials, network paths and model access still need sign-off. On the Giskard side, the work is wiring an LLM, a black-box agent or a multi-step pipeline into the v3 scan. Confirm the Python version first: giskard 3.0.1 requires Python 3.12 or newer, while garak's PyPI package requires Python 3.11 or newer, so a shared build host may need two interpreters. Then install the right extras, such as [scan] and [openai], and confirm the agent wrapper can be called from the scan process.

The slow parts are usually organizational. Agreeing the threat model comes first: which OWASP LLM Top 10 categories, which data and tools the agent can reach, and which failures count as findings. Giskard's vulnerability scan builds its suites from a plain-language description of the agent, so that description is itself a control that security and product owners should review and version. Judge model selection is the next bottleneck. Giskard's built-in LLM-judge checks default to openai/gpt-4o-mini, so a regulated team has to decide whether that judge is permitted, how it is accessed, and whether a self-hosted or approved alternative is needed. Ask the vendor and your own legal and data protection staff to confirm where judge prompts and outputs travel before any suite runs. Two more issues surface late. Mapping findings to remediation owners often takes longer than the scan itself, because a garak hit log or a Giskard report has to be triaged into application, prompt, model and platform fixes with named owners. CI integration also needs design: scans are slow enough to run on a schedule or on release candidates, and failing builds on non-deterministic results breaks trust quickly. Plan for repeated runs, a documented threshold for what fails a build, and a way to rerun and compare results. Expect that garak's default of 5 generations per prompt and LLM-judged Giskard checks will both produce variance, so set the pass criteria before the first baseline.

Questions buyers ask before choosing garak or Giskard

Is garak or Giskard free to use?

Both open-source libraries carry no license fee, because both are published under the Apache-2.0 license. garak is Apache 2.0 licensed, and its maintainers describe it as a free tool. Giskard's open-source library lives in the Giskard-AI/giskard-oss repository under the same license. Your recurring costs sit elsewhere: model API usage when a scanner calls a hosted target, and the judge model Giskard uses for LLM-judged checks, which defaults to openai/gpt-4o-mini. Engineering time and the infrastructure that runs the scans are also yours to budget. Confirm the license terms of any extra packages or hosted services you add, in writing, before you standardize on one.

Which tool needs the least setup for a model endpoint?

For a plain model endpoint, garak usually needs less setup. It installs from PyPI with pip, requires Python 3.11 or newer, and its REST generator is configured with a short YAML file that describes the endpoint. Giskard v3 requires Python 3.12 or newer and installs with pip install giskard plus optional extras such as [scan] and [openai]. Giskard can wrap an LLM, a black-box agent or a multi-step pipeline, so a bare endpoint usually needs a small Python wrapper first. garak is quicker for a one-off probe of a hosted model, while Giskard takes more setup and returns a scenario-based test suite. Authentication and response format will drive the effort for either tool, so confirm both before you plan the work.

Can Giskard test a black-box agent that garak cannot wrap?

Giskard v3 is built for black-box agents, and garak reaches more endpoints than many readers assume. Giskard can wrap a black-box agent, and its vulnerability scan generates adversarial suites from a plain-language description of that agent. garak's REST generator connects to any endpoint that returns plaintext or JSON, so many agents exposed over HTTP are reachable by garak as well. The practical difference lies in test design: garak runs its own probes and detectors against the target, while Giskard's agent description shapes the attacks. If your agent exposes multi-step tool calls or session state that the REST generator cannot represent, Giskard is the stronger first test. Confirm how your agent's tool calls appear in each tool's wrapper before you commit.

What should a Giskard v2 team do first?

Start by inventorying which v2 features your team actually uses, because several are not carried into v3. The giskard-oss README states that v2 remains available but is no longer actively maintained, and that the legacy tabular and ML scan, the v2 ML test suite and the Giskard Hub are not planned for v3. In v3, vulnerability_scan supersedes the v2 LLM scan, and quality_scan supersedes RAGET, the v2 RAG evaluation toolkit. Next, set up a Python 3.12 environment, install giskard 3.0.1 in a separate project, and run both versions against one agent to compare findings. Decide early how you will replace any ML test suites you depend on. Set your retirement date for v2 in your own change calendar, since no vendor date is supplied here.

What does the work need from our engineers?

Your engineers need Python skills, comfort with YAML and HTTP endpoints, and time to read scan output. With garak, the work includes writing the endpoint YAML, choosing probes and detectors, and reviewing the JSONL report of every attempt along with the hit log. With Giskard v3, the work includes writing a wrapper for your agent, choosing the judge model, and building checks such as string, regex or semantic similarity matches. Either tool also needs one owner who triages findings and decides which failures block a release. Wiring either into CI adds a pipeline job, secure handling of model keys and a place to store reports. The effort scales with the number of endpoints and agents you cover, so ask for that sizing in a written scope.

Can we run either without sending data to a vendor?

Both tools run on your own infrastructure, but the model you test decides where prompts go. garak runs as a local command-line tool and writes its reports to your disk. If the target is a hosted API such as OpenAI, AWS Bedrock or Hugging Face, prompts reach that provider. Giskard's default judge model is openai/gpt-4o-mini, so LLM-judged checks send content to OpenAI unless you point the judge at a model you host. Giskard's telemetry is optional, and its README states that it never includes prompts or outputs. You can disable it with DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1. Confirm the judge configuration and outbound network access in your environment before the first run.

How much does an outside implementation cost?

DeployOpen does not publish a price list, and a quote follows from a written scope. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. The main cost drivers are the number of endpoints and agents to test, whether you need CI integration, how many custom probes or checks you want, which judge model and environment must be supported, and whether you want ongoing retainer-based testing and upgrades. Ask the quote to separate one-time work, such as setup, wrapper development and the first test run, from recurring work, such as retainers, model API usage and re-testing after upgrades. Ask for each deliverable and its acceptance criteria in writing.

How DeployOpen helps choose and implement either tool

DeployOpen does not start from a preferred scanner. The choice between garak and Giskard is made from a written scope that reflects what you need to test: a model endpoint, an agent built on tools and memory, or both. DeployOpen plans, implements, integrates, hardens and documents self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and it runs security testing that includes AI and agent security. For this decision, that means the first deliverable is usually a short written comparison of your target types, your judge model options, your report format and your CI constraints, so the tool follows from your requirements and not from a vendor relationship. DeployOpen does not hold partner status with either project and does not claim certifications for either tool.

Most DeployOpen engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. For a garak or Giskard deployment, the written scope should name the items that drive effort: the generators or endpoints to connect (for garak, a rest generator configured in YAML for a custom endpoint, or a supported provider), the probes and detectors to run, where the JSONL report and garak.log are stored, and how results reach your security team. For Giskard v3, it should name the agent wrapper, the plain-language description used to build the vulnerability_scan suite, the judge model and whether telemetry is disabled with DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1. Ask for the scope to state which items are one-time setup and which recur. Confirm current release details against the garak releases page and the giskard-oss releases page before the scope is signed.

Once a tool is in production, a single scan is rarely enough, because models, system prompts, tools and retrieval sources change. DeployOpen offers ongoing support, upgrades and continuous testing as retainers, which can keep scan configurations current, review new findings, and rerun suites after a model or agent change. Decide early whether you need a one-off assessment, a repeatable pipeline scan, or a retainer that covers both, since that choice shapes the quote. DeployOpen does not publish a price list. A quote follows from the written scope, so the most useful next step is to send your target list, your Python and hosting constraints, and your reporting requirements, and ask for a fixed-scope proposal that separates setup, testing and retainer work.

garak vs Giskard