promptfoo or garak: which fits your release pipeline
promptfoo fits application-level evals and red teaming in CI, and garak fits model-level vulnerability scans. A team with an LLM application that must pass checks before release usually needs to decide what the gate is meant to protect: the behavior of the product as users reach it, or the weaknesses of the base model underneath it. promptfoo runs declarative configs locally and can run checks in CI/CD pipelines, which makes it a natural fit when a failing test should block a merge or a deploy. garak, a command-line scanner built from probes, detectors, generators and harnesses, fits when you want a broad sweep of a model or endpoint for hallucination, data leakage, prompt injection, toxicity and jailbreaks, with a JSONL report of every attempt and a hit log of attempts that found a vulnerability.
The decision turns on three questions. First, what is being tested: your system prompt, tools, retrieval pipeline and output handling point to promptfoo, while a question about how the base model behaves across attack families points to garak. Second, who owns the results: promptfoo is MIT licensed and, per its README, is now part of OpenAI, so your procurement and vendor review should cover that ownership change alongside the license. garak is Apache-2.0 and maintained in NVIDIA's GitHub organization. Neither ownership fact by itself favors one tool, but both belong in the review record. Third, whether checks gate a pipeline: if a release must stop on a result, confirm that the tool's exit behavior and report format fit your CI runner before you commit to it.
Run both on a copy of your current release candidate before you choose. Confirm which attack categories each covers for your use case, how results are stored and who can read them, and how long a rerun takes in your pipeline. For a team that wants both views, the tools are not mutually exclusive, since an application gate and a periodic model scan answer different questions.
Which scenario points to which tool
This table maps common situations to the better fit between promptfoo and garak and states the condition that would reverse each choice. Read it by finding your scenario, then checking the reversal condition against your own systems before you commit.
| Scenario | Better fit | Why and what would reverse it |
|---|---|---|
| Gating a prompt or model config change in CI | promptfoo | Its declarative configs run locally and can execute checks in CI/CD pipelines, so a failing assertion can block a merge. It would lose this role if your pipeline must gate on model-level vulnerability scores, where garak's probe and detector results are the more direct signal. |
| One-off model assessment before selecting a model | garak | A single CLI run against a model through its generators produces a JSONL report of every attempt and a hit log of successful attacks, which suits a comparison of candidates. It would reverse if the selection depends on how the model behaves inside your own prompts, tools and data, which promptfoo tests more directly. |
| Testing a RAG or agent app with its own system prompt and tools | promptfoo | Its configs target the application as deployed, so the system prompt, retrieval and tool wiring are exercised together. It would reverse if you only need to know whether the base model itself is vulnerable, independent of your application layer. |
| Security team scanning several model endpoints | garak | Its generators cover Hugging Face, OpenAI, AWS Bedrock, llama.cpp and custom REST endpoints, so one probe set can run across a fleet. Confirm that your specific endpoints and auth patterns are supported before standardizing on it. |
| Recurring regression testing after each release | promptfoo | Re-running the same config on every build produces comparable results over time and fits pipeline tooling. It would reverse if your release process depends on a vendor-neutral tool with no dependency on an OpenAI-owned product, which is a procurement judgment to make explicitly. |
| Vendor independence is a hard requirement | garak | garak is Apache-2.0 and maintained in NVIDIA's GitHub organization, with no change in ownership announced in the supplied facts. Promptfoo is MIT licensed and open source, but it is now part of OpenAI, so confirm your governance position before adopting it as a control. |
| Mapping results to OWASP LLM Top 10 or NIST AI RMF | promptfoo | Its red team plugins and strategies are aligned with both frameworks in its documentation. It would reverse if your evidence must come from a tool whose probes you can trace one by one to a named taxonomy, which you should verify against garak's documentation. |
How each tool is built and what that means for operations
promptfoo and garak both run from the command line on infrastructure you control, but they organize testing differently. One is built around configuration files you maintain for each application, the other around a library of probes and detectors that you select and run against a model endpoint.
promptfoo as a config-driven CLI
promptfoo is a CLI and library that runs evaluations and red team checks from declarative configs. Someone on the team owns the config file, which defines the prompts, providers and assertions for your application. Changes to prompts or model settings show up as diffs in version control, which makes the config the main operational artifact. Check who is allowed to edit it, since a loosened assertion can quietly make a failing test pass.
promptfoo in CI/CD pipelines
promptfoo runs locally and can execute its checks inside CI/CD pipelines, so a failed assertion can block a merge or release. The pipeline job needs access to the model provider and the test inputs, and it inherits the secrets that job holds. Confirm where the job runs, which API keys it can read, and whether red team results are retained as build artifacts that other teams can open.
garak as a probe and detector scanner
garak is a command-line scanner made of four parts: probes that generate attacks, detectors that judge whether the model failed, generators that connect to targets, and harnesses that run them together. Your team chooses which probes to run against which generator, so the operational decision is the probe selection and the target configuration. Record those choices with each run, because two scans with different probe sets are not directly comparable.
garak generators and target connections
garak's generators connect to targets such as Hugging Face, OpenAI, AWS Bedrock, llama.cpp models and custom REST endpoints. This suits teams that want to scan a model directly, including a local model that never leaves their network. Each generator carries its own credentials and endpoint settings, so the scan is only as good as the target configuration you hand it. Confirm that the endpoint under test matches the production model and system prompt.
Where garak stores results
garak writes a JSONL report that records every attempt, plus a hit log listing only the attempts that found a vulnerability. Those files are local output, so retention, access control and transfer to your reporting system are your responsibility. Decide where the files land, who can read the hit log, and how long they are kept before a release review.
Where promptfoo stores results and ownership
promptfoo's results live with the config and the environment that ran it, and its README states that Promptfoo is part of OpenAI while remaining open source under the MIT license. Ownership affects vendor review, not the local run model. Ask your procurement and security teams to confirm how results, prompts and provider traffic are handled in your environment, and whether that review needs to be repeated for your data classification.
How promptfoo and garak run inside a release pipeline
Developers use both tools as command-line programs on a workstation while they change prompts, retrieval logic or model choices. promptfoo is built around a declarative config file that lists providers, prompts and test cases, so a developer can run the same evaluation locally that CI will later run. garak is a scanner: you point it at a generator (a model or REST endpoint), choose probes and let its detectors judge the output. Local runs are where most configuration mistakes surface, so they are the cheapest place to learn each tool's behavior.
- Config as the unit of work: promptfoo runs from a checked-in config file, so a developer's local run and the pipeline run can share one definition.
- Probe selection: garak runs are shaped by the probes and generators you name on the command line, which makes scope easy to vary but also easy to forget to record.
- Credentials on laptops: provider API keys for either tool should come from a secrets manager or environment variables, not from files committed to the repository.
- Output location: garak writes a JSONL report of every attempt plus a hit log, so decide where local reports are stored and who may read them.
- Model cost per run: every test case or probe attempt calls a model, so local runs against paid endpoints carry usage charges that recur on each run.
Confirm with each vendor which providers and generators your team needs, and how your organization's data handling rules apply to prompts sent from developer machines.
On a pull request, the question is whether a change makes the application worse or introduces a known failure. promptfoo is designed for this use: its documentation describes running checks in CI/CD pipelines, and its config can fail a build when assertions do not pass. Whether garak belongs in this gate depends on runtime. A full garak scan can make many model calls, so most teams run a narrower probe set on pull requests and keep broader scans for a separate schedule. Set expectations early so that a slow or flaky gate does not get bypassed.
- Assertion-based gates: promptfoo can fail a job on specific checks, which suits regression tests for prompts and expected behaviors.
- Red team checks in CI: promptfoo red team plugins and strategies can run as part of the same workflow, aligned with the OWASP LLM Top 10 and the NIST AI Risk Management Framework according to its documentation.
- Narrow garak scope: a limited set of probes on each pull request keeps run time and model spend predictable, with the full set kept for scheduled runs.
- Build secrets: the CI system must hold provider keys with the least privilege needed and must not expose them to forked pull requests.
- Stored results: keep the promptfoo output and garak JSONL reports as build artifacts so reviewers can see which test failed.
Check how your CI platform handles secrets for external pull requests and how long artifacts are retained under your records policy.
Scheduled scans suit broad coverage: a full garak probe run against a deployed model, or a larger promptfoo red team pass, on a fixed cadence rather than on every change. This is where you also catch regressions caused by a provider updating a model behind an unchanged endpoint. The schedule should be set by the risk owner, not the tool, and each run should record which tool version, probe set and target configuration were used so results stay comparable over time.
- Pinned versions: record the exact garak and promptfoo versions used in each scheduled run, since results from different releases may not be directly comparable.
- Target identity: log the generator or provider name and model identifier for each scan so a finding can be traced to a specific configuration.
- Trend tracking: garak's hit log and JSONL attempts can be compared across runs, which supports tracking whether a finding persists.
- Ongoing model spend: scheduled scans recur, and their cost scales with the number of probes, attempts and the provider's pricing for each call.
- Alert routing: decide who receives a new hit, and whether it opens a ticket automatically or goes to a triage queue.
Confirm with each vendor how their release notes describe changes to probes and detectors, because those changes affect whether a score can be compared to last month's.
When the target model or application runs inside your own network, both tools usually need to run there too, on self-hosted runners or a scanning host that can reach the endpoint without traversing the public internet. promptfoo runs locally from declarative configs, which makes it straightforward to run inside a private network if its providers are reachable from there. garak also runs as a command-line program and can target custom REST endpoints, which is the usual path for an in-house model. Each run still sends prompts somewhere, so confirm which calls leave the network, especially when a scanner uses a hosted judge model.
- Network reachability: the runner needs a route to the target endpoint and, if used, to any hosted detector or judge model.
- Egress review: check whether either tool, or any provider it calls, sends data outside your boundary, and document the answer for your security team.
- Dependency sourcing: mirror Python packages and container images internally so runners do not depend on public registries during a scan.
- Runner isolation: keep the scanning host separate from production credentials and give it only the access that the scan requires.
- Upgrade path: plan how each tool's version will be updated in an air-gapped or restricted environment, since updates must be staged through your own repository.
Ask your provider or internal platform team to confirm the exact egress endpoints before a scan is approved.
Attack coverage, reporting and extensibility compared
This table compares how promptfoo and garak differ on licensing, maintainer, current release, attack coverage, reporting and extension points. Read each row as a question to answer for your own pipeline, and treat the cells marked for confirmation as items to verify in the vendor documentation before you commit. The main takeaway is that promptfoo is built around application-level evals and CI checks, while garak is built around scanning a model with probes and detectors.
| Capability | promptfoo | garak |
|---|---|---|
| License | MIT licensed CLI and library | Apache-2.0 licensed command-line scanner |
| Maintainer | Part of OpenAI following the March 9, 2026 acquisition announcement; remains open source | Maintained in NVIDIA's GitHub organization |
| Current release | 0.124.1, published October 8, 2026 | v0.17.1, published October 16, 2026 |
| Primary test target | Your application: prompts, models, RAG flows and agent behavior defined in declarative configs | The model or endpoint itself, scanned through probes and generators |
| Attack coverage | Red team plugins and strategies aligned with the OWASP LLM Top 10 and NIST AI RMF; confirm the current plugin list in the red teaming docs | Probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks; confirm the probe catalog for your version |
| Reporting | Eval results and red team reports generated from config runs; confirm output formats and CI artifacts in the docs | JSONL report of every attempt plus a hit log of attempts that found a vulnerability |
| Target connectors | Configurable providers for model APIs and custom endpoints; confirm supported providers for your stack | Generators for Hugging Face, OpenAI, AWS Bedrock, llama.cpp and custom REST endpoints |
| Custom extension | Custom plugins, strategies and assertions defined in config; confirm the extension interface for your version | Custom probes, detectors and generators written against the framework; confirm the plugin interface before building on it |
What OpenAI's acquisition of promptfoo means for buyers
Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI, and its README now states that Promptfoo is part of OpenAI and remains open source and MIT licensed. The license is the fact you can verify today: the MIT terms in the repository govern the code you deploy, and they do not depend on who owns the company. What the license does not settle is the roadmap, the maintainer's priorities, or how the project's governance will look after the acquisition. Those are questions for the vendor, and they belong in writing before you build a release gate around the tool.
The more important question for this comparison is neutrality. Promptfoo generates adversarial inputs and scores model responses, so if your evaluation targets OpenAI models, the party that owns the scanner also owns a model vendor under test. That does not make results invalid, but it does mean a buyer should not treat a pass as independent evidence without a second check. garak, maintained in NVIDIA's GitHub organization and published under Apache-2.0, has its own vendor relationship with the models it might scan, so the same discipline applies to both tools. A practical control is to run the same target suite on a second scanner and keep the attack set and detector settings under your own version control, so the evidence does not depend on one owner's judgment.
Before you commit, ask for the following in writing: confirmation that the MIT license applies to the version you will pin and to any future releases you adopt; a statement on whether red team plugins, strategies and reporting features will remain in the open-source distribution or move to a paid tier; the process for security fixes and the expected notice period for breaking changes; and how data sent to any hosted or cloud feature is handled, including whether results reach OpenAI systems. Record the answers in your vendor file and reconfirm them at each upgrade. If any answer is unclear, plan for a switch path and test it early.
License and editions for each platform
promptfoo is an MIT-licensed CLI and library for evaluating and red teaming LLM applications. Its documentation and README describe it as a tool that runs locally from declarative configuration files, so the evaluation harness can sit inside your own build infrastructure without a required external service. The most recent release in the supplied facts is version 0.124.1, published on October 8, 2026.
- MIT license: You may use, modify and redistribute the open-source project under permissive terms, which suits internal forks and embedding in internal tooling.
- Local execution: Configs, prompts and test results can stay within your repository and runner, which matters when test data includes regulated content.
- Ownership change: Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI, and its README states that it is part of OpenAI and remains open source and MIT licensed.
- Release cadence: Pin a specific version in your pipeline and review release notes before upgrading, since the project publishes frequent releases.
- Hosted options: The supplied facts do not describe a paid or hosted edition.
Ask the vendor in writing whether a hosted, enterprise or support offering exists, what it includes, and whether any feature you rely on sits behind a paid tier.
garak is a command-line LLM vulnerability scanner licensed under Apache-2.0. The repository is maintained in NVIDIA's GitHub organization, and the supplied facts list v0.17.1 as published on October 16, 2026. Because the project is a scanner you run against a target, its licensing governs the code you execute, while the models and endpoints you point it at carry their own terms.
- Apache-2.0 license: Permissive terms with an explicit patent grant, which legal teams often review as a separate item from MIT.
- NVIDIA maintenance: The project lives in NVIDIA's GitHub organization, so check the release and issue activity yourself before you rely on it for a standing control.
- Target licensing: Probes sent to a hosted model may fall under that provider's usage policies, so confirm those terms before running large scans.
- Pinned versions: Lock the version in your environment because probe and detector sets change between releases.
- Paid editions: The supplied facts do not describe a paid or hosted edition of garak.
Confirm with NVIDIA or your legal team whether any commercial support exists for garak, and check the Apache-2.0 notice requirements against your internal distribution process.
Neither tool's supplied facts describe paid tiers, enterprise licenses or service-level commitments. For a fixed-scope comparison, the cost of both tools is mostly the engineering time to integrate them, run tests, triage findings and maintain configurations, plus the compute used for test runs and any model API charges for attacks and judging. Open-source licensing does not remove those costs.
- One-time costs: Initial setup, writing the first test suite or scan profile, and integrating results into your reporting.
- Recurring costs: Model API usage for each run, runner compute, upgrades, and the review time for new findings.
- Cost drivers: The number of models under test, the frequency of runs in CI, and how many attack categories you enable.
- Quote request: Ask for the scope of the test suite, the target models, the run frequency and the upgrade plan, since those drive effort more than license choice.
- Vendor confirmation: Ask each vendor directly about hosted, enterprise or support options and their terms.
DeployOpen scopes implementation and testing work through a written proposal, so describe your targets and run frequency when you request one.
Identity integration and custom attack logic
Neither promptfoo nor garak is primarily an identity product, so single sign-on, secrets and audit trails sit in the surrounding pipeline and in the target's own authentication. The cards below cover how each tool reaches a custom endpoint, how each is extended with your own attacks, and what to confirm before either one touches production credentials.
Authentication lives in the target
garak reaches targets through generators, including custom REST endpoints, and promptfoo reaches them through providers defined in its config. In both cases the tool sends whatever credentials the target expects, so the real control point is the target's authentication. Confirm that the scanning identity is a dedicated, scoped service account with its own rotation and revocation, not a shared developer key.
Secret handling in CI
Both tools read API keys and endpoint tokens from environment variables or the pipeline's secret store rather than from code. Keep keys out of committed config files and out of the JSONL reports and promptfoo result files, which can record request details. Ask your platform team whether the CI runner's secret injection, masking and log retention meet your policy.
Custom REST and provider wiring
Point garak's REST generator at an internal chat endpoint, or define a custom provider in promptfoo's config, to test the exact application path your users hit. Testing the full path, including your own authentication proxy and system prompt, usually matters more than testing the bare model. Check that each adapter handles your token refresh flow, timeouts and response format before trusting results.
Writing custom probes and plugins
garak extends through its own probes and detectors, so a custom attack for your domain means writing Python that generates prompts and judges the output. promptfoo supports custom assertions and red team plugins in its config and code. For a bank or health workflow, the useful question is who will maintain these attacks as policies change, and whether your team can review them.
Enterprise identity and audit logging
Neither tool provides SSO, role-based access or a central audit log of who ran which scan. Those controls come from your CI system, your artifact store and your logging stack. Ask each vendor or your implementation partner to confirm what the local output records, how long it persists and where it can be exported to your SIEM.
Confirm before production use
Before connecting either tool to a production-like target, confirm the license terms you need, the version you pin, and who in OpenAI's ownership context can see your configuration if you use any hosted features. Also confirm that no test data leaves your environment unless you have chosen to send it. Record the answers in your third-party risk file.
Moving from promptfoo to garak or back
This sequence lets a team move its LLM security checks from promptfoo to garak, or the reverse, without losing the attack intent and pass criteria it has already agreed on. It ends with the old pipeline stage retired and a record that both tools produced comparable results on the same targets.
Inventory existing test cases and attack categories
The platform or security engineer who owns the current pipeline lists every promptfoo test case, assertion and red team plugin in use, grouped by attack category such as prompt injection, data leakage, jailbreaks and toxicity. For each entry, record the pass criteria and the business reason it exists. The step is done when the team has a single inventory file that a reviewer outside the team has signed off on, so that nothing is dropped silently during the move.
Map each case to the other tool
Using the inventory, the engineer finds the closest garak probe and detector pair for each category, or the nearest promptfoo plugin and strategy if moving the other way. Some promptfoo checks, such as assertions on application output formats or business rules in a config, have no direct counterpart in a model scanner and must be flagged as gaps. Confirm each mapping against the current garak and promptfoo documentation for the versions in use. The step is done when every inventory line has a mapped target or a documented gap with an owner.
Rebuild targets and credentials
The platform engineer recreates each target in the new tool. In promptfoo this means provider definitions in a declarative config. In garak it means choosing a generator for the model endpoint, such as a Hugging Face, OpenAI, AWS Bedrock or custom REST connector. Secrets move into the pipeline's secret store with the same scoping as before, and no key is pasted into a config file. The step is done when a smoke run reaches each target and the team has confirmed the target names and model versions match the production inventory.
Rebuild detector logic and report handling
The security engineer rewrites any custom judgment logic in the new tool's terms. garak uses detectors to decide whether an attempt failed, so promptfoo assertions that check for a specific refusal or leaked string need an equivalent detector or a documented exception. The team also confirms where each tool writes results: garak produces a JSONL report of every attempt and a hit log of successful attacks, while promptfoo produces its own result output. The step is done when a known-bad sample from the inventory is judged the same way by both tools.
Run both pipelines in parallel against a baseline
For a set period agreed in advance by the team, both the old and new stages run on every build against the same model version. The release owner compares pass and fail outcomes per attack category and investigates every disagreement by reading the raw attempts. Duration is set by the team's own release cadence, not by this guide. The step is done when the disagreement list is empty or each remaining difference has a written explanation and an accepted decision.
Carry over history and retire the old stage
The team archives the old tool's result history alongside the new reports so that trend comparisons over earlier releases stay available for audit and review. Attack intent, pass criteria and the inventory transfer directly, while configs, connectors and detector code are rebuilt. The pipeline owner then removes the old CI job and its secrets, and records the change in the release runbook. The step is done when the old job no longer runs, its credentials are revoked, and the archive is verified as readable by someone other than its author.
Where implementation projects usually run long
Implementation projects for promptfoo or garak usually slow down at the decisions around them, not at the install. The first is setting pass and fail thresholds. promptfoo asserts results per test case in a declarative config, so a team must decide which assertions block a merge and which only report. garak returns a JSONL report of every attempt and a hit log of attempts that found a vulnerability, so a team must decide how many hits in which probe families fail a build. Neither tool sets that policy for you. Agree the threshold with the application owner and the security lead before the first pipeline run, and record it in the repository next to the config so reviewers can see it change.
The second slowdown is connecting the target securely inside the client's network. A promptfoo provider or a garak generator for a custom REST endpoint must reach the model or application from the CI runner, which often means private networking, short-lived credentials and a service account scoped to test traffic only. Confirm with your platform team where the runners sit, how API keys are injected from the secrets store, and whether test prompts and responses may be logged in the runner's artifacts. The third is wiring results into CI gates, where teams often discover that report artifacts have retention rules, that false positives block releases, and that someone must own triage. Handling false positives is an ongoing cost: detectors and assertions are judged by people, so budget review time with a named owner. The last slowdown is agreeing evidence for compliance readiness with the client's auditors. Ask the auditor early which artifacts they will accept: the config under version control, the run history, the threshold record, and the triage log for each hit. DeployOpen prepares these systems and evidence for assessments, but independent auditors and certification bodies decide what is sufficient. Before starting, ask the client to confirm the control framework in scope and the evidence format the auditor expects, since those answers change how much of the work is one-time setup and how much recurs with each release.
Questions buyers ask before choosing or hiring help
Which tool should I choose for a CI gate on every pull request?
For a per-commit gate on your application, promptfoo is usually the better starting point. It runs from declarative configs, can run checks in CI/CD pipelines, and its red team plugins and strategies are aligned with the OWASP LLM Top 10 and the NIST AI Risk Management Framework, so failing cases can block a merge the way unit tests do. garak fits better as a scheduled scan of the model itself. It writes a JSONL report of every attempt and a hit log of successful attacks that a security team can review after the run. Before you choose, ask three things: which findings should fail the build, how long the job can run in your pipeline, and who triages the hit log. Confirm run times in your own environment, because they depend on your target, rate limits and probe selection. The promptfoo red teaming documentation lists the plugins and strategies available.
Does OpenAI owning promptfoo affect how we can use it?
Ownership by OpenAI does not change the license terms you can read today. Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI, and its README states that Promptfoo is part of OpenAI and remains open source and MIT licensed. Your use still falls under the MIT license text in the repository, so have legal confirm that reading and keep a copy of the license with your build record. The practical questions are different. If your evals call OpenAI models, the tool vendor and a target vendor are the same company, so consider a second model family as judge or scorer for high-stakes results. Ask the vendor in writing about its roadmap, data handling for any hosted or optional features you enable, and whether your procurement policy needs a fresh review after the change of ownership. Pinning tool versions in your pipeline protects you from unplanned changes, whoever owns the project.
Can garak test a custom REST endpoint in our own stack?
Yes. The garak README lists custom REST endpoints among the generators it can connect to, alongside Hugging Face, OpenAI, AWS Bedrock and llama.cpp models, so a self-hosted model service or internal gateway can be the target. You need a stable request and response shape, the authentication method your endpoint uses, and a test credential with rate limits your team has approved. Map the generator configuration to what your service actually accepts, since a message-format mismatch produces errors that look like results. Then choose probes that match your risks. Prompt injection, data leakage and jailbreaks are common starting points. Confirm the exact configuration keys in the garak repository for the version you deploy, since v0.17.1 was published on October 16, 2026. An internal team that owns the endpoint can do this alone.
How long does setup take and what drives the timeline?
Setup time depends on what you connect, how many systems are in scope and how much of the pipeline you automate, so ask for a timeline in writing tied to a defined scope. The main drivers are the number of applications and model endpoints, the authentication each one requires, whether test data can leave your network, how results connect to your CI system, and whether reviewers need evidence in a specific format. Custom plugins or probes, and approvals for test traffic against production-like systems, add time. Recurring effort is a separate cost. Each tool release can change configuration, so budget for upgrades and regression runs. promptfoo 0.124.1 (published October 8, 2026) and garak v0.17.1 (published October 16, 2026) show how often versions move. A team with CI and Python experience can get a first configuration running alone, but expect later passes to take longer as scope grows.
Should we hire outside help or train our own team?
A capable platform or security team can run either tool alone for a first pass, especially if one engineer owns the configuration and another owns triage. Hire help when you need someone to define scope, tune policies to your application, or stand up a program that runs continuously. The cost components differ. Internal work costs staff time, which recurs each time a model, prompt or release changes. An outside fixed-scope engagement is priced from the written scope, and retainers cover ongoing support, upgrades and continuous testing. DeployOpen plans, implements, integrates, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and it runs security testing that includes AI systems and agents. DeployOpen does not publish a price list. Whichever provider you consider, ask who maintains the configuration after handover, what runbooks you receive, and how knowledge transfers to your team.
What does a fixed-scope engagement cover for AI security testing?
Most DeployOpen engagements are fixed-scope, and scope, deliverables and timelines are agreed in writing before work starts. For this comparison, a written scope usually names the applications and model endpoints under test, the risk categories to cover, such as prompt injection, data leakage and jailbreaks, how promptfoo or garak runs in your pipeline, the report format, and the handover documents. It should also state exclusions and who fixes the findings, which is often the least clear part of a quote. DeployOpen also offers retainers for ongoing support, upgrades and continuous testing. Price follows from the scope, and DeployOpen does not publish a price list, so ask for a quote tied to a numbered deliverable list. Ask what acceptance looks like for each deliverable and how changes to scope are priced.
Does a passing promptfoo or garak run prove compliance?
No. A passing run shows that the tool's probes did not succeed against your system on that day, under that configuration. It does not certify the system. promptfoo's red team plugins are aligned with the OWASP LLM Top 10 and the NIST AI Risk Management Framework, which helps you map coverage to a shared risk vocabulary, but mapping is not assessment. DeployOpen prepares systems, controls and evidence for compliance assessments, yet it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies perform that work. Before you build a pipeline around scan results, ask your assessor which evidence they accept, how often they expect runs, and how you will show that configurations and tool versions were controlled over time. Keep scan configs, versions and reports in your evidence store.
How DeployOpen helps choose and run either tool
DeployOpen plans, implements, integrates and documents self-hosted open-source platforms inside your own infrastructure, and it runs security testing, including AI and agent security. For promptfoo vs garak, we help you decide which tool fits your release pipeline or assessment cadence, then build and hand over the setup so your team can run it.
How we scope
- We confirm which models, endpoints and applications the tool will test and where results must be stored.
- We agree in writing which attack categories, CI stages and reporting outputs are in scope.
- Most engagements are fixed-scope, with deliverables and timelines agreed before any work starts.
How we build
- We install and configure the chosen tool inside your cloud account, data center or air-gapped network.
- We connect test runs to your CI pipeline and define pass, fail and review thresholds with your team.
- We document the probes, plugins or detectors enabled and record why each was selected for your risks.
How we hand over
- We deliver runbooks covering how to add targets, update configurations and read JSONL or dashboard reports.
- We walk your engineers and security staff through triage of findings and the escalation path.
- We provide evidence of configuration and test runs that your compliance team can use in readiness reviews.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- We track vendor release notes and plan upgrades, including changes in maintainership or license terms.
- We can rerun scans after model, prompt or endpoint changes so results stay tied to what is deployed.
