Which adversarial robustness library fits your team's situation
Choose ART when your team must test classic models, such as fraud scoring, tabular risk models, vision and audio classifiers, before release and needs one library that covers more than evasion. ART is the maintained option here. Its latest release, 1.20.1, was published on July 7, 2025, and the project is MIT licensed and hosted by the Linux Foundation AI & Data Foundation. Foolbox or CleverHans make sense only when the work is evasion-focused and the team accepts their release history. Maintenance comes first because a stale release affects everything downstream: framework compatibility, dependency updates and whether your security team can rely on upstream fixes. Before you commit, review the ART release history for anything newer than your planning date.
Foolbox's latest release, 3.3.4, was published on March 4, 2024, and CleverHans' latest release, 4.0.0, was published on July 24, 2021, with its last code push in April 2024. A recent commit is not the same as a tagged release, so ask which version your build pipeline will pin and whether it runs against your current framework versions. Foolbox fits a PyTorch, TensorFlow or JAX image classifier where the test plan is limited to evasion. CleverHans fits teams that already maintain its examples and accept a 2021 release for benchmarking. Confirm each library's support for your specific model types, such as gradient-boosted trees or audio pipelines, before assuming parity with ART, which lists TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy models in its README.
Map each model in scope to its framework and to the threats you must test before you pick a library. If poisoning, extraction or inference attacks are on the list, the choice narrows to ART, because the other two are evasion-focused in the supplied sources. Run a pilot on one fraud or tabular model and record the exact library version, since that version becomes part of your release evidence. The library itself is free under its license, so the recurring cost is engineering time to maintain the test harness and to re-run it after each model update. If you want an outside team to build that harness, ask for the scope, deliverables and timeline in writing before work starts.
Which library fits which scenario
This table maps common team situations to the library most likely to fit, with the condition that makes each choice sound. Read it alongside the maintenance facts above, since Foolbox and CleverHans cover evasion attacks only, as supplied, while ART covers evasion, poisoning, extraction and inference.
| Scenario | Best fit | Why |
|---|---|---|
| Regulated team testing fraud or tabular models across evasion, poisoning, extraction and inference | ART | ART covers all four threat categories and supports scikit-learn, XGBoost, LightGBM and CatBoost, which are common in fraud and credit models. Confirm that your model framework is on its supported list. |
| Research team comparing evasion attacks in PyTorch, TensorFlow or JAX | Foolbox | Foolbox is built for creating adversarial examples in PyTorch, TensorFlow and JAX. Its latest release is 3.3.4 from March 4, 2024, so check whether your framework versions still work before committing. |
| Teaching or benchmarking evasion attacks | Foolbox or ART | Foolbox fits a benchmark limited to evasion attacks on neural networks. ART fits a course or benchmark that should also cover poisoning or inference, though it has more surface to learn. |
| Team with an existing CleverHans pipeline | CleverHans, with a migration plan | CleverHans 4.0.0 dates from July 24, 2021, with its last code push in April 2024. Keep it for existing experiments, but plan a move to a maintained library for new work. |
| Team needing audio, video or object detection coverage | ART | ART works on images, tables, audio and video and covers tasks such as object detection and speech recognition. Foolbox and CleverHans are evasion libraries for neural networks, so confirm any non-image task with your own tests. |
Licensing and editions for ART, Foolbox and CleverHans
The Adversarial Robustness Toolbox (ART) is MIT-licensed and hosted by the LF AI & Data Foundation, which means its governance sits in a foundation rather than with a single company. Its latest release, 1.20.1, was published on July 7, 2025, so the project has a recent release on record as of October 2026. Because the license is permissive, you can use, modify and redistribute it inside your own systems without a license fee. The costs that remain are the engineering time to pin versions, test upgrades and maintain your own wrappers around the library.
- License: MIT, with no paid tier supplied in the research for this page.
- Governance: Hosted by the LF AI & Data Foundation, which can matter for procurement reviews that ask about neutral stewardship.
- Latest release: 1.20.1, published July 7, 2025.
- Internal cost: Upgrade testing and dependency management fall to your team unless you contract them out.
- Commercial support: No vendor support tier was supplied, so confirm directly with the project or your implementation partner what is available.
Before you commit, confirm with the project maintainers whether any commercial support options exist and which release branch they expect you to track.
Foolbox is an MIT-licensed Python toolbox for creating adversarial examples against neural networks in PyTorch, TensorFlow and JAX. Its latest release, 3.3.4, was published on March 4, 2024. That date is more than two years before the current research window, so you should check the repository activity before treating it as a maintained dependency. Licensing imposes no fee, so the cost question is whether you will need to fork or maintain it yourself.
- License: MIT, with no paid edition supplied in the research.
- Latest release: 3.3.4, published March 4, 2024.
- Framework scope: PyTorch, TensorFlow and JAX, which may align with your model stack.
- Internal cost: If you adopt it, plan for owning compatibility fixes when framework versions move.
- Commercial support: No commercial support was supplied, so ask any vendor or partner whether they will back it.
Confirm the current maintenance status on the repository and the PyPI page before you pin a version in a production pipeline.
CleverHans is an MIT-licensed adversarial example library for constructing attacks, building defenses and benchmarking. Its latest release, 4.0.0, was published on July 24, 2021, and the last code push was in April 2024. The gap between a release in 2021 and a code push in 2024 means the project should be read as a code base with limited release discipline rather than a versioned product. The license is no obstacle, but the upgrade path is your responsibility.
- License: MIT, with no paid edition supplied in the research.
- Latest release: 4.0.0, published July 24, 2021.
- Last code push: April 2024, per the repository.
- Internal cost: Expect to own dependency pinning and any fixes needed for newer framework versions.
- Commercial support: None was supplied, so confirm with the maintainers or a partner before relying on it for a regulated workload.
Check the repository's commit history and open issues to judge whether the April 2024 activity reflects ongoing maintenance or a one-off change.
Because the three libraries share an MIT license, the total cost of ownership is driven by work rather than by license fees. One-time costs include evaluating attacks against your models, writing test harnesses and integrating results into your reporting. Recurring costs include dependency upgrades, framework compatibility work and re-running tests after model changes. The main factors that raise cost are the number of model types you cover, the size of your evaluation matrix and how often your frameworks change.
- One-time work: Harness design, baseline evaluation and integration with existing model registries.
- Recurring work: Dependency upgrades, retesting after retraining and tracking security advisories.
- Cost drivers: Framework breadth, number of threat categories you test and the release cadence of upstream libraries.
- Quote items to request: Named scope, the frameworks and model types covered, and whether upgrade support is included.
- Ownership: Decide early whether your team or an implementation partner will own patches for the library you select.
Ask each vendor or partner to state in writing which costs are included in a fixed-scope quote and which will be billed separately.
What drives the running cost of each library
Running any of these libraries costs more in engineer hours than in licenses, since all three are open source under MIT. The cards below separate who pays for each item, from framework adaptation to switching later, and name what pushes each cost up or down. Use them to build the questions you put to your team and to any implementation partner before you commit.
Adapting models to each framework
Your effort depends on how your models are built. ART wraps TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy models through its estimator wrappers, so a fraud model in XGBoost or a credit scorer in scikit-learn may need little rework. Foolbox and CleverHans are centered on neural networks, so a gradient-boosted or tabular pipeline may need your own wrapper code. Ask each project which model types your team has already validated, and who on your side owns those adapters.
Pinning and upgrading stale dependencies
The last Foolbox release is 3.3.4 from March 4, 2024, and the last CleverHans release is 4.0.0 from July 24, 2021, so both pin older versions of PyTorch, TensorFlow or JAX that your platform team may need to upgrade. Each framework upgrade can break an attack script, and nobody upstream is obliged to fix it. Your platform team pays for this work, and it recurs with every framework release. Ask how each library's maintainers handle breaking changes and who you would call when a build fails.
Building missing threat coverage in-house
ART covers evasion, poisoning, extraction and inference attacks in one library, while Foolbox concentrates on creating adversarial examples and CleverHans on attacks, defenses and benchmarking. If you need poisoning or membership inference tests, your security engineers may have to write and validate those attacks themselves on Foolbox or CleverHans. That work is one-time per attack but carries ongoing review. Map your required threat categories against each library's documented scope before you estimate the build.
Reproducing results across frameworks
Foolbox supports PyTorch, TensorFlow and JAX, so a team that trains in one framework and deploys in another can compare attacks across them. Results still differ when the same attack runs on different backends because of numeric precision, model export and preprocessing. Your data scientists spend time confirming that a finding holds across the stack you actually run. Ask whether your evaluation harness will run the same attack configuration on each framework and record the seeds and versions used.
Compute for attack runs
Adversarial attacks are iterative. Gradient-based methods run many forward and backward passes per sample, and black-box attacks issue many queries to the model. Compute is usually the largest recurring line item once you test at production data volumes, and it is paid for by whoever hosts the GPUs or cloud instances. Reduce it by testing on representative samples first, scoping attacks to the threat model, and caching results. Ask for a per-run compute estimate on your own model before committing to a schedule.
Cost of switching libraries later
Attack scripts, evaluation datasets and reports are tied to the library that generated them. Moving from a single-framework library to ART, or back, means rewriting attack configurations and rerunning baselines so results stay comparable over time. This cost is one-time but grows with the number of test suites you have built. Keep attack definitions in your own code and store results in a neutral format, so a future move does not force a full rebuild of your evaluation history.
How adversarial tests fit into a model release process
An ART-based test starts by wrapping the trained model in an estimator for its framework. ART's README lists TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy as supported model types, so a fraud model built on gradient-boosted trees and a vision model built in PyTorch can sit in the same test harness. Once wrapped, the attack classes run against a held-out evaluation set that the training pipeline never touched, and the output is a set of robustness metrics per threat category. Those metrics then feed a release gate: a pipeline step that fails the build when accuracy under attack drops below a threshold the model owner has agreed in writing. The thresholds are a governance decision, not a library default, so decide who signs them before the first run.
The same harness serves two audiences. The README presents ART as a toolkit for both red teams, who use the attacks to probe a model, and blue teams, who use the defenses and detectors to harden it. In practice the red team runs evasion and extraction probes against a staging copy of the model, and the blue team reruns the same suite after each retraining to confirm nothing regressed. Confirm with your own model owners which artifacts the gate must store, such as attack parameters, seeds and the exact estimator version, so an auditor or incident responder can reproduce a result months later.
Keeping this wiring portable matters if the estate is mixed. A single-framework library may be simpler for one model family, but each new framework then needs its own attack code, its own gate script and its own evidence format. ART's shared estimator interface keeps one gate across frameworks, which lowers the cost of onboarding the next model. Foolbox and CleverHans can still fit a PyTorch or TensorFlow pipeline well, so the question for your team is how many frameworks the gate must cover in the next two years and who maintains the glue code when a dependency changes.
What implementing each library involves
The sequence below turns a shortlist into a repeatable adversarial test that runs against your model, gates releases and leaves evidence an assessor can review. The same order applies to all three libraries, but the work at the attack and threat-coverage steps differs.
Confirm framework and data types
Start by recording the model framework (for example, TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost or GPy), the input type (image, tabular, audio, video or text) and the task (classification, object detection, speech recognition or generation). ART's README lists the frameworks and data types it supports, so check your stack against that list first. Foolbox targets PyTorch, TensorFlow and JAX networks, and CleverHans is organized around neural-network attack and defense workflows, so tabular tree models may need extra adapter work in those two. The step is done when the owning team signs off a one-page statement of framework, data type and task.
Select threat categories in scope
Decide which threats the test must cover: evasion, poisoning, extraction or inference. ART documents all four categories in its README. Foolbox and CleverHans are centered on adversarial examples, which maps most directly to evasion, so poisoning, extraction and inference tests usually mean writing additional code or pulling in other tools. Ask the security owner which threats the risk assessment names for this model, and record the decision with the reason. The step is done when the scope list is approved and each excluded category has a documented rationale.
Pin library versions and dependencies
Pin exact versions of the library, the model framework and their transitive dependencies in a lock file or container image. ART's latest release in the supplied facts is 1.20.1, published July 7, 2025. Foolbox's latest is 3.3.4 (March 4, 2024), and CleverHans' latest is 4.0.0 (July 24, 2021), with its last code push in April 2024. Pinning matters most for the two older projects, because a framework upgrade can break their APIs and no new release may arrive to fix it. The step is done when the lock file is committed and a clean install reproduces the same test results.
Build the model wrapper
Wrap the production model so the attack library can query it, read gradients where the attack needs them and return outputs in the expected shape. ART provides estimator classes for its supported frameworks, which is the main reason its wrapper work is usually lighter for mixed model types. Foolbox and CleverHans expect a model object that matches their framework conventions, so expect more adapter code for non-neural models or models served behind an API. Confirm the current wrapper API in each repository before estimating effort. The step is done when a clean-input prediction through the wrapper matches the production model on a held-out sample.
Define pass or fail thresholds
Agree the metrics and limits that decide whether the model passes, such as accuracy under attack at a fixed perturbation budget, and the attack parameters used to measure it. Write the thresholds into the test configuration so a reviewer can see them without reading code. The security owner and the model owner should both sign off, because a threshold set too loosely hides weakness and one set too tightly blocks every release. The step is done when the threshold file is versioned with the model and the risk owner has approved it in writing.
Add runs to continuous integration
Place the adversarial test in the pipeline that trains or validates the model, with a fixed random seed and a stored sample set so results are repeatable. Decide whether a failed threshold blocks promotion or only raises a ticket, and record that choice in the pipeline definition. Attacks on large models can be slow, so consider running the full suite on a schedule and a reduced suite on each commit. The step is done when a deliberate regression in a test branch fails the pipeline as designed.
Document evidence for assessors
Store each run's configuration, library versions, dataset identifiers, threshold file, output metrics and the reviewer's decision together. These records are the evidence that auditors and internal risk teams will ask for, and they show how the test was scoped. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any report or attestation. The step is done when one run can be reproduced from its stored records and the evidence package is filed with the model's documentation.
Confirm the current API before estimating effort
Before committing to a schedule, read the current documentation and examples in each repository and run a small spike on a representative model. The spike shows how much wrapper, threat and reporting work each library requires for your case. Ask the vendor or the maintainers any open question about supported attacks, and record the answer. Effort estimates should come from this spike, not from the library comparison alone. The step is done when the team has a written estimate tied to the spike results.
Questions buyers ask before choosing or hiring help
How much effort does ART need in a classic machine learning pipeline?
ART is a Python library, so the effort sits in wiring it into your existing training and evaluation code rather than in installing a separate platform. You wrap a trained model in an ART estimator for the framework it uses, such as PyTorch, TensorFlow, Keras, scikit-learn or XGBoost, then run attacks and defenses against held-out data. The main cost drivers are the number of model types, the data modalities (images, tables, audio or video) and how many attack categories you test. ART covers evasion, poisoning, extraction and inference, so each category you add is additional test design work. Ask for a written list of models, frameworks and threat categories in scope before estimating effort.
Can our team run ART without outside help?
A team with strong Python skills and someone who understands the model under test can usually run ART on its own for a single model. Outside help tends to add value when you need repeatable test suites in CI, robustness results tied to a defined threat model, or several models across different frameworks. Be realistic about the skills involved: adversarial testing requires knowing which attack suits which model and how to interpret a drop in accuracy under perturbation. Confirm internally who will own the test code, the threat assumptions and the decision to retrain after a finding.
What does an ART engagement need from our side?
Plan for three inputs: access to the model or a representative copy of it, a data sample that reflects production inputs, and named owners. Model access determines which white-box or black-box attacks you can run. Data matters because robustness results depend on the input distribution, and poisoning or inference tests need training data or a faithful substitute. Owners should include a model owner who can act on findings, a security reviewer who sets the threat model, and someone who controls the compute environment. Ask any implementation partner to list these inputs in the written scope before work starts.
How long does an ART implementation take, and what drives it?
No duration is fixed for this work, and the timeline depends on scope. The main drivers are the number of models, their frameworks, the modalities involved, whether attacks run in a controlled environment or inside an existing CI pipeline, and how quickly the data and model access are approved. Defenses such as adversarial training can add cycles because each candidate needs re-evaluation. An engagement with a written scope, deliverables and milestones lets you compare quotes on equal terms. Ask for the timeline as dependencies on your side, not only as the vendor's working days.
Should we migrate from CleverHans or Foolbox to ART?
Migration makes sense when maintenance is the deciding factor. ART's latest release, 1.20.1, was published on July 7, 2025, and it is hosted by the Linux Foundation AI & Data Foundation. Foolbox's latest release, 3.3.4, dates from March 4, 2024, and CleverHans' latest release, 4.0.0, from July 24, 2021, with its last code push in April 2024. If your existing scripts run without problems and only use attacks you already rely on, you can keep them while you plan the move. If you need threat coverage beyond evasion, such as poisoning or inference tests, plan a migration and budget for re-validating results against the new library.
Does ART cover large language models?
ART is built for classic machine learning models, and its documented coverage is evasion, poisoning, extraction and inference against frameworks such as PyTorch, TensorFlow and scikit-learn. It is not the tool to start from for testing LLM applications. For LLM prompt injection, jailbreaks and guardrail testing, see the AI security tooling pillar on this site, which covers that tool category separately. Before you assume coverage in either direction, confirm the exact model types and attack methods you need against the current ART documentation.
Is ART or any adversarial library certified for compliance?
No. Libraries such as ART, Foolbox and CleverHans are testing tools, and none of them produces a certificate or attestation. Compliance frameworks that touch AI risk are assessed by independent auditors or regulators, who decide whether your evidence is sufficient. DeployOpen can prepare systems, controls and evidence for a compliance assessment, including test results from tools like ART, but it does not issue audit reports, attestations or certifications. Ask your assessor early which evidence they expect from adversarial testing before you design the test program.
How DeployOpen helps choose and implement adversarial testing
DeployOpen plans, implements, integrates and documents self-hosted open-source platforms inside the client's own infrastructure, so the choice between ART, Foolbox and CleverHans is made against your models, data and environment. We start from the maintenance and framework facts on this page and turn the shortlist into a written scope before any build begins.
How we scope
- Confirm which model families, data types and threat categories the testing must cover before comparing libraries.
- Record the frameworks your models run on, such as TensorFlow, PyTorch or scikit-learn, and check each candidate against them.
- Agree scope, deliverables and timelines in writing; most engagements are fixed-scope, and the quote follows from that scope.
How we build
- Install and configure the chosen library inside your cloud account, data center or air-gapped network.
- Write attack and evaluation scripts against your own models, with results logged for each run.
- Integrate the test suite into your existing pipelines and review access controls over the test environment and its data.
How we hand over
- Document the setup, pinned versions, dependencies and the steps to rerun each test suite.
- Provide evidence packages that show what was tested and how results were produced, for use in your own assessments.
- Walk your engineers through the test code so they can extend attacks and modify evaluation criteria after handover.
How we support
- Plan upgrades when a library release changes APIs, supported frameworks or maintenance status.
- Offer ongoing support, upgrades and continuous testing as a retainer if you want the suite maintained over time.
- Confirm with your team which maintenance window, ownership and review cadence will apply after the engagement ends.
