Adversarial Robustness Toolbox vs Counterfit: which fits your model testing
Choose the Adversarial Robustness Toolbox (ART) if your team must assess classic machine learning models such as fraud classifiers, vision models or tabular scoring models and wants code it can version, review and run in a pipeline. ART 1.20.1, published on July 7, 2025, is an MIT-licensed Python library hosted by the Linux Foundation AI & Data Foundation, and it covers evasion, poisoning, extraction and inference attacks across TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy models. Counterfit may still suit a team that wants a command-line workflow with one tool over ART, TextAttack and AugLy. Counterfit is also MIT-licensed, but its latest release, v1.1.0, is dated November 14, 2022, so it needs a deliberate maintenance review before it becomes part of a standing assessment process.
The decision turns on four questions. First, maintenance expectations: if your assessment program must rely on upstream fixes for dependencies and new model frameworks, check the release and commit history of each project on the day you decide, since the last Counterfit code push was in July 2025 and its last release is more than three years older than ART's. Second, data types: Counterfit's built-in targets span text, image and tabular data, and an attack must support the target's data type to run, so list your model inputs before you pick a tool. Third, team skills: ART asks engineers to write attack code against their own model wrappers, while Counterfit gives analysts a menu. Fourth, whether you want a menu or code, which determines how well each option fits repeatable regression testing.
Before you commit, confirm with each project which Python versions and operating systems your environment needs. The Counterfit README lists Python 3.8 or higher on Linux or Windows through WSL, and you should verify that against your build images and your model serving stack. For ART, check the supported framework versions for your specific model type in the ART documentation and the ART releases page rather than assuming parity with your current environment. Record the outcome of this check in your assessment plan, because a tool that cannot load your production model format will not produce evidence you can use.
Which platform fits which testing scenario
This table maps common testing scenarios to the platform that fits them better, and the condition that makes each choice hold. Read it by matching your model type, team skills and delivery pipeline to a row, then checking the condition before committing.
| Scenario | Better fit | Condition that makes it hold |
|---|---|---|
| Fraud or tabular model needing evasion and inference tests | ART | ART lists tables among its supported data types and covers evasion and inference; confirm the model framework (scikit-learn, XGBoost, LightGBM or CatBoost) is on its supported list. |
| Image classifier needing evasion and poisoning tests | ART | ART covers images and all four threat categories, and Counterfit's image targets only accept attacks that support image data, so ART gives you broader control over the attack. |
| Text classifier (NLP model) | Neither alone; see the TextAttack comparison | ART's coverage is multi-framework across images, tables, audio and video. For Hugging Face text models, review the sibling TextAttack comparison before choosing. |
| Repeatable CLI for non-developers running assessments | Counterfit | Counterfit's command-line layer is designed to run assessments without writing attack code, provided your target and attack data types match and you accept a last release in 2022. |
| Tests shipped into CI on a maintained codebase | ART | Scripting ART directly lets you pin versions and wire tests into your pipeline; ART's latest release is 1.20.1 from July 7, 2025, so check its release cadence before adopting. |
| Long-term maintenance and security fixes | ART | Counterfit's latest release is v1.1.0 from November 14, 2022, while ART is hosted by the Linux Foundation AI & Data Foundation; confirm current maintenance status yourself. |
How Counterfit sits on top of ART and what that means for operations
Counterfit is a command-line layer that brings existing adversarial frameworks under one tool, and ART is one of the frameworks it credits. The four cards below cover what ART supports directly, how Counterfit's runtime requirements narrow your environment, and what it means to depend on an upstream layer that has not shipped a release since 2022.
Model frameworks ART supports directly
ART works with TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy models, so a fraud model built on gradient boosting or a vision model in PyTorch can be attacked from ordinary Python code. Teams that script ART decide the model wrapper, data pipeline and attack parameters themselves. Before you commit, confirm that your exact model library version is covered by the ART release you plan to pin, because framework support is the first thing that determines whether Counterfit's wrapper even reaches your model.
Threat categories and what they test
ART covers evasion, poisoning, extraction and inference, which means it can probe a deployed classifier for crafted inputs, tampered training data, model stealing and leakage of training membership. Counterfit exposes attacks through its own command set, so coverage depends on which attacks its built-in targets accept, and an attack must support the target's data type to run at all. Map each threat category to a named test in your assessment plan, then check whether that test is available through the CLI or only in the ART code you would write yourself.
Counterfit runtime requirements
Counterfit requires Python 3.8 or higher and runs on Linux or Windows through WSL, which rules out a native Windows install for many teams. Its built-in targets cover text, image and tabular data, so a tabular fraud model may need a custom target definition. Check the Python version pinned by your platform team and the dependency set against your build images before you schedule an assessment, because an interpreter mismatch is the most likely reason a first run fails.
Operating on an upstream layer without releases
Counterfit's latest GitHub release is v1.1.0 from November 14, 2022, although its code was last pushed in July 2025. By contrast, ART 1.20.1 was published on July 7, 2025. Depending on Counterfit means your assessment workflow inherits the release cadence of a tool with no recent release, and any compatibility break with newer ART or Python versions falls to your team to patch. Ask whether the Counterfit repository has an owner who will merge fixes, and plan to keep a direct ART script as the fallback path.
What Counterfit's CLI adds for assessments and where that value ends
The main value of Counterfit is a single entry point for assessments. Counterfit brings existing adversarial frameworks under one command-line tool, crediting ART, TextAttack and AugLy as the frameworks it builds on, so an assessor can pick a target, pick an attack and run it with one consistent invocation instead of learning each library's API. Its built-in targets cover text, image and tabular data, which matters for mixed-skill assessment teams where a security engineer who does not write Python can still run and record structured tests. For that group, the CLI is the practical advantage, and it matters most when the assessment is a repeated, scoped exercise rather than a research project.
The limits sit in the same design. Counterfit enforces that an attack must support the target's data type, so a text attack will not run against an image or tabular target, and teams testing fraud or credit models must confirm which attack and target pairs their model type can use before planning a run. The wrapper also adds a dependency on a project whose latest GitHub release, v1.1.0, dates from November 14, 2022, although its last code push was in July 2025. Before relying on it, confirm current status with Microsoft's repository, test the exact Counterfit version against the ART version you have installed, and decide whether the CLI convenience is worth pinning two projects instead of one. Counterfit releases is the place to check the release history.
Capabilities, maintenance and fit side by side
This table compares the two Microsoft-linked options on license, release history, hosting, threat coverage, data types, framework support and Python requirements. Read the maintenance rows first, because they decide whether either tool is a safe base for a new assessment program, then use the coverage rows to judge what each one can test.
| Capability | Adversarial Robustness Toolbox (ART) | Counterfit |
|---|---|---|
| License | MIT | MIT |
| Latest release | ART 1.20.1, published July 7, 2025 | v1.1.0, published November 14, 2022 |
| Last code push | Confirm the latest commit date in the ART repository before you plan work | July 2025, per the GitHub repository |
| Hosting and ownership | Hosted by the Linux Foundation AI & Data Foundation | Microsoft project that automates attacks from ART, TextAttack and AugLy |
| Threat categories | Evasion, poisoning, extraction and inference | Inherits what its built-in attack frameworks provide, so confirm the attack list per target in the repository |
| Data types | Images, tables, audio and video | Built-in targets span text, image and tabular data |
| Model framework support | TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy | Confirm per target and backend in the Counterfit README |
| Python requirement | Confirm the supported Python versions in the ART installation docs | Python 3.8 or higher on Linux or Windows through WSL |
| Community signal (October 2026) | About 6.3 thousand GitHub stars | Not supplied, so check the repository star count and open issues directly |
What each license allows and what no edition sells
ART is an MIT-licensed Python library for machine learning security, hosted by the Linux Foundation AI & Data Foundation. MIT allows you to use, copy, modify, merge, publish, distribute, sublicense and sell copies of the software, provided the copyright notice and permission notice travel with it. For an internal assessment, your team can run ART in its own environment, change its attack scripts and keep the results without paying the project. For a client engagement, ART can be part of the test harness you deliver, as long as the notices remain intact.
- Internal assessments: Your staff can install and run ART from the ART repository under the MIT terms.
- Modified builds: You can patch attack implementations for a specific model family and keep those changes internal, subject to the notice requirement.
- Client deliverables: Scripts that call ART can be handed over with the original notices. Ask counsel whether the engagement contract adds its own terms on third-party code.
- Foundation governance: Hosting under LF AI & Data means project decisions follow a foundation process. Confirm current governance and release documents before you plan around a particular release.
- Commercial support: Confirm with the maintainers or the foundation whether any paid support exists and what it covers.
Check with counsel before you ship ART inside client deliverables, and record its license notice in your third-party software inventory.
Counterfit is an MIT-licensed command-line automation layer from Microsoft for assessing the security of machine learning systems. It brings existing adversarial frameworks under one tool and credits ART, TextAttack and AugLy as the frameworks it builds on. Its MIT grant matches ART's, so internal use and modification are allowed on the same basis. What differs is what you inherit: a wrapper carries its own terms plus the terms of the libraries beneath it, and the project's maintenance pattern determines how much of the upkeep falls to your team.
- Internal assessments: Teams can run the Counterfit CLI for assessments under MIT terms, provided the notice is kept. The project's repository lists the supported targets and setup steps.
- Dependency licenses: TextAttack and AugLy are separate projects with their own licenses. Check each one before you package Counterfit with them.
- Release age: The latest GitHub release is v1.1.0 from November 14, 2022, and the last code push was in July 2025. If you depend on Counterfit, plan for your own patching.
- Forks: MIT allows you to fork the project and maintain a private copy, which moves security fixes and Python version support onto your team.
- Hosted offerings: Confirm with Microsoft whether any managed service exists and what terms would apply to it.
Before you embed Counterfit in a client deliverable, record the license of every dependency it pulls in and confirm who will patch it.
Both projects are MIT-licensed, so the license does not separate them. For most teams the license question is settled in one step: MIT allows internal use, modification and redistribution with the notice retained, and it attaches no license fee to any of those activities. Neither project offers a paid edition, so there is no feature tier to buy and no license line to budget. What you pay for is the work around the tools: the people who script them, the compute that runs them and any support contract you choose to buy from someone else.
- Staff time: Setting up scripts, model access and test data is usually the largest one-time cost. Recurring time goes to updating tests when models change.
- Compute: Attack runs against large models or many samples consume infrastructure. Costs rise with model size, sample count and how often the tests run in CI.
- Support contracts: Any paid support is a separate agreement with a named vendor. Ask for response scope in writing.
- Assessment services: Fixed-scope engagements are quoted from written scope, so ask for deliverables, timelines and exclusions before comparing totals.
- Quote checklist: Ask who maintains the code, who is paid for what, which items recur and which inputs drive the total.
Ask every vendor and partner for these items in writing before you compare totals.
Some points cannot be settled from public project materials alone. Put them on the procurement checklist before a decision, and get written answers wherever the answer affects cost, liability or client deliverables. Answers from maintainers, counsel and your own vendor records carry more weight than an assumption about how an open-source license usually works.
- Maintainer support: Ask whether the ART maintainers or the foundation offer commercial support, and what that support covers.
- Counsel review: Ask counsel to confirm that MIT terms and your client contracts fit together for both internal and delivered tooling.
- Dependency inventory: Record ART, Counterfit and every transitive dependency with its license and version.
- Hosted terms: Confirm whether a hosted offering exists for either project and read its terms before any client data touches it.
- Ownership of fixes: Agree who patches a fork or vendored copy, and how quickly that patching must happen.
Keep the written answers with the assessment file so the license basis remains traceable later.
What drives the running cost of ART and Counterfit
Licence fees are zero for both tools, so the running cost comes from people, compute, keeping versions aligned and deciding who owns the work. The four cards below separate what you pay once to build a test harness from what recurs each time you rerun assessments, upgrade frameworks or refresh evidence.
Staff skills and ownership
ART is a Python library, so the people who write attacks need working knowledge of PyTorch, TensorFlow, scikit-learn or the model framework under test. Counterfit's CLI lowers the entry barrier for an assessor who only runs known attacks, but it still needs a Python environment, and someone must understand the attack configuration to interpret results. Check whether your team can own the harness after the first assessment, or whether that knowledge will sit with one engineer.
Compute and test environments
Attacks on large models, especially those that run many iterations per sample, consume substantial compute, and that cost scales with the size of the test set and the number of attacks you run. Both tools expect a Linux host or Windows through WSL, so plan a dedicated, isolated environment with GPU access where the model needs it. Ask who pays for that capacity and whether it is one-time provisioning or an ongoing cloud bill.
Upgrade and compatibility effort
ART releases track the frameworks it supports, so a framework upgrade can change attack behavior and force you to retest previously passing models. Pin ART and framework versions in a lockfile and record them in every assessment report. If Counterfit stops tracking ART, your team may need to maintain a fork or a thin wrapper, which becomes a recurring engineering cost. Confirm with the vendor or your own tests how each dependency change affects your results.
Internal ownership or external support
Internal ownership keeps the harness close to the models but concentrates risk in a few people and requires time for reruns, upgrades and evidence refresh. A fixed-scope implementation, agreed in writing before work starts, covers the initial harness and first assessment, while a retainer covers ongoing upgrades and reruns. Ask any provider to state which deliverables are one-time and which recur, and what triggers a new quote.
Moving from Counterfit to ART, and what transfers
This sequence turns an existing Counterfit assessment into scripted ART tests that run against the same models and produce comparable evidence. It moves one model at a time, so each result can be checked against the Counterfit baseline before the next model changes hands.
Inventory existing Counterfit targets and attacks
The assessment owner lists every Counterfit target in use, the model file or endpoint behind it, the data type it handles (text, image or tabular), and each attack and its settings from past runs. Export the configuration and any saved result files from the Counterfit session so nothing lives only in the CLI history. The step is done when every target has one row in a shared inventory with its model framework, data type and attack parameters, and the owner has signed off that the list matches the assessment scope.
Map each target to an ART estimator
For each inventory row, the security engineer identifies the model framework (for example, scikit-learn, PyTorch, Keras or XGBoost) and selects the matching ART estimator wrapper, which exposes the model to ART's attack classes. Where a model cannot be wrapped directly, the team records that gap and decides whether to wrap a reimplementation or drop the target from the scope. The step closes with a mapping table that pairs each Counterfit target with one ART estimator and a note on any model that needs manual handling.
Port attack invocations to ART attack classes
The engineer rewrites each Counterfit attack invocation as a direct call to the corresponding ART attack class, keeping the same epsilon values, iteration counts and norms that the Counterfit configuration used. Attack names rarely map one to one, so the team checks the ART documentation for each class's parameters rather than copying option names. The step is done when each attack runs as a Python script that loads one estimator and writes its adversarial output to a known path, with the parameter values documented beside the code.
Confirm data type handling for each model
Counterfit blocked attacks that did not match a target's data type, so the team now has to make that check explicit. For each model, the engineer confirms the input shape, value range, preprocessing and label format that the ART estimator expects, and runs one clean prediction and one adversarial example through the wrapper. The step is done when each script asserts the input and output shapes on a small sample and fails loudly on mismatch, which replaces the implicit guard Counterfit provided.
Rerun baselines and compare outputs
The team reruns the ART scripts on the same models and the same samples used in the last Counterfit assessment, then compares accuracy under attack, perturbation sizes and success rates against the saved Counterfit results. Differences are expected where attack implementations or random seeds differ, so each one is explained in writing before the scripts are accepted. The step closes when every target has a reconciliation note that either matches the baseline within the explained tolerance or records why it does not.
Version control the scripts and wire them into CI
The scripts, the estimator mapping, pinned ART and framework versions, and the sample data references go into a repository with review requirements. A CI job runs the scripts on a schedule or on each model change and stores the output as build artifacts, so results are reproducible and traceable to a commit. The step is done when a clean run from a fresh environment reproduces the baseline reconciliation, and the team can show who approved each change to the test suite.
Retire Counterfit in stages
Migrate one model at a time, keep Counterfit available for read-only reference until each replacement has passed its reconciliation, and avoid a single cutover date for the whole inventory. Counterfit's latest release is v1.1.0 from November 14, 2022, so the team should record that version and its known gaps in the migration log before it is retired. The step is done when every target runs under ART in CI, the migration log is archived with the evidence trail, and the Counterfit environment is removed or frozen with a documented owner.
What implementing ART or Counterfit involves and where projects run long
Implementing ART is mostly engineering work done by your own team or a partner. The first task is selecting attacks for each threat category the library covers: evasion, poisoning, extraction and inference. Each attack must be matched to the model type and data modality, then the model must be wrapped in the correct ART estimator for its framework, such as a PyTorch or scikit-learn wrapper. Compute sizing follows from the attack chosen, since some evasion and extraction runs are far heavier than others, so test a representative model before committing to a schedule. The lasting work is writing repeatable test scripts, keeping them in version control and running them from CI with pinned dependencies and recorded results. Confirm the framework versions your models use against the ART release you pin, because ART 1.20.1 was published on July 7, 2025, and you must verify compatibility with your own stack.
Counterfit front-loads different work. Your team installs the Python environment (the README lists Python 3.8 or higher on Linux or Windows through WSL), configures each target, and then checks that every attack supports the target's data type, since an attack must match text, image or tabular inputs to run. Because Counterfit's latest release is v1.1.0 from November 14, 2022, expect to pin and test its dependencies yourself and to confirm that the wrapped frameworks still load on your versions. Projects run long for four recurring reasons. First, ownership of the test suite is often unclear, so nobody maintains it after the first assessment. Second, framework version drift breaks models silently between runs. Third, data type mismatches surface late, when a target is already scheduled for review. Fourth, evidence is kept as console output rather than as dated, reviewable records that risk teams and auditors accept. A fixed-scope engagement with written deliverables, covering named owners, pinned versions, target inventories and evidence formats, is one way to bound this work before it grows.
Questions teams ask before choosing ART or Counterfit
Is Counterfit still maintained after its 2022 release?
Activity and maintenance are different things, and Counterfit's signals are mixed. Its latest GitHub release is v1.1.0 from November 14, 2022, with no tagged release since then, but its last code push was in July 2025. Treat that as an open maintenance question rather than a settled one. Before you adopt it, review the commit history and open issues, confirm that your build hosts meet its stated requirement of Python 3.8 or higher on Linux or Windows through WSL, and decide whether you can carry patches yourself. For a new assessment program that needs a recent release line, ART is the stronger baseline, as the next answers explain. Counterfit repository
Can ART be used without writing any code?
No, ART is a Python library, so running it means writing code. Its README presents it as a toolkit for developers and researchers, and assessments are built by passing your model to ART's attack and defense classes in a script. A team without Python skills needs an engineer or an outside consultant to write and maintain the harness, although non-coders can review the results. Counterfit is closer to a command-line workflow, since it wraps several adversarial frameworks behind one tool. Its built-in targets cover text, image and tabular data, and each attack must support the target's data type. Confirm which model types and attacks you need before choosing either tool.
How much effort does switching from Counterfit to ART take?
The effort depends on how much of your Counterfit workflow is tied to its own commands, target definitions and output handling. Counterfit builds on ART, so attacks that it runs through ART map back to the same library, which reduces rework for those cases. The main drivers are the number of target models, any custom target code written for Counterfit, how many people parse Counterfit's output, and the Python environment you pin. One-time costs include rewriting harnesses and revalidating results, because outputs may differ. Recurring costs include maintaining scripts and upgrading dependencies, including ART itself, whose latest release is 1.20.1 from July 7, 2025. Ask any implementation partner for a written inventory of targets and attacks before agreeing a scope.
What skills and infrastructure does a team need to run ART in CI?
A team needs three things in parallel. First, Python engineers who can write attack scripts, pin dependency versions and keep tests reproducible across runs. Second, machine learning expertise that matches your model stack, since ART supports TensorFlow, Keras, PyTorch, scikit-learn, XGBoost, LightGBM, CatBoost and GPy models. Third, CI infrastructure with enough compute for the attacks you choose, which may mean CPU or GPU runners, controlled access to model artifacts and secrets, and a place to store reports. Confirm the runtime requirements for your chosen ART version against your build images before committing to a pipeline design.
Are ART and Counterfit free to use commercially under MIT?
Both are MIT licensed, so you can use them in commercial work without a license fee. The license does not remove your other costs, which include engineering time, compute for test runs, storage for reports and any consultant work. The MIT terms require you to keep the copyright and license notice, so check the LICENSE file in the exact version you install. Also review the licenses of the model frameworks and datasets you test against, because those are separate from ART and Counterfit. ART is hosted by the Linux Foundation AI & Data Foundation, while Counterfit comes from Microsoft. Your legal team should confirm how these terms apply to your internal distribution practices.
Is a consultant worth hiring for a first ML security assessment?
It depends on your in-house skills, your timeline and how much evidence you need to produce. A consultant is most useful when you lack engineers who know adversarial machine learning or when you need a defensible record of what was tested. DeployOpen plans, implements and runs security testing, including AI systems, and most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. A written scope should name the models and model types in scope, the ART version pinned, the threat categories (evasion, poisoning, extraction and inference), the test environment, data handling, deliverables such as reports and reproducible scripts, CI handover, retesting and exclusions. DeployOpen does not publish a price list, and a quote follows from that scope.
Do text models need a different tool than ART?
Usually, yes. ART covers images, tables, audio and video, and its supported frameworks are classic machine learning libraries, so it is not the natural first choice for language models. For a text classifier, such as a fraud or content model built on NLP architectures, the sibling comparison of ART with TextAttack covers which tool fits. Counterfit includes text targets among its built-in options, but you should confirm that your model type and attacks are supported before relying on it. For LLM applications facing prompt injection or jailbreak risk, tools covered in the AI security tooling pillar are designed for generative systems. Test your actual model and workflow before deciding.
How DeployOpen helps teams choose, implement or migrate
DeployOpen plans, implements and documents self-hosted open-source platforms inside the client's own infrastructure, including security testing for AI systems. For Adversarial Robustness Toolbox vs Counterfit, the work starts with a written scope that decides whether ART, Counterfit or a script-based ART setup fits the models under test.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope lists which model types, data modalities and attack categories are in play, such as evasion or poisoning.
- We confirm which Python and framework versions your environment runs before choosing between ART and Counterfit.
- Quotes follow from the written scope, so ask for each line item to be itemized by test, build and handover work.
How we build
- We script ART directly against your models in your own environment, including your cloud accounts, data centers or air-gapped networks.
- Where a Counterfit assessment is in scope, we confirm the target data type matches the attack before any run.
- Test harnesses are versioned with your code so that assessment results can be rerun after model changes.
- We check release currency and the maintenance status of each dependency at build time and record it in the plan.
How we hand over
- You receive the test scripts, configuration, run instructions and a record of the attack categories covered.
- Documentation explains how to rerun assessments, read results and interpret findings for your model owners.
- We prepare the evidence and controls that compliance assessors may ask for, without issuing any audit report or certification.
- Handover includes a short list of what to confirm with the library maintainers and your own security team.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- A retainer can cover moving assessment scripts forward when ART or Counterfit releases change APIs or dependencies.
- Periodic re-testing can follow model retraining, so new versions are checked against the same attack categories.
- Support terms, response expectations and included work are set out in the retainer scope agreed in writing.
