NIST Dioptra: open-source AI risk testing platform explained

Is NIST Dioptra the right way to run AI RMF Measure testing?

Dioptra is a good fit for a team that must show repeatable AI security and robustness testing under the AI RMF, provided the team is ready to host and operate it. Dioptra is a NIST-developed, open-source, self-hosted test platform that supports the Measure function, and it produces test evidence, not a certificate. The AI RMF is voluntary, and NIST does not certify or endorse any organization's use of it, so no Dioptra run, report or badge can stand in for an independent assessment. Its value is that experiments are designed, run and tracked in a way others can rerun and compare.

Fit depends on three things. The first is the AI systems in scope: Dioptra's documentation describes a modular, microservice-based environment for AI workflows, so confirm that your model types, data pipelines and inference endpoints can be wired into it. The second is the attack classes you need to test. NIST's documentation says the platform helps with the combinatorial explosion of testing possibilities across attacks, but it does not follow that every attack class you care about is covered. Ask the platform team which attacks and metrics are included, and which you would have to build yourself. The third is who operates the platform. Setup involves building Docker containers, starting services and choosing options such as Compose overrides and GPU-enabled workers, so someone must own the infrastructure, upgrades and access control.

Reproducibility also has limits. AI 100-2e2025 notes that most mitigations against adversarial machine learning are empirical and can be defeated by new attacker techniques, so a passing Dioptra run is one input to Measure, not proof of security. Plan to define which metrics matter most for your mapped risks, record the configuration of each run, and schedule re-testing when models or threats change. Before committing, read the Dioptra documentation and the usnistgov/dioptra repository, then confirm the license terms and supported version with NIST's materials and your own legal review.

Where Dioptra fits in AI RMF Measure

The AI RMF Measure function asks organizations to choose metrics, test trustworthy characteristics, track risks after deployment and check that measurement works in context. Dioptra supports the testing and tracking work inside that cycle. The organization decides which AI RMF subcategories apply to its systems, and Dioptra covers only part of the measurement work those subcategories require.

Selecting metrics for mapped risks

Measure 1 starts with choosing metrics for the most significant risks found during Map. Dioptra does not choose those metrics for you. Its value here is that each candidate metric can be written down as part of a repeatable experiment, so the team can see which risks a chosen metric actually exercises. Check that the metric definitions, datasets and attack assumptions are recorded before any run starts, since Dioptra can only track what the team defines.

Testing security and robustness

Measure 2 asks for evaluation of each trustworthiness characteristic, and secure and resilient is one of seven in AI 100-1. Dioptra is built for assessing these characteristics in AI systems, and NIST notes it helps with the large number of combinations across attack types. Confirm which attack classes and models your environment supports before you commit to a test plan, because coverage depends on what your team configures.

Tracking test results over time

Measure 3 expects risks to be tracked after deployment, including risks that current methods cannot yet measure. Dioptra records experiments so that workflows can be reproduced and reused when a model, dataset or attack changes. Ask who owns the stored run records, how long they are kept, and how they link to the risk register and go or no-go decisions made under Manage 1.

Checking measurement in context

Measure 4 asks whether measurement works in the setting where the system is used. Dioptra can run tests against the components you deploy, but the team must judge whether those tests reflect real users, data and operating conditions. Plan expert review of results and document any gap between the test environment and production. This review is an organizational task that the platform cannot perform.

Running self-hosted workflows

Dioptra runs inside your own infrastructure, built from Docker containers and started as services, with options such as GPU-enabled workers. Someone on the team must own the hosting, access control and updates, since those controls become part of the evidence an assessor reviews. Confirm the hardware, network boundaries and access model with your platform team before scheduling tests.

CC BY 4.0 licensing and what the license does not settle

Dioptra is available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license, to the extent NIST holds copyright in it, according to the usnistgov/dioptra repository. That license governs NIST's copyright only, not every component you will run. Before you deploy, ask your legal team to read the license terms against your intended use, including any internal redistribution, modification and publication of test results. Then run a separate dependency review. A Dioptra deployment is built from Docker containers and a stack of third-party libraries, and each of those packages carries its own license, which may impose different obligations. Record each license in your software bill of materials so that the license position is documented rather than assumed.

The license also does not settle whether your tests prove anything. AI 100-2e2025 states that there are no information-theoretic security proofs for widely used machine learning algorithms, so most mitigations against adversarial ML attacks are empirical and may themselves be defeated by new attacker techniques. A clean Dioptra run therefore shows that a defined attack set did not succeed against a specific model, configuration and data at a specific time. It does not show that the system is secure. Write that boundary into the test plan, state the attack families, model versions and datasets covered, and record which attacks were not tested.

For the Measure function, this means a Dioptra result should feed a decision rather than close one. Decide in advance which results will trigger a MANAGE response, such as withholding a go decision, adding a control or re-running tests after a model change, and document who signs off on residual risk. Confirm with your legal team and with the platform documentation at NIST Dioptra documentation which terms apply to the version you deploy.

How self-hosting Dioptra works from first build to first experiment

This sequence takes a team from an empty host to a first tracked experiment on Dioptra 1.2.1. Each step ends with a check the team can verify before moving on, so problems surface while they are still cheap to fix.

Record hosting, access and storage decisions first
Before anyone runs a build, the platform owner decides where Dioptra will run, such as which cloud account, data center or isolated network will host it, and who in the organization may reach it. Access control covers user accounts for the web interface, the REST API and the Python client, plus who may change configuration. Storage needs a decision too: where experiment records and test artifacts are kept, how long they stay and who can read them. Confirm each decision against the Dioptra documentation for version 1.2.1. The output is a short written record approved by both the security and platform teams.
Prepare the host and Docker environment
The platform engineering team prepares the host that will run Dioptra, installs Docker and confirms the host meets the prerequisites listed in the 1.2.1 documentation. If the tests will need accelerators, decide now whether the host will have GPU drivers and a container runtime that can expose them to containers. Record the Docker version and host configuration in the change log. The check is simple: Docker runs containers on the host, the deployment account has only the access it needs, and the host sits inside the network segment chosen in the first step.
Build the Dioptra Docker containers
Using the source from the official repository, the platform engineer builds the Docker containers that make up Dioptra. Pin the source to the 1.2.1 release so the build matches the documentation the team is following. Keep the build logs, since they show which base images and dependencies went into each container. This step is complete when every container image builds without errors and the team has recorded the image names and tags for later audit.
Start the services and verify they respond
The engineer starts the Dioptra services using the container tooling described in the documentation and checks that each one reports healthy. Verification means reaching the web interface and sending a basic request to the REST API from an approved admin account, not from an open network path. A failing service usually points to a port, environment variable or dependency problem, so fix it here before adding anything else. The output is a running stack that answers both interfaces, with startup logs showing that each service started cleanly.
Apply Docker Compose overrides for your environment
Docker Compose overrides let the team change the default stack without editing the upstream files. Use them for settings that differ in your environment, such as network bindings, storage mounts for results and secrets handling, and confirm which settings the documentation allows you to override. Keep the override file in version control with a named owner. Review the merged configuration before restarting, so the team can see the effective settings. The output is a versioned override file, a reviewed merged configuration and a restarted stack that still passes the checks from the previous step.
Configure GPU-enabled workers where tests need them
Only some workloads need GPUs, so this step applies when robustness or adversarial tests on large models are too slow on CPU. The platform team configures a GPU-enabled worker following the option described in the documentation, then confirms that the host driver and container runtime expose the GPU to the container. If capacity is shared, restrict which users can schedule GPU jobs. The check is a test job that reports GPU use from inside the worker. Document the hardware model and driver version, because reproducing results later depends on them.
Run the first tracked experiment
A data scientist or security tester runs the first experiment through the web interface, the REST API or the Python client, whichever the team has approved. Dioptra tracks experiments, so the team should name the experiment, record its inputs and parameters, and confirm the run appears in the tracking records. Repeat the same experiment once with identical inputs and compare the outputs to test reproducibility. The output is a tracked run, its stored results in the location chosen at the start, and a note on any differences between the two runs.

Who runs Dioptra inside the organization and who partners on it

The organization owns everything that Dioptra measures. It decides which AI system is in scope, which trustworthiness characteristics matter for that system, what counts as an acceptable result, and whether the system goes live. Dioptra produces test results, but those results only mean something when the organization interprets them and acts on them. The AI RMF is voluntary and NIST does not validate any organization's use of it, so the accountability for the test program sits entirely inside the organization.

System ownership: Name an accountable owner for each AI system under test and for the decision to deploy, pause or retire it.
Risk decisions: Set the risk tolerances and the metrics that Measure results will be judged against, in line with the Map and Govern outcomes.
Test program: Define the attack and robustness scenarios, the datasets, the models under test and the review cadence.
Operation of the platform: Decide whether internal staff run Dioptra day to day or whether a partner supports it under a written scope.
Retained records: Keep the experiment definitions, run histories and sign-offs that show how results led to decisions.

Before engaging anyone, confirm internally who signs off on the test results and who receives the residual risk documentation.

What a team needs before Dioptra tests produce usable evidence

Dioptra runs experiments, but it cannot decide what matters for your organization or prove that results are trustworthy. The six items below are the decisions and records that have to exist before test output can support a Measure function claim, and each one has a simple check your team can run before the first experiment.

A defined inventory of AI systems in scope

Dioptra tests only make sense against a named system, model version and deployment context. Confirm that every system in scope appears in a current AI inventory with its owner, purpose, model source and the environment where it runs. For each test campaign, record which inventory entry it targets. If two teams can point to different versions of the same model under one name, the results cannot be compared or reproduced.

Mapped risks that the tests must address

Measure results matter only when they answer a risk identified earlier in the Map function. Before configuring attacks or evaluations, list the mapped risks for each system and link each test to one of them. Check that every high-priority risk has at least one test and that every test traces back to a risk. Tests with no mapped risk usually produce numbers nobody can act on.

Named metrics and acceptance criteria

Decide in advance which metric each test reports and what result counts as acceptable, degraded or failed. Write the threshold, the dataset it applies to and the person who signs off on exceptions. Confirm with the risk owner that the thresholds reflect the organization's stated risk tolerance. Adjusting thresholds after seeing results weakens the evidence, so record the criteria with a date before the first run.

Compute sized for the models under test

Some attack and evaluation workloads need GPU-enabled workers, and Dioptra's documentation describes GPU-enabled workers as one deployment option. Confirm which models require GPUs, how many concurrent experiments the team expects, and where the workers will run inside your own cloud account, data center or isolated network. Ask the platform owner to verify that container images and model weights can reach the workers under your network policy.

Access control and logging for experiments

Dioptra users design, run and track experiments through a REST API, a web interface or a Python client, so each of those paths needs an identity, a role and an audit trail. Confirm who can start attacks against which models, who can change datasets or metric definitions, and where experiment records are stored. Test that a completed experiment can be traced to the user, code version and inputs that produced it.

A plan for documenting results against subcategories

Decide how each test result will be recorded against the AI RMF subcategory it supports, such as the MEASURE 2 subcategories for trustworthiness evaluation. Define the evidence package for each result: the experiment identifier, configuration, dataset version, metric output, reviewer and decision. Ask the assessment lead to confirm that this format matches what internal audit or an external reviewer will request, because NIST does not certify a documentation format.

Questions teams ask before self-hosting Dioptra

Is Dioptra a certification or compliance tool?

No. Dioptra is an open-source software test platform that NIST developed for assessing the trustworthy characteristics of AI. It supports the Measure function of the NIST AI Risk Management Framework, but the AI RMF itself has no certification program, and NIST does not validate or endorse any organization's use of it. Dioptra produces test runs and records that can serve as evidence for your own risk program. Only your own governance decides whether that evidence is sufficient, and any external assurance comes from an independent assessor, not from the tool or from a partner that helps you deploy it.

What skills and staff time does a Dioptra deployment need?

The work spans three skill areas: container and infrastructure operations, machine learning engineering, and security testing. Effort rises with how many of these your team already covers. Container work includes building Docker images and running services, while GPU-enabled workers add driver and scheduling considerations. Machine learning engineers must define the models, datasets and attack configurations under test. Security testers decide which threats matter for your system. Ask any partner to name who does each part and how much of your staff time the plan assumes. No duration figure has been published for this, so treat any quoted timeline as something to verify against the written scope.

Can our team self-host Dioptra without outside help?

Often yes, if you have engineers comfortable with Docker and an ML team that can supply test targets. Dioptra's documentation describes setup as building its Docker containers and starting the services, with options such as Docker Compose overrides and GPU-enabled workers. The harder questions tend to be operational: keeping test runs reproducible across model and data changes, managing access for the REST API, web interface and Python client, and maintaining the deployment as the project changes. Teams that lack these skills in-house can scope a fixed engagement with a partner that installs, documents and hands over the environment, while keeping ownership of the tests and results.

What drives the effort of a Dioptra deployment up or down?

Several factors move the effort most:

The number of AI systems and attack types under test, since NIST notes Dioptra helps manage the combinatorial explosion of testing possibilities across attacks.
Whether you need GPU-enabled workers and how your existing cluster or cloud accounts are configured.
How many people access the platform and whether they need role separation and audit trails.
The maturity of your existing test data, model artifacts and metrics, which determines how much preparation comes before the first run.
Integration with your CI/CD, evidence store and ticketing systems.

Ask for each driver to be listed as a separate line in a quote, so you can see which items are one-time setup and which recur as the platform and models change.

How does Dioptra relate to the NIST AI RMF Playbook and Generative AI Profile?

Dioptra is a test tool and does not replace either document. The Playbook offers voluntary suggested actions, documentation prompts and references for each AI RMF subcategory, and NIST says it is neither a checklist nor a set of steps. The Generative AI Profile in NIST AI 600-1 adds suggested actions for generative AI risks, each tied to an AI RMF subcategory. Dioptra can help produce measurement results that feed MEASURE subcategories, but you must choose which suggested actions apply to your context. Confirm with your team which Playbook entries and Profile actions your test plan addresses, and document the rest as out of scope.

What should we ask a partner to put in a written scope for Dioptra?

Ask for the scope in writing before work starts, including:

The systems, models and data sets to be tested, and the trustworthy characteristics each test addresses.
The deployment target, such as your cloud accounts, data center or air-gapped network, and whether GPU workers are included.
Who owns the deployed environment, test definitions, results and any access credentials after handover.
Documentation delivered, including setup, run procedures and how results are tracked.
Which parts remain your responsibility, such as threat selection, risk decisions and independent assessment.

DeployOpen works on fixed-scope engagements agreed in writing, and it does not issue audit reports or certifications. Ask any partner to confirm the same boundaries.

Does Dioptra cover all AI security testing needs on its own?

No. Dioptra provides a modular environment for reproducible, trackable and reusable AI workflows, but it does not decide which attacks matter or which mitigations work. NIST's AI 100-2e2025 report notes that most mitigations against adversarial machine learning are empirical and can be defeated by new attacker techniques, so test results describe a point in time. Plan for threat modeling, red teaming, and ongoing measurement alongside the platform. Confirm with your security team which coverage gaps remain and who is responsible for closing them.

What to check when choosing an implementation partner for Dioptra

Dioptra is self-hosted, so most of the work happens inside your own infrastructure and against your own test data. The checks below cover who can run that work safely, how the scope is fixed, how evidence comes out the other end, and where responsibility sits once the testing is done.

Self-hosted platform experience

Dioptra is set up by building its Docker containers and starting the services, with options such as Docker Compose overrides and GPU-enabled workers. Ask the partner to show prior work deploying self-hosted open-source platforms in your cloud account, data center or air-gapped network. Ask who manages the GPU workers, how image builds are pinned, and how upgrades are tested before they reach your environment.

AI system security testing capability

Dioptra runs the experiments, but someone must design attacks, choose metrics and interpret results. Ask whether the partner tests AI systems and agents as a regular practice, not only web applications. Ask how they handle the combinatorial spread of attack and defense combinations, and how they document why a particular set of tests was chosen. Mitigations against adversarial ML are largely empirical, so ask how the partner reports residual uncertainty.

Written fixed scope and deliverables

Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts. For Dioptra, the scope should name the AI systems under test, the attack and robustness categories, the number of experiment workflows, the compute environment and the handover items. Ask what changes the scope, who approves a change, and what happens to experiments that are still running at the end date.

Evidence handover and documentation

Dioptra tracks experiments so they can be reproduced later, which only helps if the partner hands over the artifacts that make reproduction possible. Ask for the experiment definitions, dataset and model versions, container image identifiers, random seeds where used, configuration files and the raw results. Confirm the format, where the files are stored, and how long your team keeps access after the engagement ends.

Clear division of responsibility

The organization owns its AI risk decisions and controls. The implementation partner prepares systems, test results and documentation that feed the Measure function. Only an independent assessor or certification body issues reports or certificates, and the AI RMF itself has no certification. Ask the partner to state this split in the contract, and to confirm in writing that it will not describe its work as an audit or attestation.

How DeployOpen works on this

DeployOpen deploys NIST Dioptra inside your own infrastructure, from scoping through to documented test runs your team can repeat. We prepare the platform and the test evidence, while your organization owns its AI risk decisions and any independent assessment is performed by an outside assessor.

How we scope

Agree the written scope, deliverables and timeline before any work starts, since most engagements are fixed-scope.
Identify which AI systems, model types and trustworthiness characteristics the Dioptra test workflows will cover.
Confirm the hosting environment, including cloud accounts, data centers or air-gapped networks, and GPU needs for workers.

How we build

Build and start the Dioptra Docker containers in your environment, with Compose overrides and GPU-enabled workers where the scope requires them.
Configure access for the REST API, web interface and Python client so your teams can design, run and track experiments.
Set up reusable test workflows for attack and robustness experiments, so results can be rerun under the same conditions.

How we hand over

Document the deployment, configuration, workflows and run procedures so your staff can operate the platform without us.
Package test records and run tracking outputs as evidence mapped to the Measure subcategories you selected.
Note measurement limits and state that mitigations against adversarial ML attacks are empirical, so results must be read in that context.

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after handover, scoped in writing.
Plan upgrades of Dioptra and its containers in your change process, with a rollback path agreed before work begins.
Ask us to confirm current licensing and version details against the official repository at the start of each engagement.

NIST Dioptra: open-source test platform for AI risk measurement