MEASURE 2.3 deployment-like testing: NIST AI RMF evidence

What MEASURE 2.3 requires and why teams struggle with it

If you are the compliance lead or engineer who must show that an AI system performs as expected in realistic conditions, MEASURE 2.3 is the outcome that asks for that proof. It expects the organization to measure performance or assurance criteria, qualitatively or quantitatively, and to demonstrate them under conditions similar to the deployment setting, with the measures written down. The risk it targets is familiar: a model that scores well on a curated test set in the lab and then degrades once real users, real data pipelines and real latency constraints arrive. The NIST AI RMF is voluntary and has no certification, so nobody grants a badge for meeting MEASURE 2.3. The work is preparing evidence that an internal reviewer, a customer's due diligence team or an independent assessor can inspect and test.

Teams struggle for three reasons. First, the test environment rarely matches production: the evaluation may use clean inputs, a different hardware profile, a smaller retrieval corpus or no concurrent load, and nobody records where those differences lie. Second, the metrics are often undefined or chosen after the results are known. The Playbook points to measures such as false positive and false negative rates for validity, and training time and prediction latency for efficiency, but a team still has to decide which of these matter for its use case and what threshold counts as acceptable. Third, there is no written record tying a test run to a configuration, a dataset version and a deployment decision. Confirm with your assessor which artifacts they will request, and build the record before the next release rather than reconstructing it afterward.

Which systems, data and teams fall under deployment-like testing

Scoping decides how much MEASURE 2.3 testing is needed and who must sign off on it. The cards below cover the decisions that change the work, and the ones to document before any test design starts, because later results only mean something against a fixed system, setting and owner.

Systems and versions in scope

Name each AI system, model version, prompt or configuration, and retrieval index that the deployment will use. A test run on one model version does not demonstrate performance for the next, so each change to the model, prompts, tools or data sources should be tied to a version record. Ask whether a change triggers a fresh deployment-like test or only a regression check, and write that rule down before testing starts.

Built in-house or bought

The testing duty stays with the organization that deploys the system, even when a vendor built it. For purchased AI, the deploying organization needs enough vendor documentation, test access or contractual evidence to run deployment-like tests in its own setting. Confirm early which test artifacts the vendor will supply, which the buyer must generate, and who pays for any environment the vendor must provide.

Deployment settings that count

Record the cloud region, expected user volume, data freshness, connected integrations and human review steps that the system will face in production. A test environment that differs on any of these points should be listed as a known gap in the documentation. Illustrative example: a model tested on a static data extract may behave differently once it reads live records through an API connector.

Human review and workflow steps

Decide whether a person reviews, edits or can override the output before it reaches a user or customer, and include that step in the test flow. Measuring the model alone can overstate performance when a reviewer corrects errors in practice. Document which steps are automated, which are human, and how the reviewer's accuracy is recorded.

Generative AI risks in scope

Generative AI systems also map to Generative AI Profile actions for MEASURE 2.3 in NIST AI 600-1. Name the GAI risks the system could produce, such as confabulation, data privacy, information security and harmful content, so that each one has a test method and a documented result. Scope should not be limited to accuracy when the system generates text, code or images.

Owners of design, sign-off and go or no-go

Assign one team to design the deployment-like tests, one to run and document them, and one accountable party to give the go or no-go input. Testing and evaluation staff, deployment owners and risk leaders often hold different views of acceptable performance. Write down who approves the test plan, who accepts residual differences between test and production, and how disagreements are escalated.

How an assessor tests MEASURE 2.3 and what they ask for

This table walks through the steps an independent assessor is likely to take on MEASURE 2.3, the evidence each step asks for, and the findings that come up most often. Read it as a pre-assessment checklist: each row shows what the assessor will look for and where documentation usually falls short. The independent auditor or assessor reviews the evidence and issues any report, not DeployOpen.

Assessor step	Evidence requested	Common findings
Read the documented deployment conditions	Written description of the deployment setting, user population, data sources, integrations and operating limits, linked to the MAP outputs	Deployment conditions are described only in general terms or live in a different document from the test plan
Check that tests ran before deployment under those conditions	Dated test plans, test run logs and a go or no-go record showing testing preceded release	Tests were run on a development environment with different settings, or the dates cannot be reconciled with the release
Review the validity metrics chosen	Metric definitions with false positive and false negative rates, and the rationale for each metric's relevance to the mapped risks	Metrics are named but not defined, or only accuracy is reported for a system with costly error types
Review the efficiency metrics chosen	Measured training time, prediction latency and throughput under expected load, with the test configuration recorded	Latency is measured on idle hardware, so results do not reflect production load
Confirm testing in non-optimized conditions	Test scenarios that include noisy, incomplete, out-of-range or adversarial inputs, with results for each scenario	Tests use curated or cleaned data only, and no evidence shows how the system behaves on messy production inputs
Check metric thresholds	Acceptance thresholds for each metric, the owner who approved them and the date they were set	Metrics are reported without thresholds, so results cannot show pass or fail against risk tolerance
Compare test and deployment environments	Side-by-side record of hardware, software versions, data pipelines, access patterns and user roles in each environment	Environment differences are not recorded, or the comparison is made from memory rather than configuration files
Confirm differences were recorded and assessed	Documented list of known differences, the risk each one introduces and any compensating measure or follow-up test	Differences are noted in a tester's notes but not carried into the risk register or the go or no-go decision
Trace results to the go or no-go decision	Decision record that cites the deployment-like test results, the accountable approver and any conditions attached to release	Test results exist but no decision references them, or the approver is not identified

How to implement deployment-like testing step by step

This sequence turns MEASURE 2.3 into repeatable work: it ties test conditions to the deployment context, measures performance under both expected and harsher conditions, and leaves a record that explains every gap between the test and production environments. The Playbook suggestions behind these steps are optional, so adapt each one to the system and its risk.

Define deployment conditions from the mapped context
Start from the context recorded under MAP 1 and MAP 3: intended users, input sources, operating hours, hardware and network limits, human oversight points and the decisions the output feeds. The AI team and the business owner write these conditions down together, and the system owner signs them off. The step is complete when each condition names a concrete source, such as a data feed, user group or infrastructure component, and nobody can point to a deployment condition that is missing from the test plan.
Select performance and assurance metrics
Choose the metrics that show whether the system meets its intended purpose and its risk tolerance. Validity metrics such as false positive and false negative rates belong here, along with efficiency metrics such as training time and prediction latency. Each metric gets a definition, a calculation method, an acceptance threshold agreed before testing, and a named owner. The output is a metric register that TEVV and the accountable owner approve, which also lets you explain later why a metric was chosen.
Build or select a test set that reflects expected inputs
Assemble test data that matches the input distribution described in the deployment conditions, including the common cases and the edge cases the MAP work identified. Record where the data came from, how it was split from training data, and which known gaps remain. Privacy and data handling rules apply to this set as well. The step closes when the test set has a documented composition and the TEVV team confirms that it covers the conditions defined in the first step.
Run pre-deployment tests under expected and non-optimized conditions
Run the system on the test set under the conditions the deployment will expect, then run it again under non-optimized conditions, such as constrained compute, degraded or noisy inputs, slower network paths, and inputs that were not tuned for the model. The second run shows whether performance holds once the system is no longer in its best-case setup. Keep the run configuration, software versions and hardware details with each result so that a reviewer can repeat the test. The output is a set of raw results for both conditions.
Record validity and efficiency metrics for each run
Log the results from each run against the metric register, including validity measures and efficiency measures such as training time and prediction latency. Compare results with the thresholds set in advance, and flag any metric that falls outside them. Store the logs in a form that an independent assessor can inspect, and keep the tool versions used to produce them. The step is complete when every metric has a recorded value for each condition, and every failed threshold has an entry in the findings log.
Document every test and deployment difference with the decision
List each difference between the test environment and the deployment environment, such as data volume, latency profile, user behavior, hardware or third-party dependencies, and state the likely effect on the measured results. For each difference, record the decision that followed: accept the risk, add a control, narrow the scope, repeat the test or hold the deployment. Name the accountable person who made the decision and link it to the go or no-go record maintained under MANAGE 1. This record is the evidence that the measures were both documented and acted on.

Evidence that passes MEASURE 2.3 and how to keep it current

MEASURE 2.3 is demonstrated through a small set of linked records, not a single report. Each artifact below answers one question an assessor or internal reviewer will ask about deployment-like testing, and each needs a named owner and a trigger for refresh. The organization owns and signs these records; an independent auditor, if one is engaged, tests them.

Deployment condition specification

This document describes the conditions the system will actually face: user populations, input types and volumes, latency expectations, hardware, data sources and operator workflow. It is the yardstick for every later test, so MEASURE 2.3 assessors will check whether test conditions were drawn from it. Refresh it whenever the intended use or user population changes, and have the accountable business owner and the system owner sign it. Check that it matches the MAP 1 context record.

Metric definitions and thresholds

Each performance or assurance measure needs a written definition, the calculation method, the data it uses and the threshold that counts as acceptable. Validity metrics such as false positive and false negative rates should sit alongside efficiency metrics such as prediction latency and training time. Review thresholds at each re-test, and have the TEVV lead and the risk owner approve changes in writing so that no threshold moves quietly after a result is known.

Pre-deployment test reports with configuration

Each report should record the test date, the exact model or system version, the dataset version, the software and infrastructure configuration, and the results against each threshold. Include tests run in non-optimized conditions, since the Playbook suggests them. The TEVV engineer prepares the report and the system owner signs it. Reports are regenerated for each release candidate, not reused from an earlier version.

Environment comparison document

This record lists every difference between the test environment and production, such as hardware class, network path, data freshness, concurrency and integrations with other tools. Each difference should carry an estimate of its effect on the results and a plan to close it or monitor it. Assessors look for this document to judge whether deployment-like testing was honest. Update it whenever infrastructure changes, and have the platform engineering lead sign it.

Results summary with accepted risks

A one-page summary states which thresholds passed, which failed, and which residual gaps the accountable owner accepted before go-live. Each accepted risk needs a named approver, a reason and a review date. This is the artifact that connects MEASURE 2.3 to the MANAGE 1 go or no-go decision. The accountable executive or risk owner signs it, and it is reviewed at least at each scheduled assessment.

Change log and re-test record

The change log lists each change to the model, training or fine-tuning data, prompts, retrieval sources, dependencies or infrastructure, and links each one to a re-test result. Without it, earlier test evidence no longer describes the running system. The change log is maintained continuously by the engineering team and reviewed by the TEVV owner before each release. For generative AI, keep the benchmark baseline and the empirically validated capability evidence described in NIST AI 600-1 alongside it, since those actions cover selecting and augmenting models.

Which open-source tools can produce MEASURE 2.3 evidence

MEASURE 2.3 asks for documented performance under conditions that resemble deployment. Two open-source tools can generate much of the raw record behind that claim: one for adversarial testing of language models, and one for tracing requests in a running application. Neither decides whether a system is acceptable, so the thresholds and conclusions remain the organization's responsibility.

garak for adversarial probe records

garak (Apache-2.0) probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. It writes a JSONL report covering every attempt and evaluation, plus a hit log of attempts that found a vulnerability. Run it against the model configuration you plan to deploy, with the same system prompt, tools and guardrails, so the report reflects deployment-like conditions. Check which probes ran, which generators were used, and whether the record can be reproduced later.

Langfuse traces for request-level evidence

Langfuse (MIT) captures each trace of a request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. These traces let you compare latency, token use and step behavior in a staging environment against production, which is the environment comparison MEASURE 2.3 asks you to document. Confirm that your pipeline sends traces from both environments with the same metadata fields, and decide how inputs containing personal data are masked before storage.

NIST Dioptra as a testing environment

The Generative AI Profile names NIST Dioptra as an example of a purpose-built testing environment for evaluating claimed capabilities with empirically validated methods. Use it as a reference point when you plan a controlled test rig, not as a finished evidence tool. Before adopting it, confirm its license, supported model types, hardware requirements and the maintenance effort your team would carry, since these determine fit for your environment.

Tools produce data, not conclusions

Neither tool states whether a system meets its performance criteria. Your team must set the acceptance thresholds for false positive and false negative rates, latency and other metrics before testing, record why each threshold was chosen, and interpret the results against the intended use. Assessors will look for that decision record alongside the raw logs. Treat tool output as input to a documented judgment by accountable staff, reviewed by someone independent of the build team where possible.

What meeting MEASURE 2.3 costs in money and effort

This table lists the cost components of demonstrating MEASURE 2.3, who is paid for each, whether it is one-time or recurring, and what drives it up or down. No supplied figures exist for these costs, so use the drivers to scope a quote and the final column to shape what you ask for.

Cost component	One-time or recurring	What drives it and how to keep it down
Engineering time to define deployment-like conditions	One-time, revisited on change	Number of deployment environments and user populations. Keep it down by writing one conditions document per system that names data sources, latency targets and hardware, and reusing it across releases.
Building test sets that mirror production inputs	One-time build, recurring refresh	Volume and variety of real input types and the effort to label them. Keep it down by sampling from logged production traffic where privacy rules allow, and by documenting each set's provenance for MEASURE 2.1.
Tooling and licenses	Recurring where commercial	Commercial evaluation and observability licenses scale with seats or trace volume. Self-hosted open-source tools such as garak (Apache-2.0) and Langfuse (MIT) remove license spend, but hosting, patching and staff time remain and recur.
Compute for repeated test runs	Recurring	Model size, number of test cases and how often runs repeat after each change. Keep it down by running a targeted subset on each change and the full suite on release or scheduled intervals.
Evidence upkeep after model or data changes	Recurring	Frequency of model updates, prompt changes, retraining and data pipeline changes. Keep it down by tying test reruns to a change log so only affected measures are repeated and documented.
Documenting differences between test and deployment environments	Recurring per release	Count of environment differences such as non-optimized hardware, reduced data or shared infrastructure. Keep it down by maintaining a single comparison table that engineering updates with each deployment.
External testing or independent review	Per engagement	Scope of the review, number of systems, and whether the reviewer runs tests or only inspects records. Ask for the reviewer's deliverables in writing, and confirm that an independent party, not the implementation partner, issues any assurance opinion.
Auditor or assessor sampling time	Per assessment cycle	Number of systems and releases sampled and how easily evidence can be retrieved. Keep it down by storing test runs, configurations and results in one retrievable location with dates and version identifiers.

Who owns each part of deployment-like testing

Your organization owns MEASURE 2.3. The AI RMF is voluntary and has no certification, so the accountability for choosing what counts as deployment-like, which metrics matter and whether the system is fit to go live stays with you. Your AI, risk and business owners set the performance and assurance criteria before testing starts, because a test cannot show a system meets a bar nobody defined.

Deployment conditions: Define the expected users, data, traffic, integrations and failure modes that the test must reproduce, and record them in the AI system documentation.
Metric decisions: Select validity measures such as false positive and false negative rates, plus efficiency measures such as prediction latency, and document why each one matters for the intended use.
Gap documentation: Record every difference between the test and production environments and state the risk each difference leaves open.
Sign-off and go or no-go: Name the accountable person who reviews results and decides whether deployment proceeds, and keep that decision in the record.
Outsourcing scope: You can hand off environment builds and tooling work, but the criteria and the final decision remain internal.

Before you engage anyone, confirm that a named owner can explain every criterion and every documented gap without reading the test report aloud.

Where organizations fall short on deployment-like performance evidence

MEASURE 2.3 asks for evidence that a system performs under conditions resembling its deployment, with the measures documented. The gaps below recur across programs, and each one weakens that evidence in a different way. Each card names the gap and the step that closes it.

Tests run only on clean data

Teams often evaluate on curated, cleaned datasets that never contain the missing fields, malformed inputs, duplicates or unusual phrasing found in live traffic. The result looks strong but says little about the conditions the system will face. Under MEASURE 2.3, the test set should be drawn from or compared against the operational input stream. Add a test set built from sampled production-like inputs, including messy records, and document how it was sampled.

No documented test-to-production gap

Test environments commonly differ from production in hardware, latency, data volume, user population, integrations and access rights, and those differences are rarely written down. An assessor will ask how the test results carry over to the deployment setting. The Playbook suggests documenting differences between test and deployment environments. Keep a written comparison table that lists each difference and states its expected effect on the measured results.

Metrics without agreed thresholds

A metric reported without an acceptance threshold cannot show pass or fail. False positive and false negative rates, prediction latency and training time may all be recorded, yet nobody has agreed what value is acceptable for the use case. Thresholds should be set before testing, tied to the risk tolerance from MAP, and approved by an accountable owner. Record each threshold and its approver before the test run starts.

Results not linked to go or no-go

Pre-deployment results are frequently filed as a report without showing who reviewed them or what decision followed. MEASURE 2.3 evidence is stronger when the results feed the deployment decision that MANAGE 1 describes. An assessor will look for a dated record linking the measured performance to a named decision maker. Attach the test results to a signed go or no-go record, including any accepted residual risk.

No re-test after changes

A model that passed deployment-like testing can degrade after a model version change, a retrained weight, a new retrieval source or a shift in input data. Without a re-test trigger, the original evidence no longer describes the running system. Define the changes that require re-testing, such as a new model version, a changed prompt or a new data source, and rerun the same test set. Tie each re-test to a documented change request.

Generative AI tested without a baseline

Generative AI systems are often evaluated against claimed capabilities without first recording how the underlying model performs on established benchmark suites. NIST AI 600-1 suggests considering baseline performance on benchmark suites when selecting a model to fine-tune or augment with retrieval, and evaluating claimed capabilities with empirically validated methods. Record the base model's benchmark results before any fine-tuning or retrieval changes, so later gains and losses can be measured against it.

What buyers ask before bringing in help with MEASURE 2.3

How much effort does MEASURE 2.3 work take, and what drives it?

The effort depends mostly on how many AI systems are in scope, how many distinct environments they run in, and how mature the existing test sets and metrics are. MEASURE 2.3 asks you to show performance under conditions similar to deployment and to document the measures, so the work splits into designing the test conditions, running the tests, recording validity and efficiency metrics, and writing down where test and production environments differ. Systems with no existing test harness need more setup than systems that already log evaluation results. Ask any provider for a written scope that lists each system, each environment, the metrics to be recorded and the documents to be delivered, because that list is what a quote should price.

How long does a MEASURE 2.3 engagement take?

The timeline is set mainly by the number of systems and environments in scope, followed by how quickly your team can provide access, data and sign-off. A single model deployed in one environment is a different job from several models that each run in staging, production-like and edge settings. Engagements are usually fixed-scope, so the start, deliverables and completion dates are agreed in writing before work begins. Confirm with any provider which milestones depend on your staff, such as access approvals, test data release and review cycles, because those dependencies often move the schedule more than the testing itself.

What does the work need from our own staff?

Your staff own the decisions and the knowledge of the system, so they need to be involved throughout. Typical needs include an accountable owner who can approve test conditions, engineers who can grant access to model endpoints and logging, and a domain lead who can say what a realistic deployment looks like and which failures matter most. Someone also needs to review the documented measures and confirm they match how the system is actually used. Ask for a list of named roles and expected hours in the written scope so the commitment is visible before the engagement starts.

Can our team do MEASURE 2.3 without outside help?

Yes, an internal team can do the work, and the NIST AI RMF is voluntary, so no outside party is required. The main questions are whether your team has testing experience for the specific system type, time to build repeatable test conditions, and a way to keep the documentation current after each change. Teams new to structured TEVV often underestimate the documentation and the work of reproducing non-optimized conditions. The NIST AI RMF Playbook offers suggested actions and documentation prompts for MEASURE 2.3, which can serve as a self-assessment starting point, though it is not a checklist. Outside help is most useful for independent testing, for building the environment, or for preparing evidence for a customer or regulator review.

Does NIST AI RMF have a certificate for MEASURE 2.3?

No. NIST does not offer a certificate, accreditation or conformity assessment for the AI RMF, and NIST does not validate or endorse any organization's use of it. Customers and regulators may ask you to show how you address MEASURE 2.3, but the evidence is a set of documented tests, metrics and decisions rather than a certificate. If a vendor offers an AI RMF certificate, ask who issued it and under what program, because NIST's published materials describe no such program. Confirm the exact evidence a customer expects before you commit to a format.

Who signs off on the MEASURE 2.3 evidence?

Your organization owns its controls and signs off on its own AI risk decisions, including whether a system's measured performance is acceptable for deployment. An implementation partner prepares the test conditions, runs the tests, records metrics and drafts the documentation. Neither DeployOpen nor any other partner signs the evidence as an attestation. If an external assurance step is required by a customer or regulator, an independent auditor or assessor reviews it and issues any report, and that role is separate from the people who built the tests. Ask in writing which party approves each document.

Does DeployOpen issue an assessment report or certificate for MEASURE 2.3?

No. DeployOpen plans, implements, tests and documents self-hosted open-source platforms inside your own infrastructure, and it prepares systems, controls and evidence for compliance assessments. It does not issue audit reports, attestations or certifications. For MEASURE 2.3, the deliverables are the test design, the recorded metrics, the documented differences between test and deployment environments, and supporting records that your organization reviews and owns. Where a report is needed, an independent auditor or assessor produces it. The scope document should name each deliverable and the party responsible for approving it.

How DeployOpen prepares MEASURE 2.3 evidence

DeployOpen helps organizations prepare the test setups, measurement records and deployment-comparison documentation that MEASURE 2.3 asks for. The organization owns its AI system and its risk decisions, and an independent assessor is the party that tests and reports on conformance.

How we scope

Agree in writing which AI system, components and deployment settings the MEASURE 2.3 work covers
Record the expected production conditions, user populations and data sources the testing must mirror
Confirm which performance and assurance metrics the organization has chosen to document and why
Note which parts of the work the organization keeps in-house and which DeployOpen will prepare

How we build

Set up pre-deployment test environments that replicate expected production inputs, integrations and load
Run non-optimized test passes alongside tuned runs so gaps between the two are visible in the records
Capture validity measures such as false positive and false negative rates, plus efficiency measures like latency
Where generative AI is in scope, run garak probes and log traces in Langfuse to keep attempt-level evidence

How we hand over

Deliver a written record of test conditions, metrics, tool versions and results for each measured characteristic
Document every known difference between the test environment and the deployment environment, with its likely effect
Provide raw outputs such as garak JSONL reports and Langfuse trace exports for the assessor to inspect
Hand over the repository, configuration and runbooks so the organization can rerun the tests itself

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts
Ongoing support, upgrades and continuous testing are available as a retainer for measurements that must be repeated
Re-run deployment-like tests when the system, its data or its environment changes, as the organization decides
Confirm with your assessor or auditor which records they will accept before the scope is finalized