MEASURE 2.1 Test sets, metrics and tools documentation

What MEASURE 2.1 requires for AI test documentation

MEASURE 2.1 requires that the test sets, metrics and details of the tools used during testing, evaluation, verification and validation (TEVV) are documented. When a customer, contract or internal review asks how an AI system was tested, the answer has to be a record someone else can reread, rerun and challenge. Without that record, a result is a claim with no way to check it. A team may know a model scored well on a benchmark, but if nobody can say which data split, which metric definition, which tool version and which thresholds produced the number, the result cannot be reproduced after staff change or the system is updated. The NIST AI RMF is voluntary and has no certification, so no outside body will sign off that this documentation exists. The reader's own customer or regulator will judge it, and they will ask for the records directly.

The NIST AI RMF Playbook, which is a voluntary companion and not a checklist, suggests industry practices such as datasheets for datasets and model cards for MEASURE 2.1. It also suggests that organizations regularly assess whether their documentation tools still work and update them as needed. In practice, a useful first step is to confirm which artifacts your organization already produces for each evaluation: the test set identifier and its provenance, the metric definitions and the code version that computed them, the evaluation tool and its version, and the acceptance thresholds. Ask who owns each artifact and where it is stored. Those answers show whether you can meet the outcome before any template is chosen.

The hard part is that documentation is usually written after the fact, and it drifts. Models are retrained, prompts are edited, datasets are refreshed and evaluation scripts change, while the model card still describes last quarter's version. Illustrative example: a team writes a card at launch, then swaps in a new retrieval index and never updates the test set description, so a later reviewer cannot tell which results apply to the system in production. Preventing this means tying each document to a version of the model, data and code, and making updates part of the release process rather than a separate clean-up task. Treat this as an ongoing cost in staff time, not a one-off exercise.

Which AI systems, datasets and teams fall under MEASURE 2.1 documentation

MEASURE 2.1 documentation only works if the scope is settled first. The cards below cover which models, datasets, tools and people fall inside the record, and who is accountable for each. The Playbook tags this subcategory with the TEVV actor category, so the TEVV team is the natural owner, but product and third-party evaluators also create or consume records.

Models and AI components under test

Start by listing every model and component that TEVV exercises, not only the production endpoint. That includes fine-tuned variants, retrieval components, prompt templates, guardrail filters and any pre-trained model the system depends on. Each item needs a version identifier so a test result can be tied to the exact artifact it measured. Confirm with the TEVV lead that a test run cannot point at an unlisted component.

Training, validation and test datasets

Every dataset that influenced a measurement belongs in scope: training, validation, held-out test and any sample used to calibrate thresholds. Record the source, collection date, version and known gaps for each one. Datasets that were reused across releases need a clear lineage note, because a metric is only meaningful against the data it was computed on.

Red-team and adversarial test sets

Prompts, attack corpora and scenario libraries used for adversarial testing are test sets in their own right, and they are often the most sensitive records a program holds. Document what each set targets, who wrote or sourced it and how it is stored and access-controlled. Treat these sets as confidential where they contain exploit details, and confirm the access rules with the security team.

Evaluation tools, scripts and harnesses

The scoring scripts, evaluation harnesses and scanners that generate results are part of the measurement method. Record the tool name, version, configuration and the metrics each one computes. For open-source scanners such as garak, which is licensed under Apache-2.0 and writes JSONL reports of every attempt, keep the report output alongside the configuration that produced it.

Producers and consumers of TEVV records

The TEVV team produces most records, but product owners consume them to decide releases, and third-party evaluators may generate results under contract. Name each role in the documentation and state who signs off on a test set and who may change a metric. Ask every external evaluator to deliver records in a format your team can reproduce.

Third-party models and external datasets

Pre-trained models, licensed datasets and vendor evaluation reports fall inside MEASURE 2.1 when your results depend on them. Document the provider, the version you received and any limits the provider states about its test data. Where the provider does not share its test sets or metrics, record that gap so reviewers know which claims you can and cannot reproduce.

How an assessor tests MEASURE 2.1

This table walks through how a reviewer tests MEASURE 2.1, from the documents requested to the sampling steps and the findings that come up most often. Read each row as a question the reviewer asks, the evidence that answers it, and where the answer usually breaks down. Because AI RMF 1.0 has no certification, the outcome is an internal or independent review finding, not a NIST determination.

Test step	What the reviewer does	Evidence requested	Common finding
Scope the test documentation	Confirms which AI systems, versions and TEVV cycles fall under MEASURE 2.1 using the AI inventory	Inventory entries, system version list, TEVV schedule	Test records exist for some versions but not the one in production
Review test set documentation	Reads how each test set was sourced, labeled, split and kept separate from training data	Datasheets for datasets, provenance records, data split logs	Undocumented test set provenance or unclear origin of evaluation data
Review metric definitions	Checks that each reported metric has a written definition, formula, threshold and intended use	Metric register, evaluation plan, threshold rationale	Metrics reported in dashboards without definitions or acceptance thresholds
Review tool documentation	Confirms the evaluation tools, libraries and versions used in each test run are recorded	Tool and version list, configuration files, run settings	Tool versions missing, or tools changed between runs without a record
Trace a sample of results	Selects reported results and follows them back to the test run, inputs and output files	Run logs, raw outputs, JSONL or report files, sign-off records	Reported figures cannot be reproduced from the stored run records
Compare documents to the live system	Checks that model cards and test records describe the system currently deployed	Model cards with version and date, change records, release notes	Stale model cards that still describe an earlier model or configuration
Check review and approval	Looks for evidence that TEVV owners reviewed the documentation and that changes were approved	Review records, version history, approver names and dates	Documentation edited without a traceable reviewer or change history
Confirm the review's authority	Identifies whether the reviewer is an independent auditor or the organization's own review function	Engagement letter or internal review charter	Internal reviewers without independence from the team that built the system

How to document test sets, metrics and tools step by step

This sequence turns MEASURE 2.1 into a working documentation routine, so every test set, metric and tool used in TEVV is inventoried, justified, linked to the runs that used it and kept current. Each step names the record it produces so an assessor can trace a result back to its inputs.

Inventory every test set and evaluation tool
The TEVV owner lists every test set, benchmark, evaluation script and tool used for the AI system, recording its version, source, owner and the system components it covers. Third-party tools and open-source scanners go on the same list as internal code, because an assessor will ask what was used, not only what was built. The step is done when every tool in the test pipeline appears in the inventory with a named owner and a pinned version. Confirm the version details with each tool's own release notes or repository, since these change over time.
Define each metric and its acceptance context
For each metric used in TEVV, the team writes what it measures, how it is calculated, the data it is computed on and the threshold or context in which a result counts as acceptable. A metric without an acceptance context cannot show whether a result is good, so the record should also state which mapped risk the metric is meant to detect. Owners include the measurement lead and a reviewer from the risk function. The step closes when each metric has a definition card that a reviewer outside the TEVV team can read and reproduce.
Adopt datasheet and model card templates
The team selects one datasheet template for datasets and one model card template for models, then makes them mandatory for every test set and model version in scope. Industry practice, which the NIST Playbook points to for MEASURE 2.1, covers provenance, collection method, known gaps and intended use for datasets, and intended use, evaluated conditions and limitations for models. Templates should live in a controlled space with version history so changes are visible. The step is done when each dataset and model in the inventory has a completed template with an owner and a date of last review.
Link each test run to its documented inputs
Every test run records the exact test set version, metric definitions, tool versions and model version it used, so a reported result can be traced back to its inputs. Experiment tracking or observability platforms can capture run metadata automatically, which reduces manual copying errors. The team checks a sample of runs by reproducing one result from its recorded inputs. The step is done when a reviewer can start from a reported metric, find the run, and identify the exact test set, tool versions and model version behind it.
Store documentation under version control and access rules
The team keeps inventory entries, metric cards, datasheets and model cards in a wiki or documentation platform that records every change with its author and timestamp, and that lets reviewers compare or roll back versions. Access rules limit who can edit each record, while reviewers keep read access. Open-source wiki tools such as XWiki provide page version history, and BookStack provides role-based permissions at shelf, book, chapter and page level. Confirm that the chosen platform's version history and permission model meet your evidence retention requirements before relying on it. The step closes when edits to TEVV records are traceable and restricted to named roles.
Set a review cadence for documentation tools
The team sets a recurring review in which the TEVV owner checks whether the templates, tools and metric definitions still work for the current system, and updates them where they do not. Reviews also follow major model changes, new test sets, and any incident that exposed a gap in measurement. The review produces a dated log of findings and changes, which is the evidence that the documentation practice is maintained rather than written once. Ask the assessor or internal audit which review frequency they expect, because the cadence is set by your own risk tolerance and not by the framework.

Why datasheets and model cards are the practical starting point

The NIST AI RMF Playbook points to datasheets for datasets and model cards as established measurement documentation practices, and they are the most direct way to meet MEASURE 2.1. A datasheet records where a test set came from, how it was collected, what it covers and what it leaves out. A model card records the model version, intended use, the metrics reported and the conditions under which those metrics were measured. On their own, though, these documents are only descriptions. They become useful evidence when each one is tied to a specific test run, identified by a model version, a dataset version, the metric definitions used and the date of execution, so an assessor can trace a reported result back to the inputs that produced it.

Version control is what makes that tie hold over time. Documentation kept in a wiki or shared drive without history cannot show what the test set looked like when a result was produced, or who changed a metric threshold after the fact. A versioned wiki such as XWiki keeps every change to a page, records what changed, when and by whom, and lets reviewers compare two versions and roll back to an earlier one. Pair that with role-based permissions, for example in BookStack, so that only the TEVV owner can approve edits to test documentation. Confirm with your own platform team how the wiki history is backed up and retained, because the record is only as reliable as its storage.

Documentation also has to be reviewed when the model or data changes, not only at the initial release. Set a rule that any new model version, retrained weights, changed preprocessing or refreshed evaluation dataset triggers a review of the affected datasheet and model card, with a recorded sign-off from the accountable TEVV lead. The Playbook suggests that organizations regularly assess whether their documentation tools still work and update them as needed, so schedule that check as a recurring task. An illustrative example: if a model card still cites a benchmark from before a retraining, an assessor will treat the reported results as unsupported until the card is corrected and the rerun is documented.

Evidence that satisfies MEASURE 2.1 and how to keep it current

MEASURE 2.1 is met by records that let a reviewer reconstruct what was tested, with which data, metrics and tools, and when. The artifacts below connect to one another, so a change to a model or dataset should flow through all of them. Each card names the upkeep that keeps the record accurate between formal reviews.

Dataset datasheets

A datasheet records where each dataset came from, what it contains, how it was collected, known gaps and the uses it was and was not built for. For MEASURE 2.1, it shows which data the test sets were drawn from. Keep one datasheet per dataset version and reissue it whenever collection criteria, labeling or filtering changes. Check that each test set points to the datasheet version it was built from.

Model cards

A model card describes a model's intended use, evaluation results, limitations and the conditions under which the reported metrics hold. Reviewers use it to confirm that published results match the evaluated model. Tie each card to a model release identifier and update it at every retraining, fine-tune or change of prompt or system configuration, with the approver's name and date recorded in the change history.

Test set register with provenance

A test set register lists every evaluation set in use, its owner, version, source datasets, creation date and the splits it excludes from training. Provenance matters because a test set that overlaps training data produces results that cannot be trusted. Review the register at each model release and confirm that no test item has entered a training pipeline, recording the check and its outcome.

Metric definition records

Each metric needs a written definition covering its formula, threshold, aggregation method, the population it applies to and the tool that computes it. Without this, two teams can report the same metric name with different results. Store definitions under version control and require sign-off from the TEVV owner before any threshold changes, so a reviewer can see when and why a target moved.

Tool and version register

The register records each evaluation tool, library, scanner or harness used in TEVV, with its version, license, configuration and the test runs it supported. A result is only reproducible if the tool version is known. Refresh the register when a tool is upgraded, and rerun a fixed reference test after any upgrade to confirm that results remain comparable with earlier runs.

Test run logs and documentation reviews

Test run logs capture each execution's date, model version, test set version, tool versions, raw outputs and pass or fail results, and they should be kept unchanged after the run. Documentation reviews record who checked the datasheets, model cards and registers, what they found and what changed. Schedule reviews at each release and at least on the cycle the organization sets in its policy, and keep the approval trail with the records it covers.

What meeting MEASURE 2.1 costs in money and effort

This table breaks the cost of documenting MEASURE 2.1 test sets, metrics and tools into components, showing who is paid, whether each cost is one-time or recurring, and what pushes it up. Use it to build a quote request, because the largest recurring cost is staff time, not software licenses.

Cost component	Who is paid or who spends effort	One-time or recurring	Main cost drivers
Initial datasheets and model cards	Internal TEVV and data science staff, possibly an implementation partner for templates	One-time per model and dataset, then updated	Number of models, number of datasets, depth of test set description
Ongoing documentation upkeep	Internal TEVV owners, model owners	Recurring	Retraining frequency, dataset churn, changes to metrics or evaluation tools
Documentation tooling and hosting	Internal platform team; open-source tools carry no license fee but need hosting, backup and upgrades	One-time setup, recurring operation	Self-hosting footprint, access control design, upgrade cadence, staff hours for administration
Evidence upkeep after each model or data change	Model owners and TEVV staff re-running tests and updating records	Recurring, triggered by change	Release frequency, number of deployed model versions, degree of automation in test logging
Periodic review of documentation tools	Governance or TEVV leads, as the Playbook suggests regular effectiveness checks	Recurring	Number of teams using the tools, regulatory scrutiny, changes to the AI RMF Playbook
External review by an independent party (optional)	Independent reviewer or consultant, selected and paid by the organization	Usually one-time per review cycle, may recur	Scope of review, number of models reviewed, depth of sampling you request in writing
Assessor time during audit sampling	Auditor or assessor, where a framework such as ISO/IEC 42001 is in scope	Recurring at each audit cycle	Number of sampled models, completeness of records, how quickly evidence can be retrieved
Staff training on documentation practice	Internal staff or external trainers	One-time, then refresher	Number of teams, staff turnover, complexity of the documentation templates

Which open-source tools support MEASURE 2.1 evidence

garak is an open-source scanner, released under the Apache-2.0 license, that probes language models for weaknesses such as hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. For MEASURE 2.1, its output is useful because it records what was tested and what happened, which gives assessors a starting point for the test set and tool details behind a language model security evaluation.

Attempt-level record: garak writes a JSONL report covering every probe attempt and its evaluation, so the test set can be reconstructed from the file rather than from memory.
Hit log: A separate hit log lists the attempts that found a vulnerability, which helps separate confirmed findings from the full test population.
Tool and probe versions: Record the garak version and the probe modules used in each run, since results depend on them.
Model under test: Store the model name, version and endpoint alongside each report so results map to a specific system.
Retention: Keep reports with the run's configuration and date under the organization's own retention rules.

Before relying on garak output as MEASURE 2.1 evidence, confirm with your security team which probes are in scope, how the reports are stored, and whether the coverage matches the risks identified under MAP.

Where MEASURE 2.1 documentation usually breaks down

Most MEASURE 2.1 documentation fails in predictable places: the data behind a test, the thresholds behind a metric, the model a card describes, and the tools that produced the results. Each gap below can be checked against records an assessor will ask to see, and each has a fix that a documentation owner can put in place before the next TEVV cycle.

Test sets without provenance

A test set is documented when its source, collection date, labeling method, known exclusions and version identifier are recorded alongside the results. Without that record, a reviewer cannot tell whether a strong score reflects the system or a test set that overlaps its training data. Check that every evaluation run points to a dated, versioned test set and that a datasheet for each dataset states how it was built. Confirm the owner can explain how the test set was split from training data.

Metrics without thresholds or context

A reported accuracy or error rate means little unless the document states the acceptance threshold, the population it was measured on, the conditions and the reason that threshold was chosen. Teams often record the number and omit the rest, so later reviewers cannot judge whether a result passed. For each metric, check that the document records its definition, the acceptance threshold, the decision it supports and the person who approved that threshold.

Model cards describing a previous version

A model card is only accurate for the model it describes. When a model is retrained, fine-tuned or swapped for a newer release, the card often keeps the earlier intended use, training data summary and results. Check that each card carries the exact model version, the date of its last evaluation and a change log linking the card to the evaluation run that supports it. Any version without a matching card should be treated as undocumented.

Tool versions not recorded

The evaluation tools themselves shape results. A change in a scanner, a test harness or a scoring library can move results without any change to the model. Record the name, version and configuration of every tool used in each TEVV run, along with the random seeds and parameters that affect output. Teams running open-source probes such as garak should keep the version and the probe settings with each report, since the JSONL attempt log and hit log only support conclusions when the tool that produced them is known.

Documentation kept outside version control

Evaluation reports stored as loose files or in shared folders lose their history. Without version control, nobody can show what a document said at the time of a decision, who changed a threshold or when a result was revised. Keep TEVV documentation in a wiki or document system that records each change with its author and date, and allows comparison and rollback. Check that approved reports are locked against silent edits and that the assessor can see the revision history.

No scheduled review of documentation tools

MEASURE 2.1 expects the documentation practices themselves to be checked, not only the model. Many teams set up datasheets, model card templates and evaluation logs once and never review them. Set a recurring review that tests whether the templates still capture the fields needed for decisions, whether the tools still run and whether the access settings on the documentation spaces still match the people who need them. Record each review and the changes it produced.

Questions buyers ask about MEASURE 2.1 documentation

How much effort does MEASURE 2.1 documentation take?

The effort depends mostly on how many AI systems are in scope, how many evaluations each one runs, and how much of the test history already exists in usable form. MEASURE 2.1 asks that the test sets, metrics and details of the tools used during TEVV are documented, so the work is less about writing new reports and more about capturing what each test used and produced. Teams with stable evaluation pipelines usually need to add structure and versioning. Teams that run ad hoc notebooks need to reconstruct records for past runs and set a routine for new ones. Ask any partner to size the effort by system, by evaluation type and by the number of tool changes you expect over a year. Keep in mind that the documentation must be maintained, not written once.

Can we document MEASURE 2.1 in-house without a consultant?

Yes. NIST AI RMF is voluntary and has no certification, so an organization can carry out MEASURE 2.1 with its own TEVV, data science and governance staff. The Playbook suggests practices such as datasheets for datasets and model cards, and these can be produced by the people who run the tests. Outside help is most useful where the team lacks a documentation template, needs wiki or permission configuration, or wants a second reviewer to check that the records would let someone else repeat a test. Whatever route you choose, the organization owns the content. Confirm internally who signs off on test sets and metrics before work starts.

How long does it take to get the first MEASURE 2.1 records in place?

The timeline is driven by four factors: the number of systems in scope, whether test runs are already logged, how many people must review the templates, and how tightly your wiki and access controls must be set up. A team that already stores evaluation outputs in one place can often move faster than a team that must first agree on what a test set is. Ask for a timeline tied to a written scope with named systems and named deliverables. Any date given without that scope should be treated as an estimate, not a commitment. Your internal approval cycle is often the longest step.

Is there a MEASURE 2.1 certificate or accredited assessment?

No. The NIST AI RMF has no certification, accreditation or conformity assessment program, and NIST does not validate or endorse any organization's use of the framework. Nothing issued under the AI RMF can be called a certificate. What a customer or regulator can review is your own documentation: test sets, metrics, tool versions and the records showing how results were produced. If a vendor offers an AI RMF certificate, ask who issued it and under what published scheme. If a contract asks for evidence of MEASURE 2.1, provide the records and the process that produces them, and agree on the format with the requester.

What does the Generative AI Profile add for MEASURE 2.1?

The Generative AI Profile, NIST AI 600-1, lists no suggested actions for MEASURE 2.1. The profile states that not every AI RMF subcategory is included and that subcategories it does not address may be added later. For generative AI systems, you still apply the base MEASURE 2.1 outcome and the Playbook practices, such as documenting test sets, metrics and tools. Teams using generative AI often extend those records to cover prompts, model versions, safety probes and tool configurations, but that extension is your design choice, not a profile requirement. Check the profile text directly before citing it in a contract.

What should we ask a partner for in a fixed-scope quote?

Ask for a written scope that names the AI systems, the evaluation types in scope, the documentation tools and the deliverables. A useful quote lists templates for datasheets and model cards, a review of the test sets and metrics currently in use, the configuration of version control and access permissions in your wiki or documentation tool, and a handover of the process. Confirm which items are one-time work and which recur, such as updating records when a test set changes. Ask who in your organization must review and approve each deliverable. Ask what the partner will not do: DeployOpen prepares systems and evidence, and only an independent auditor or assessor issues reports or certificates.

How DeployOpen prepares MEASURE 2.1 evidence

DeployOpen starts with a written scope that names the AI systems in question, the TEVV activities that produce test sets, metrics and tool records, the repositories and wiki spaces that will hold them, and the people who approve each artifact. Before any build work begins, the client and DeployOpen agree on which documentation practices apply, such as datasheets for datasets and model cards for models, and on who owns each one. Because MEASURE 2.1 is tagged to the TEVV actor category, the scope should also confirm which internal team members perform the testing and which ones review the documentation. The scope document is the reference point for every later decision, so changes to it should be agreed in writing.

DeployOpen deploys the documentation and tooling inside the client's own cloud accounts, data centers or air-gapped networks. XWiki keeps every change to a page under version control, recording what changed, when and by whom, and it allows any two versions to be compared and rolled back. BookStack assigns abilities through roles, which can be overridden per role at the shelf, book, chapter or page level, so write access to test records can be separated from read access for reviewers. Testing outputs such as garak JSONL reports and hit logs, or Langfuse traces for LLM applications, can be stored alongside those pages. Clients should confirm that each tool's license, version and data retention settings meet their own policies before deployment, and that backups of these systems are covered by their existing procedures.

MEASURE 2.1 Documenting test sets, metrics and tools