MAP 2.3 Scientific integrity and TEVV evidence

What MAP 2.3 scientific integrity and TEVV evidence proves about an AI system

When a team or auditor asks how an AI system's test and evaluation reasoning is recorded, the answer should point to MAP 2.3, which asks organizations to identify and document scientific integrity and TEVV (test, evaluation, verification and validation) considerations. In practice that means showing the experimental design, how training and test data were collected and selected, the basis for claims of trustworthiness, and whether the measures used actually capture the property being claimed (construct validation). NIST AI 100-1 is voluntary, and NIST does not certify or validate any organization's use of the AI RMF, so no one will issue a certificate for this work. The value lies in whether your own records would let an internal reviewer or outside evaluator reconstruct why a result should be trusted.

The main risk is a result that looks valid on a dashboard but rests on data nobody examined or on a metric that measures something other than the intended behavior. A benchmark score may reflect overlap between training and test sets, a non-representative sample, or a proxy that does not track the harm you care about. This is hard to catch because the evidence is spread across teams. Data scientists hold the labeling and sampling decisions, engineers hold the pipelines and deployment conditions, and domain experts know what a correct output looks like. Before a go or no-go decision, ask each group to write down its assumptions, and confirm that the data lineage for training and test sets can be traced from source to evaluation run.

A practical starting point is a short TEVV protocol for each model and each deployment configuration, covering the experimental design, the data documentation and the metrics chosen for each claim. Ask whether the protocol names who can corroborate the results and whether an independent evaluator can rerun them from the recorded inputs. Confirm these points with your own governance owners and with the Playbook text for MAP 2.3 before you rely on them, because the Playbook's audit log shows its last content update in August 2023, and NIST's AI RMF Playbook page describes it as a companion that is neither a checklist nor a set of steps.

Which systems, datasets and teams fall under MAP 2.3

MAP 2.3 applies to whatever the organization builds, adopts or relies on to produce AI outputs, so the scope decisions come first. Each card below is a boundary question that determines how much scientific integrity and TEVV documentation the organization must keep and who must be able to explain it.

In-house versus pre-trained models

Models the organization trains or fine-tunes carry the full set of design, data and evaluation records, because the team controls every step. Pre-trained or third-party models shift part of that burden to the provider, but the organization still needs documented assumptions about the model's training data, its tested behavior and the gaps it cannot see. Ask the provider for its TEVV summary before scoping the model out.

Generative versus classical systems

Generative systems raise the TEVV bar because outputs vary and cannot be checked against a fixed label. MAP 2.3 for generative AI adds accuracy, authenticity and fact-checking evidence against ground truth, which NIST AI 600-1 ties to this subcategory. Classical classifiers and recommenders can rely more on held-out test sets and error analysis, so the documentation burden is smaller but not absent.

Training, test and retrieval datasets

Every dataset that shapes the system's behavior belongs inside the boundary, including fine-tuning sets, evaluation sets and any retrieval corpus. The organization must document how data was selected, curated and labeled, and must trace lineage so an assessor can see which data trained or tested which version. Datasets used only for ad hoc experiments should be recorded too if their results informed a decision.

Labeling vendors and annotation work

Outsourced labeling affects representativeness and suitability, so labeling guidelines, annotator qualifications and quality checks must be documented and available to the organization. Confirm with each vendor how it records labeling decisions, handles disagreements and allows sampling audits. Without that record, the organization cannot explain how labels were produced or defend them to an independent evaluator.

Internal and external TEVV teams

Anyone who designs tests, runs evaluations or accepts results is inside the MAP 2.3 boundary, whether employed internally or contracted. The Playbook tags this subcategory to TEVV and domain experts alongside AI development, so each group needs documented responsibilities and a way to corroborate results. Independent evaluators need enough access to repeat tests, which the boundary decision should plan for in advance.

Deployment and operation environments

TEVV protocols must cover the system as deployed, not only the model in a notebook. Include serving infrastructure, prompt templates, tool integrations and monitoring pipelines that change outputs in production. An illustrative case: a model tested alone may behave differently once a retrieval step or a new tool is added, so that configuration belongs in the documented scope.

How an assessor tests MAP 2.3

This illustrative table shows how an assessor might test each element of MAP 2.3, the evidence they would ask for and the findings they often raise. NIST publishes no certification assessment procedure, so treat the approach as one reasonable method, not an official test. Use it to prepare evidence before an outside review.

Element	Test approach	Evidence requested	Common findings
Experimental design	Confirm the assessor can trace each evaluation to a stated question, success criteria and the population the system will serve, then check that the design was fixed before results were seen.	Evaluation plan with hypotheses, acceptance thresholds, dated versions of the plan and records of any changes made after testing began.	Thresholds set after results were known, or evaluation scope that omits the deployment context described in MAP 1.
Data collection and selection	Sample the training, tuning and test datasets, then check documented assumptions on availability, representativeness and suitability against the intended use.	Data sheets, selection and curation criteria, labeling guidelines, inter-annotator checks and lineage records from source to test split.	No record of why data was chosen, test data overlapping training data, or labels created without guidance or quality checks.
System trustworthiness and construct validation	Check that the metrics measure what the organization claims the system does, then rerun key tests on realistic inputs and review capabilities beyond those planned.	Metric definitions with rationale, construct validity analysis, results on out-of-scope and edge inputs, and TEVV protocols for the model and its deployment.	Proxy metrics presented as proof of the claimed capability, or no test for behavior outside the planned task.
Independent corroboration	Ask whether someone outside the build team can reproduce results and whether independent evaluators had access to the protocols and data needed to check them.	Reproduction instructions, pinned environments, access records for independent evaluators and their written findings.	Results that only the developing team can rerun, or evaluator access limited to summary tables.
Generative AI actions (MP-2.3-001, 003, 005)	For generative systems, assess whether output accuracy is compared against ground truth, whether fact-checking is applied and whether adversarial testing follows a planned schedule.	Ground truth sources, fact-check procedures, test schedule and results of regular adversarial tests with dates.	Accuracy claims with no ground truth reference, or adversarial testing run once before launch and never repeated.
Change control over TEVV	Verify that a model, dataset or deployment change triggers a repeat of the affected TEVV steps and that the record shows the link.	Change tickets linked to test runs, version history for models and datasets, and sign-off by the owner of the TEVV protocol.	Model updates released with no re-evaluation, or test records that cannot be matched to a deployed version.

Sequence for documenting TEVV considerations in order

This sequence turns MAP 2.3 from a statement of intent into a documented record: how the system was tested, what data it learned from and whether its results can be checked by someone other than the team that built it. Each step produces an artifact that the next step depends on.

Record the experimental design and assumptions
The AI development team, with TEVV and domain experts, writes down what the system was built to test, the hypotheses behind the design, the assumptions about users and inputs, and the conditions under which results should not be trusted. Record the evaluation approach before data is pulled for testing, so later results cannot quietly reshape the question. Done means a dated design record exists, names an accountable owner, and lists each assumption that a later step will test against.
Document data sourcing and selection
Development and TEVV staff record where each dataset came from, why it was chosen, how well it covers the population and use cases the system targets, and whether it is suitable for the task. Note gaps in availability and representativeness, and any known skews. Done means a data sourcing record covers every dataset, with a stated rationale and a list of known limitations that domain experts have reviewed.
Document curation and labeling decisions
Staff record the cleaning, filtering, transformation and labeling steps applied to each dataset, along with who labeled the data, the instructions they followed and how disagreements were resolved. Where labels were produced by people, record their qualifications and any quality checks. Done means each decision can be reproduced from the record, and the assumptions from the design step are either confirmed or flagged as open.
Trace training and test data lineage
Maintain a lineage record that links each training and test dataset version to the model versions it produced and the evaluations that used it. Confirm that test data was kept separate from training data and that no version of a test set leaked into training. Done means an assessor can start from any reported result and follow it back to the exact data and code that generated it.
Define TEVV protocols for models and deployment
Write the test, evaluation, verification and validation protocols for the model, its subcomponents, the deployment environment and the operating conditions. Specify the metrics, pass criteria, test environments and who runs each test. Include deployment-like testing where the target environment differs from development. Done means each protocol is approved by TEVV and domain experts, is version-controlled, and maps back to the assumptions and risks recorded in earlier steps.
Test for capabilities beyond those planned
Run adversarial and exploratory testing to find behavior outside the intended purpose, such as unexpected outputs, data leakage, or failure under unusual inputs. For generative systems, the Generative AI Profile calls for regular adversarial testing and fact-checking against ground truth. Record each finding, its severity and whether it changes the design record or the risk treatment. Done means the test log is complete, and any new capability is either documented as in scope, restricted or escalated for the go or no-go decision.
Arrange independent corroboration
Give the design record, lineage record, protocols and test results to evaluators who did not build the system, and record their findings. Independent evaluators may be internal teams outside the development chain or external testers. Their reproduction of key results, and any disagreement they raise, is retained with the TEVV file. Done means the independent review is documented and each open finding has an owner and a disposition before the record is used for a deployment decision.

How NIST AI 600-1 adds generative AI actions under MAP 2.3

NIST AI 600-1 adds five suggested actions under MAP 2.3 for organizations that build or run generative AI. The Generative AI Profile, published July 26, 2024, numbers them MP-2.3-001 to MP-2.3-005 and ties them to four of its twelve generative AI risks: Harmful Bias and Homogenization, Information Integrity, Information Security, and Intellectual Property. The profile does not replace the base MAP 2.3 subcategory in NIST AI 100-1. It adds generative-specific ways to show that the scientific integrity and TEVV work for a language or image model is sound. Before you rely on any action, open the July 2024 profile and confirm the wording, the risk tag and the actor roles it lists, because the revision now under way may change how the AI RMF core is cross-referenced.

The most concrete actions are MP-2.3-001 and MP-2.3-003. MP-2.3-001 asks you to assess the accuracy, reliability and authenticity of generative output against ground truth data. In practice this means building a labeled reference set for the tasks the system will perform, scoring outputs against it, and keeping the scores with the model version and prompt template used. MP-2.3-003 asks for fact-checking techniques. Typical choices include retrieval-grounded checks against a trusted source, citation verification and human review of high-impact answers. Decide who owns the ground truth set, who can change it, and what accuracy threshold blocks a release. Without those decisions, the scores cannot support a go or no-go call under MANAGE 1.

MP-2.3-005 asks you to plan regular adversarial testing. Use it to define a cadence, a scope and a register of findings. Open-source probes such as garak can run hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreak tests, and write a JSONL report of every attempt plus a hit log of successful attempts, which gives the evaluation a repeatable record. Your team should decide who reviews the hit log, how findings feed back into training data or guardrails, and whether independent evaluators can rerun the tests, which the Playbook suggests under MAP 2.3. Confirm the test categories and any new triggers, such as a model swap, against the profile before you publish the plan.

Evidence that passes a MAP 2.3 review

A MAP 2.3 review asks whether the organization can show how its system was designed, what data it learned from, and how its trustworthiness was tested. The cards below name the records that answer those questions, and each one needs a routine that keeps it accurate between formal audits rather than a one-time rebuild before an assessment.

Experimental design records

These records state the hypotheses, the comparison conditions, the success criteria and the reasons each design choice was made before results came in. A reviewer uses them to judge whether a reported result could have failed. Keep them under version control next to the code, and add a dated entry whenever the evaluation plan changes, with the reason for the change. Check that every test result links back to the design it was run under.

Data sheets with provenance

A data sheet records where each dataset came from, who collected it, how it was selected, and how it was labeled, along with known gaps in representativeness and suitability for the task. The MAP 2.3 Playbook suggestions ask for documented assumptions behind data selection, curation and labeling. Refresh the sheet whenever a source is added, a labeling guide changes or a dataset is filtered, and record the reviewer who approved the change.

Training and test data lineage traces

Lineage traces show how each training and test dataset moved from source through transformations into the model build, so that a reviewer can confirm test data stayed out of training. Generate traces from pipeline runs rather than writing them by hand, and store the run identifier with each model version. Re-check lineage after any pipeline change, since a silent join or sampling step can leak test records into training.

TEVV protocols and version control

A TEVV protocol specifies how the model, its subcomponents, its deployment and its operation are tested, including the tools, thresholds and frequency. Store protocols in a repository with protected branches so that only approved roles can change the acceptance criteria. Review the protocol at each model release and whenever the deployment context changes, and confirm that the protocol in force matches the one used for the latest results.

Ground truth comparisons and test results

Test results should compare outputs against an agreed ground truth for accuracy, reliability and authenticity, and should keep the raw outputs next to the scores. For generative systems, the Generative AI Profile ties this work to fact-checking and adversarial testing. Rerun the comparison on a schedule and after each change, and retain the failed cases along with the passed ones so the record shows what the system gets wrong.

Independent evaluator input

The Playbook suggests enabling corroboration by independent evaluators, which means giving people outside the build team access to the protocols, data sheets and results they need to reproduce findings. Keep a log of each external or internal review, its scope, its findings and the team's response. Confirm access rights remain current, because a reviewer who cannot reproduce a result gives the organization no corroboration.

Which open-source tools produce MAP 2.3 evidence

Probing tools generate evidence for the parts of MAP 2.3 that call for testing a language model against known failure modes and planning regular adversarial testing. garak, released under the Apache-2.0 license, probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. Its output is useful because it records what was tried, not only what passed.

Attempt record: garak writes a JSONL report of every attempt and its evaluation, which gives reviewers a complete denominator rather than a selection of successes.
Hit log: A separate log lists attempts that found a vulnerability, so the team can trace each finding to the probe that produced it.
Probe coverage: Record which probe families ran against which model version, since results from one model build do not transfer automatically to another.
Repeat schedule: Keep dated runs to show the regular adversarial testing that the Generative AI Profile actions call for.
Ground truth gap: Probes do not supply your own ground truth data, so accuracy checks against domain data still need separate test sets.

Confirm with your assessor which probe results and run dates they expect to see, and whether the report format satisfies their documentation needs.

What meeting MAP 2.3 costs in money and effort

This table compares the cost components of MAP 2.3 by who pays for each, whether it is one-time or recurring, and what moves it up or down. Use it to build a quote request, since no supplied figures give prices or durations for this work. The main takeaway is that most of the cost sits in internal TEVV time and documentation effort, while open-source tooling mainly removes license spend rather than labor.

Cost component	One-time or recurring	Who is paid or who spends time	What drives it up or down
Internal data science and TEVV time	Recurring	Your own data scientists, ML engineers and TEVV staff	Number of models, subcomponents and deployment contexts in scope; reuse of a standard TEVV protocol lowers effort
Data documentation and labeling effort	One-time build, recurring updates	Your data owners, labelers and domain experts	Volume and age of training and test data; whether lineage and label decisions were recorded when the data was collected
Test protocol and assumption documentation	One-time build, recurring review	Your AI development and TEVV leads, with domain experts	Number of documented assumptions and data selection techniques; how often the system changes
Open-source evaluation and observability tooling	Mostly one-time setup, recurring hosting	Your platform team for installation, configuration and hosting	Tool licenses are often free to use (for example, Apache-2.0 or MIT), so cost shifts to setup, integration and compute
Independent evaluation or adversarial testing	Recurring per review cycle	External evaluators or an outside testing firm you contract	Scope of adversarial testing, frequency of reviews and whether results must be reproducible by a third party
Auditor sampling and evidence review	Recurring per assessment	Independent auditor or assessor, not DeployOpen	Quality of traceable evidence; a clean lineage trail and versioned test results reduce sampling requests
Remediation of TEVV gaps	One-time, occasional	Your engineering and data teams, sometimes an implementation partner	Count of gaps found in pre-assessment, such as undocumented data sources or untested capabilities beyond the plan

Who owns MAP 2.3 work, who can help and where the auditor stops

The organization owns MAP 2.3 in full. Its AI development, TEVV and domain expert staff decide the experimental design, choose which data is collected and selected, and accept or reject the results of testing. No outside party can take these decisions over without the organization's sign-off, and the NIST AI RMF does not allow the organization to treat them as delegated. The framework is voluntary, so NIST does not certify anyone for doing this work, and the organization must be able to explain its own choices to a regulator or customer.

Experimental design decisions: Record which hypotheses the system is tested against, which failure modes are in scope, and why certain scenarios were excluded.
Data selection and curation: Document the assumptions behind data selection, curation and labeling, including who labeled it and under what guidance.
Data lineage ownership: Keep traceable records linking training and test data to their sources and to each evaluation run.
Acceptance of TEVV results: Name the person who reviews results and decides whether the system meets its stated trustworthiness goals.
Retained records: Keep the protocols, decision logs and results in a location the organization controls, so they remain available after any vendor engagement ends.

Before you engage anyone, confirm which named individuals hold each of these decisions and whether they can explain them without help.

Where organizations fall short on TEVV documentation

Most gaps under MAP 2.3 are not missing tests. They are missing records: why data was selected, where labels came from, what the evaluations never covered, and who checked the results. Each card below pairs one gap with a check that closes it.

Undocumented data selection

Teams often record which dataset was used but not why it was chosen, what was excluded, or whether it represents the people and conditions the system will face. MAP 2.3 asks for documented assumptions and techniques used in data selection and curation. Check that every training and evaluation dataset has a short record covering its source, selection criteria, exclusions, and known gaps in availability and representativeness.

Missing lineage from labeling vendors

Labels produced by an outside vendor often arrive as a file with no record of annotator instructions, revisions, or quality checks. When a model later behaves oddly, the team cannot trace which labels shaped it. Check that the contract requires labeling guidelines, version history, and sampling-based quality results, and that each label batch is linked to the dataset version it entered.

Evaluations limited to planned capabilities

Test suites usually cover the tasks the team designed for, so behavior outside those tasks goes unmeasured. MAP 2.3 suggests checking for capabilities beyond those planned. For generative systems, this means probing for hallucination, data leakage, prompt injection, and jailbreaks, and recording every attempt and result so that gaps in coverage are visible.

No independent corroboration

When the team that built a system is also the only team that tests it, errors and blind spots go unchallenged. The Playbook suggests enabling corroboration by independent evaluators. Check whether an evaluator outside the development team can reproduce key results, review the test protocol, and access the logs needed to verify claims about accuracy and reliability.

TEVV protocols that stop at the model

Protocols often cover the model alone, while the risks show up in subcomponents, retrieval steps, tools, and deployment settings. MAP 2.3 asks for TEVV protocols covering models, subcomponents, deployment, and operation. Check that each component in the system inventory has a named test owner, a test scope, and a rerun trigger tied to changes in that component.

Results without a traceable record

Evaluation results are often kept in notebooks or slide decks, so no one can later show which model version, dataset, and configuration produced a given score. Check that each evaluation run stores its inputs, outputs, and settings in a system that can be queried, and that the record links back to the protocol and the approval decision it supported.

Questions buyers ask about MAP 2.3 cost, effort and scope

How much effort does MAP 2.3 take?

The effort depends on how many AI systems are in scope, how much training and test data they use, and how many TEVV activities the team must document. No supplied figure gives a standard duration, so ask for a scoped estimate rather than relying on a generic number. The main work is documenting experimental design, data selection and lineage, system trustworthiness and construct validation for each system. A single narrow classifier with well-known data will take far less effort than a generative system that draws on third-party models and retrieval sources. Effort also rises when the organization has no existing record of how its data was collected or labeled, because that history must be rebuilt before it can be documented.

Can we do MAP 2.3 in-house?

Yes, an organization can do MAP 2.3 with its own staff, because NIST's framework is voluntary and no external party must perform the work. The outcome requires documented experimental design, data lineage and TEVV protocols, which are internal artifacts that the organization owns. Teams with mature data governance and an established model evaluation practice usually find the work manageable. Teams without those practices often bring in outside help to set up the documentation method and evidence trail. An implementation partner can prepare systems and records, but the organization still decides which claims about its AI system are true and accepts the risk.

What does MAP 2.3 need from data science and domain experts?

MAP 2.3 needs data scientists to explain how training, validation and test data were collected, selected, curated and labeled. The Playbook tags this subcategory with the AI Development and TEVV actor categories, and it also lists Domain Experts. Domain experts are needed to judge whether the data is representative of the real setting and whether the evaluation measures what matters in that context. In practice, the data science team writes the lineage and protocol records, while domain experts review the suitability of data and the construct validity of the tests. Ask who will sign off each record and how much of their time the review requires.

Is there an AI RMF certificate for MAP 2.3?

No, there is no AI RMF certificate, and no one can certify that an organization meets MAP 2.3. NIST does not validate or endorse any individual organization's use of the framework, and it describes no certification, accreditation or conformity assessment program. Be cautious with any vendor or consultant who offers an AI RMF certificate. If a customer or regulator asks for proof, the organization can provide its documented MAP 2.3 records and, if required, an assessment performed by an independent party under a separate agreement. Only that independent party can issue a report, and the report describes its own findings, not an AI RMF certificate.

How does MAP 2.3 relate to the NIST AI RMF Playbook?

The Playbook gives suggested actions, documentation prompts and references for MAP 2.3, but it is not a checklist and does not require a set sequence of steps. Its suggested actions include developing TEVV protocols, documenting data selection and labeling assumptions, tracing training and test data lineage, checking for capabilities beyond those planned, and enabling corroboration by independent evaluators. The Playbook's audit log shows its last content update in August 2023, and NIST's AIRC says it will be updated after the AI RMF is revised. Organizations should use the Playbook as a starting reference and confirm current wording against NIST's own pages before relying on it in contracts.

What should we ask an implementation partner for in a fixed-scope quote?

Ask for a written scope that names each AI system, each data source and each TEVV activity covered by MAP 2.3. The quote should list the deliverables, such as lineage records, TEVV protocols, data selection documentation and the evidence trail for each system, along with who approves them. Ask which items are one-time work, such as the initial documentation, and which recur, such as updating lineage when data changes or re-running tests after a model update. Confirm which tasks the organization must still perform itself, and that the partner prepares records but does not issue audit reports or certificates. DeployOpen plans and delivers most engagements against a written scope agreed before work starts.

How DeployOpen prepares TEVV evidence for MAP 2.3

DeployOpen prepares the systems and records behind MAP 2.3 in writing, before any tooling is installed. The scope names which AI system is covered, which TEVV protocols apply to the model, its subcomponents, deployment and operation, which data sources and labeling steps must be traced, and which independent evaluators will review the results. The organization owns the decisions the outcome depends on, including what counts as acceptable experimental design and who signs off on data suitability. DeployOpen's role is to build the environment in which those decisions can be evidenced, and the scope document should state that division of responsibility plainly.

Inside the client's own cloud account, data center or air-gapped network, DeployOpen can deploy self-hosted open-source tooling that produces the raw material for MAP 2.3 evidence. For generative AI systems, garak can run probes for hallucination, data leakage, prompt injection and misinformation, and its JSONL report and hit log record every attempt and evaluation. Langfuse documentation on observability describes traces that capture LLM calls, retrieval steps, tool executions, inputs, outputs and metadata, which supports lineage and reproducibility checks. GitLab protected branches and Semgrep scans in CI/CD can keep the test code and evaluation scripts under change control. Confirm each tool's fit with the client's model stack before committing to it, since the tooling covers the test evidence, not the choice of metrics.

Evidence packs are assembled from these outputs and handed to the organization and its chosen independent evaluators. A typical pack, which DeployOpen would agree on in the written scope, could include the TEVV protocol, the documented data assumptions and lineage records, probe reports, trace exports, and a list of open findings with owners. DeployOpen can support retesting and continuous testing under a retainer if the scope calls for it. DeployOpen does not issue audit reports, attestations or certifications, and NIST AI RMF has no certification program, so the evaluation conclusions belong to the organization and to any independent assessor it engages. Ask any partner, including DeployOpen, to state in writing which artifacts it delivers and which decisions remain with the client.

MAP 2.3 Scientific integrity and TEVV considerations