MEASURE 2.2 human subject evaluations: evidence guide

What MEASURE 2.2 asks for when people take part in AI testing

If your team runs a model or generative AI system past human participants, MEASURE 2.2 expects those evaluations to meet the human subject requirements that apply to them and to include people who represent the population the system will actually affect. The participants might be labelers scoring outputs, testers probing the system, members of a user panel or future end users in a pilot. The NIST AI Risk Management Framework is voluntary, and there is no AI RMF certificate. Nobody certifies that you meet MEASURE 2.2. An auditor, customer or regulator who asks about it will want to see how you did the work and what you can show for it. The MEASURE 2.2 outcome is defined in NIST AI 100-1, which you should read directly alongside the NIST AI RMF Playbook for its suggested actions.

The risks run in two directions. Participants can be harmed through exposure to offensive or distressing content, unclear data reuse, or pressure to take part, and the organization can face legal and reputational exposure as a result. The evaluation itself can also be wrong: a panel drawn from convenient staff or a single region can produce results that look strong but do not hold for the users who matter. Both problems are hard in practice. Consent has to be specific enough that people understand what their judgments will be used for. Compensation has to be fair for the time and exposure involved. Sampling has to reach groups that are harder to recruit, such as older users, non-native speakers or people in specific regions, and that often takes more time and budget than the testing team expects.

Before the first session, confirm three things with your legal, privacy and research leads: which human subject rules apply to your study in each jurisdiction, what consent and withdrawal process you will run, and which population segments the evaluation must cover. Record those decisions as documentation that a TEVV owner, a human factors specialist and an AI developer can each read. If the results will feed a go or no-go decision, also confirm who signs off on the sample design. External assessors can review your design and records, but only they can issue an independent opinion, and they will ask for the same evidence this step produces.

Which human subject evaluations fall under MEASURE 2.2

MEASURE 2.2 applies whenever people, not only the model, shape the evidence about a system. The scope questions below decide which activities need human subject protections, which data about people enters the evaluation, which teams own the protocol and which outside parties are in the chain. Settle these before any testing starts, because each answer changes the consent, documentation and representativeness work that follows.

Staff red teaming and internal testing

Red teaming by employees, contractors or security staff often feels like internal QA, but when testers rate outputs, probe for harmful content or record their own reactions, the activity can still be a human subject evaluation. Check whether testers are asked for personal judgments about content, whether their responses are stored with identifiers, and whether participation is voluntary. Treat staff testing as in scope whenever individual responses are recorded as evaluation data.

Paid annotators and labelers

Annotators who label outputs for accuracy, toxicity or bias create evaluation data about the people doing the labeling and often about the people described in the content. MEASURE 2.2 suggestions point to following human subjects research requirements, including informed consent and compensation. Confirm what annotators agreed to, how they are paid, whether their demographic information is collected, and how their labels are checked for consistency across groups.

Beta users and pilot participants

Beta testers and pilot users interact with the system in real conditions and their feedback becomes evaluation evidence. Their use may also expose personal data they did not expect to share. Check whether the pilot notice explains that interactions are studied, whether participants can withdraw, and whether the pilot population matches the people the system will serve. Illustrative example: a pilot limited to one region may hide performance gaps that appear once the system is deployed elsewhere.

Data about people in test sets

Test sets often contain real records, conversations or documents that describe identifiable people. Using them for evaluation raises privacy and intellectual property questions that MEASURE 2.2 asks organizations to address. For generative AI, the Generative AI Profile lists withdrawal of consent and privacy-enhancing methods such as anonymization and differential privacy. Confirm the legal basis for each source, which fields are removed, and whether any subject can revoke consent for evaluation use.

Ownership across TEVV, Human Factors and AI Development

The Playbook tags MEASURE 2.2 to three AI actor groups: TEVV, Human Factors and AI Development. Each group holds a different piece of the protocol, so an unowned step is a common gap. TEVV usually owns test design and metrics, Human Factors owns participant experience and consent wording, and AI Development supplies the system under test and handles fixes. Ask for a named owner for consent, sampling, disaggregated results and sign-off before any evaluation begins.

Third-party testing vendors and crowdsourcing platforms

External red teamers, evaluation firms and crowdsourcing platforms can handle participant recruitment, consent screens and payment on your behalf. Responsibility for human subject protections does not transfer with the work. Put in the contract who recruits participants, what consent text they use, where responses are stored, how long records are kept and what the vendor reports back on demographic coverage. Ask for the same documentation you would expect from an internal study.

How an assessor tests human subject evaluations

This table compares what an internal auditor or independent reviewer checks for each part of MEASURE 2.2, the evidence they request and the gaps they most often report. Because no AI RMF certification body exists, the reader should treat these findings as internal assurance results, not a third-party report.

Review area	Evidence the assessor requests	Common findings
Protocol review	Approved study or evaluation protocol, ethics or institutional review decision, and the record of who reviewed and approved any change to the protocol	Evaluations with human participants run from a protocol that was never reviewed, or protocols that changed after approval without a record
Consent and withdrawal records	Consent forms in the language participants used, consent logs linked to each evaluation cohort, and a documented process for withdrawal or revocation of data use	Consent captured for the evaluation but not for reuse of the data in later generative AI development, or no working path for withdrawal
Compensation records	Payment or incentive schedule, payment logs by participant or cohort, and confirmation that compensation terms were stated before participation	Compensation terms missing or inconsistent across cohorts, or payments that cannot be traced to a specific evaluation
Disaggregated results by segment	Result tables broken out by demographic and other relevant segments, with the segment definitions and the method used to assign participants to them	Only aggregate scores reported, so performance gaps for affected groups stay hidden
Representativeness thresholds and alerts	Documented thresholds for how far a sample may diverge from the relevant population, alert configuration, and records of alerts raised and how they were resolved	Thresholds never defined, or alerts configured but no record of anyone reviewing them
Dataset intellectual property and privacy handling	Data licenses or permissions for third-party datasets, data minimization and retention records, and records of anonymization, differential privacy or other privacy-enhancing techniques where applied	Evaluation datasets reused without a documented license review, or personal data kept longer than the protocol allows

Why consent and representativeness must be evidenced together

A protocol can pass ethical review and still produce results that misrepresent the population the system will serve. Ethics approval shows that the study design protects participants. It does not show that the people who were evaluated match the people who will be affected by the AI system in deployment. MEASURE 2.2 asks for both properties, so a file that contains an approval letter but no demographic breakdown of the evaluation sample leaves the outcome unproven. The reverse also holds. A sample can look balanced on paper while some of its participants never validly consented to their data being used. Consent and representativeness are separate evidence trails, and an assessor will want to see each one traced to the same evaluation dataset.

The Playbook's suggested actions point to the same pairing: follow human subjects research requirements including informed consent and compensation, use disaggregated evaluation across demographic and other segments, and set thresholds and alerts for dataset representativeness. The practical check is to take the evaluation dataset and ask two questions of every record. Is there a consent basis that still applies to this use? Does the group it belongs to stay within the representativeness thresholds the team set in advance? Generative AI systems make the first question harder because data can be reused across prompts, fine-tuning and evaluation. NIST AI 600-1 adds suggested actions under MEASURE 2.2 (MS-2.2-003 and MS-2.2-004) that call for giving human subjects options to withdraw or revoke consent for use of their data in generative AI applications, and for using anonymization, differential privacy or other privacy-enhancing technologies. Organizations should confirm with their privacy and legal teams which of these mechanisms apply to their data flows before they design the evaluation.

How to implement MEASURE 2.2 in order

This sequence turns the MEASURE 2.2 outcome into a documented, repeatable process: applicable human subject rules are confirmed first, then the population, consent terms, evaluation segments and representativeness thresholds are fixed before any evaluation runs. Each step closes with a named artifact that an assessor or internal reviewer can inspect.

Confirm applicable human subject requirements
Legal counsel and the ethics owner identify which human subject rules apply to each evaluation, such as institutional review, national research ethics rules or contractual obligations with participants or data providers. The AI product owner documents the scope of each evaluation, including whether it involves staff, customers, crowd workers or data subjects. The step is complete when a signed requirements register lists each evaluation, the rules that govern it and the named approver. Confirm the specific obligations with counsel, since the rules differ by jurisdiction and sector.
Define the relevant population from MAP context
The TEVV lead takes the intended users, affected groups and deployment setting recorded in the MAP function and states which population the evaluation must represent. Human factors specialists note any groups who will use the system differently or bear its impacts more heavily. The output is a population definition document that links each group to its source in the MAP records. The step is complete when the definition has been reviewed by the ethics owner and any gaps in coverage are recorded as open items.
Write consent, compensation and withdrawal procedures
The ethics owner and legal counsel draft the consent notice, the compensation terms for participants and the procedure for withdrawing or revoking consent to use their data. The Generative AI Profile specifically asks that human subjects can withdraw or revoke consent for use of their data in generative AI applications, so the procedure should say how data is removed from evaluation sets and what happens to outputs already produced. The output is an approved consent template, a compensation schedule and a withdrawal runbook. The step is complete when participants can see each document before they join.
Set disaggregated evaluation segments
The TEVV team breaks the population into the demographic and other segments that matter for the system's risks, using the groups named in the population definition. Each segment is given a stated purpose, such as detecting performance gaps between groups or checking that a rare user type is not excluded. The output is a segmentation plan listing each segment, its data source and the metrics that will be reported for it. The step is complete when the plan is attached to the MEASURE 1 metrics record so every segment has a traceable reason to exist.
Set representativeness thresholds and alerts
The TEVV lead and the accountable owner agree thresholds for how far each segment's share of the evaluation data may drift from the share in the relevant population, and the minimum sample size needed before results are reported. Alerts are configured so that the team is notified when a segment falls below its threshold during data collection. The output is a threshold table with the owner's sign-off and the alert configuration. The step is complete when someone other than the data collector has confirmed that each threshold is measurable with the data the team can actually gather. Confirm your thresholds against any specific sector or customer requirements before adopting them.
Run and record the evaluations
Trained evaluators collect data under the approved consent and compensation procedures, and the team records every session, dataset version and protocol deviation. Data sets are checked against the intellectual property and privacy terms set in step one, with anonymization or other privacy-enhancing techniques applied where the plan requires them. Versioned records of protocols, participant counts per segment and data handling decisions are kept in a controlled wiki or knowledge base. The output is a dated evaluation log that can be traced back to the approved plan. The step is complete when the log reconciles with the consent records and the segmentation plan.
Review results against thresholds before decisions
The TEVV lead compares the results for each segment with the thresholds and alerts set earlier, and records any segment that is underrepresented, underperforms or triggered an alert. The accountable owner decides whether the findings require remediation, further data collection or a documented residual risk before the go or no-go decision. The output is a threshold review memo signed by the accountable owner, linked to the MANAGE 1 decision record. The step is complete when every threshold breach has a recorded disposition, and the memo is stored with the evaluation log for later review.

Which evidence passes a MEASURE 2.2 review

An assessor reviewing MEASURE 2.2 looks for proof that human subject evaluations were approved, consented to, paid for, representative and monitored over time. The cards below cover six evidence artifacts, each with a practical way to keep it current between reviews so the record reflects what actually happened.

Approved protocol and version history

The protocol sets the study purpose, participant criteria, consent process, risk review and data handling before any subject is enrolled. Keep every approved version, the approver, the date and the reason for each change. Check that the protocol in force matches the protocol the evaluation team actually ran. XWiki keeps page history with what changed, when and by whom, and lets you compare versions and roll back, which suits a protocol that is revised during a study. Confirm with your ethics or review board which approval record they expect to see.

Signed consent and withdrawal records

Consent records must show each participant's agreement and any later withdrawal or revocation. For generative AI evaluations, NIST AI 600-1 suggests giving subjects options to withdraw or revoke consent for use of their data. Store the signed or logged consent with its version, and log every withdrawal with its date and the downstream action taken, such as removing records from future test sets. Check that withdrawn data is excluded from later runs, not only from the original study file.

Compensation ledger

The Playbook suggests following human subjects research requirements including compensation. A compensation ledger links each payment to a participant pseudonym, a study phase and an approved rate, so a reviewer can confirm that payments match the protocol. Record amounts, dates and any withheld or adjusted payments with a reason. Reconcile the ledger against the finance system at each study close. Ask your review board whether it has rules on compensation level or timing for your study type.

Disaggregated results by segment

Aggregate scores can hide weak performance for a subgroup, so MEASURE 2.2 expects evidence that the evaluation population was represented and results were broken out by demographic and other relevant segments. Keep the segment definitions, the sampling method, the counts per segment and the results for each one. Note where a segment is too small to support a conclusion and what was done about it. Ask the evaluation team to explain the segment choices, since a reviewer will ask why a given group was included or left out.

Threshold and alert configuration with history

Set thresholds and alerts for dataset representativeness, then keep the configuration and the alert history together. A configuration that changed after a result was seen weakens the evidence, so store each threshold with its approval date and change record. Keep a log of every alert, who reviewed it and the decision that followed, including alerts that were dismissed. Langfuse traces can record evaluation runs with inputs, outputs and metadata, which helps tie an alert to the run that triggered it.

Dataset rights, privacy and access records

The Playbook suggests respecting intellectual property and privacy rights in datasets. Keep a record of each dataset's source, license, permitted use and privacy treatment, including any anonymization, differential privacy or other privacy-enhancing technique applied under MS-2.2-003 and MS-2.2-004 of NIST AI 600-1. Restrict who can open participant records by assigning roles in BookStack, which can override permissions at shelf, book, chapter or page level. For probing results from models, garak writes JSONL reports and a hit log that can be stored under the same access rules.

What meeting MEASURE 2.2 costs in money and effort

This table lists the cost components of meeting MEASURE 2.2, who is paid for each, whether it is one-time or recurring, and what pushes it up or down. Read the one-time rows to size the initial build and the recurring rows to plan the ongoing budget. No figures are given because a quote follows from written scope.

Cost component	Who is paid or who carries the cost	One-time or recurring	What drives it up or down
Protocol and ethics review time	Internal ethics or IRB committee, legal counsel, and any external review board the organization uses	One-time per protocol, recurring when protocols are amended	Number of populations studied, sensitivity of the data, and whether generative AI prompts or outputs involve personal information
Participant compensation	The organization pays participants directly or through a research panel provider	Recurring with each evaluation round	Participant volume, number of evaluation waves, time required per session, and the rate set for each population
Consent and records tooling	Internal platform team or implementation partner for setup; licensing if a commercial consent platform is used	One-time setup, recurring hosting and maintenance	Whether consent withdrawal must propagate to stored data, number of study cohorts, and whether tools are self-hosted or licensed per seat
Engineering time for disaggregated evaluation and alerts	Internal ML engineers, TEVV staff, or an implementation partner	One-time build, recurring tuning and reruns	Number of demographic and other segments, number of representativeness thresholds, and frequency of model updates
Observability and evaluation tooling	Self-hosted open-source tools carry no license fee; hosting and staff time remain the cost	Recurring infrastructure and upkeep	Trace volume, retention period, and whether generative AI evaluations run against third-party platforms with usage-based fees
Evidence upkeep	Compliance lead, document owners, and the platform team	Recurring	Frequency of evaluation cycles, number of systems in scope, and how many evidence sources must be linked for each subcategory
External review	Independent reviewers or domain experts engaged by the organization	Recurring, usually per review cycle	Scope of the human subject study, how many populations are reviewed, and whether reviewers need access to raw participant data
Auditor or reviewer sampling	Independent auditor or assessor, whichever the organization engages	Recurring per assessment cycle	Sample sizes the assessor selects, quality of the evidence index, and how quickly records can be produced

Who does what for MEASURE 2.2

Your organization owns the evaluation itself. The people who run the AI system and answer for its outcomes must decide who is studied, why, under what consent terms and whether the results justify a go or no-go decision. MEASURE 2.2 asks that human subject protection requirements be met and that the sample represents the relevant population, and those judgments depend on your own mission, users and legal context. Expect internal ownership of the following:

Protocol and purpose: the research question, the evaluation design and the criteria for stopping or pausing a study.
Consent and compensation: the consent language, how withdrawal is handled and what participants receive, reviewed against the human subjects requirements that apply to you.
Population definition: which groups count as the relevant population and which segments must be represented or disaggregated.
Thresholds and sign-off: the representativeness thresholds you accept, and the accountable owner who records the go or no-go decision.
Review of partner output: confirming that dashboards and records reflect what was actually agreed.

Before accepting partner work, confirm that a named internal owner can explain each of these decisions without the partner in the room.

Where organizations fall short on human subject evaluations

Most gaps under MEASURE 2.2 are not missing policies. They are records that cannot be retrieved, data that does not follow the participant, and results that nobody checks against the population they claim to represent. The cards below name the gaps assessors tend to find and the fix for each.

Consent held in email

Consent given by email or chat is hard to produce months later, and it rarely shows which version of the study information the participant saw. Store each consent as a dated record linked to the participant ID, the protocol version and the consent text shown. Check that an assessor can retrieve the record for any participant in your sample without asking an engineer to search mailboxes. The Playbook suggestion on informed consent and compensation assumes this kind of traceable record.

Withdrawals that miss the dataset

A participant who withdraws consent may still be in the evaluation set, the prompt logs or a fine-tuning export because the request went to a person rather than a system. Route withdrawal requests through one workflow that removes the participant from every dataset, cache and trace store, and records the completion date. For generative AI work, NIST AI 600-1 suggests giving subjects options to withdraw or revoke consent for use of their data, so test that path end to end before you rely on it.

Convenience sample passed off as population

Teams often evaluate with whoever was easiest to recruit, such as internal staff or early pilot users, and then report results as if they describe the deployment population. Define the relevant population from the context established under MAP, then document how the sample differs from it. Where the gap is large, state which user groups the results do not cover. Assessors will ask for this comparison, so write it before they do.

Results reported only in aggregate

A single overall accuracy or satisfaction score can hide poor performance for specific demographic or user segments. The Playbook suggests disaggregated evaluation across demographic and other segments. Report results per segment with sample sizes, and flag any segment too small to support a conclusion. Keep the segment definitions in the test documentation so the same breakdown can be repeated after each model change.

Representativeness thresholds without alerts

Teams sometimes define a minimum share for each group in a dataset, then never check it after the first collection round. Set the thresholds in writing, run a check whenever the dataset changes, and send an alert to the owner when a group falls below its limit. Record each alert and the action taken. The Playbook suggests thresholds and alerts for dataset representativeness, and an unanswered alert is evidence that the control is not operating.

Participant data copied without access control

Evaluation exports often land in notebooks, shared drives or third-party annotation tools with broader access than the study approval allowed. Limit access to participant data by role, log who opened or exported it, and remove copies when the study closes. Confirm which tools hold copies before the study starts, and check that privacy-enhancing methods such as anonymization or differential privacy are applied where the protocol requires them.

Questions buyers ask about MEASURE 2.2

Is there a NIST AI RMF certification for MEASURE 2.2?

No. The NIST AI RMF has no certification, accreditation or conformity assessment program, and NIST does not validate or endorse any organization's use of the framework. Evidence for MEASURE 2.2 is normally reviewed through internal audit, customer due diligence or a voluntary independent review. Only an independent auditor or assessor can issue a report on your evaluations, and a claim of "AI RMF certified" should be treated as a red flag. Ask any vendor or consultant which independent body issued the statement they are citing. DeployOpen prepares systems, protocols and evidence for review but does not issue audit reports or certificates.

Is MEASURE 2.2 mandatory under the NIST AI RMF?

The framework itself is voluntary, so no law requires your organization to adopt MEASURE 2.2 simply because NIST published it. The obligation usually comes from elsewhere: a customer contract, a procurement questionnaire, a sector regulator or a state law that names the framework. Read the source requirement closely, because it determines what you must show. A contract may ask for documented evaluation practices, while a regulator may ask how human participants were protected. Confirm with legal counsel which requirement applies to your system and who in your organization owns that interpretation.

How much effort does MEASURE 2.2 take, and what drives it?

Effort depends mostly on whether your evaluations involve people at all and how many populations they cover. Work usually splits into a one-time setup, such as drafting consent and compensation procedures, defining representative populations and setting dataset thresholds, and recurring work, such as rerunning disaggregated evaluations after model or data changes. Effort rises when systems are used in high-stakes settings, when evaluators recruit participants across many demographic segments, or when generative AI features touch personal data. Ask for a quote that lists each evaluation, each population segment and each recurring review separately.

How long does the work take, and what should the schedule include?

The duration depends on the scope, the number of evaluations and how quickly your ethics review, legal review and participant recruitment can move. Do not assume a fixed timeline from another organization's project. Ask the implementation partner for a written scope that lists deliverables, dependencies, review gates and the owner of each approval, and check that the schedule reflects your internal approval cycles. Participant recruitment and institutional review often set the critical path, so confirm those lead times with your own teams before agreeing to dates.

Can our own team do MEASURE 2.2 without outside help?

Often yes, if you already run human-subject research or user studies and have staff who can write protocols, manage consent and analyze results by segment. Gaps usually appear in three places: independent review of protocols, statistical design for disaggregated evaluation and consistent documentation that an assessor can follow. Many organizations keep ownership of decisions and sign-off while bringing in outside help for specific work, such as building the evaluation harness, setting up logging or preparing evidence packs. Assess your capacity honestly against each sub-activity before deciding what to outsource.

What do we need to provide to a partner for this work?

A partner needs a clear picture of the evaluation program and the people who can make decisions about it. Prepare the following before kickoff:

Existing consent, privacy and research ethics procedures, plus any institutional review requirements
Definitions of the populations the system will affect and the segments you expect to compare
Access to evaluation datasets, with the privacy and intellectual property terms that govern their use
Named decision owners for protocol approval, go or no-go calls and participant compensation

Missing any of these typically delays the work more than technical setup does.

What will an assessor ask for first on MEASURE 2.2?

An assessor usually starts with the evidence trail rather than the technology. Expect requests for the documented evaluation protocol, proof that consent was obtained and that participants could withdraw, the list of populations and how they were sampled, and the disaggregated results with the thresholds used to judge representativeness. They will also ask who approved each evaluation and what changed after results came in. Keep these records in a version-controlled location from the start, because reconstructing them later is far harder than maintaining them as you go.

How DeployOpen prepares MEASURE 2.2 evidence

DeployOpen helps organizations prepare the systems, records and evaluation evidence behind MEASURE 2.2 Human subject evaluations inside their own infrastructure. The organization still owns its controls and its human subject protection decisions, and DeployOpen does not issue any assessment report or certificate.

How we scope

The written scope names which human subject evaluations, AI systems and populations fall under MEASURE 2.2.
The scope states which consent, compensation and withdrawal processes the organization owns and which DeployOpen will document or configure.
Engagements are fixed-scope, so deliverables and timelines are agreed in writing before work starts.

How we build

We set up self-hosted tooling that records evaluation runs, participant segments and representativeness checks in the client's environment.
Traces and version histories are configured so changes to evaluation procedures and consent materials stay traceable over time.
Role-based access is configured so only authorized reviewers can view evaluation records that contain participant-linked data.

How we hand over

Handover includes documented evaluation procedures, consent and withdrawal workflows, and disaggregated results ready for internal review.
Evidence is packaged so the organization's own assessor or auditor can trace each claim back to a dated record.
Documentation states which items the organization must confirm with its ethics board, legal counsel and independent assessor.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after the initial handover.
Retainers can cover updates to evaluation records when the population, the system or the applicable requirements change.
Support requests are handled against the written scope, so changes to the evaluation approach are agreed before they are implemented.