MEASURE 1.2 Assessing metric appropriateness and controls

What MEASURE 1.2 asks for and why metrics go stale

MEASURE 1.2 asks an organization to keep checking whether its AI metrics still measure the right risks and whether its controls still work. Picture a system that went live with a set of accuracy, toxicity or drift metrics chosen at launch. Months later, the users, the data, the model version and the business process may all have changed, yet the dashboard still shows the original numbers and nobody has asked whether they still mean anything. The subcategory, listed in Table 3 of NIST AI 100-1, describes that recurring review. It covers the appropriateness of the metrics, the effectiveness of existing controls, and reports of errors along with their potential impacts on affected communities. NIST AI 100-1 is voluntary, and no one certifies conformance to it. Its Playbook suggestions are not a checklist, so the organization decides how often the review runs, who runs it and what it must produce.

In practice, the review tends to stall for three reasons. It usually has no named owner, because the team that built the metrics is not the team that should judge them, and the risk function often assumes someone else is watching. It usually has no cadence, so it happens only after an incident. And it is rarely connected to incident data, support tickets, user complaints or software quality records, even though the Playbook points to measures such as bug rates, severity and time to repair as inputs. Before the next review, confirm three things: which role signs off on metric changes, which triggers force an out-of-cycle review, and which incident sources feed the metric register. Those answers matter more than any template, and they are the part an outside assessor will ask about first.

Which systems, data and teams fall under the metric review

Before any metric or control is reviewed, the organization has to decide what sits inside the review. The decisions below set the population of AI systems, the sources feeding each metric, the teams that own each review, and how affected communities take part.

AI systems and metrics in scope

Start with the inventory of AI systems that GOVERN 1 and MAP 1 already document, then decide which ones carry metrics that MEASURE 1.2 must review. A system with a high-impact use or a costly error belongs in scope even if it is only a small internal tool. Record which metrics apply to each system, so that a metric retired from one system is not silently dropped from another. Ask whether the scope list is updated when a system changes purpose, not only when it is first deployed.

Third-party models and data feeds

Decide whether pre-trained models, hosted APIs, vendor datasets and vendor-supplied evaluations feed any metric. If they do, the review must cover their behavior as well as your own, because a metric built on a vendor's output can drift when the vendor changes its model. The Playbook suggests monitoring third-party inputs as part of this subcategory. Confirm which vendor updates trigger a metric review and whether the contract gives you the notice needed to act on them.

TEVV ownership of metric reviews

Assign the test, evaluation, verification and validation team as the owner of the recurring review of metric appropriateness and control effectiveness. The team should be able to test whether a metric still measures what it was chosen to measure, and to compare results across releases. Keep the owner separate from the people who built the metric where your risk tolerance requires it. Check that the TEVV team has documented access to the production data and logs it needs to run the review.

Impact assessment and deployment reviews

Impact assessment staff and deployment owners each see different failures. Impact assessors look for effects on people and groups that the original metrics did not capture, while deployment owners see operational errors, user workarounds and changes in input data. Agree who receives each kind of finding and who has authority to change a metric or pause a control. Confirm that both groups sign off on the review record, so the outcome is not decided in one team alone.

Development teams and their limits

Development teams hold the knowledge of how a metric was built and what it assumes, so their input is needed in every review. They should not be the only reviewers, because they tend to assess their own metrics as sound. Define which development changes, such as a new training set or a new prompt template, automatically reopen the metric review. Ask the team to document known limits of each metric so reviewers can test them.

Affected communities in the review

Decide how people affected by the system take part in the review, and which reports they can file about errors or harm. Options include structured feedback channels, panels of affected users and documented complaint routes that feed into the metric review. The reviews should show how community reports changed a metric or control, or why they did not. Confirm with legal and privacy staff what consent, notice and data handling rules apply before collecting this input.

How an assessor tests MEASURE 1.2

This table sets out the steps an independent assessor typically takes for MEASURE 1.2, the evidence each step requests and the findings that often appear. Read it as a guide to what your own evidence file must show, since the assessor tests your records and does not certify the AI system.

Test step	What the assessor does	Evidence requested	Common finding
Scope the metric population	Lists the AI metrics and control effectiveness measures in use for each system in scope, then compares them with the risks mapped under MAP.	Metric register, mapped risk list, system inventory entries for the systems in scope.	Metrics exist but no record links them to a specific mapped risk.
Sample metrics for appropriateness	Selects a sample of metrics and asks whether each still measures what it was chosen to measure, including external validity.	Metric definitions, validity rationale, dated reviews of each sampled metric.	Metric definitions unchanged since launch with no documented review of whether they still fit the system.
Review revision history	Reads version history for sampled metrics and controls to confirm changes follow a documented review trigger.	Change logs, version tags, approval records for metric and threshold changes.	Metrics never revised after deployment, even after model, data or use-case changes.
Check lifecycle coverage of control reviews	Confirms control effectiveness reviews occurred at planned intervals and after significant events across the lifecycle.	Review calendar, review minutes, test results for each control, records of event-triggered reviews.	Reviews documented for launch only, with no later checks in production.
Trace error and impact reports to metrics	Follows a sample of reported errors and impact concerns to see whether they led to metric or control changes.	Incident and error tickets, root cause notes, linked metric or control change requests.	Error reports are closed without any link to metric changes or new measures.
Test feedback from affected communities	Checks how reports from users and affected individuals are collected, assessed and fed back into metric review.	Feedback intake records, triage decisions, evidence of changes or documented reasons for no change.	Feedback channels exist but no adjudication or follow-up is recorded.
Check software quality metrics	Reviews whether defect and response data for the AI system are collected and used in control effectiveness decisions.	Bug rate and severity reports, time to response and time to repair records, related review notes.	Quality data is gathered by engineering but never reviewed by the team that owns AI risk.
Confirm third-party inputs are monitored	Verifies that changes from vendors, pre-trained models and data sources are reflected in metric and control reviews.	Vendor change notices, third-party review records, impact assessments of upstream changes.	Upstream model updates deployed without a corresponding metric review.

Steps to build a recurring metric and control review

This sequence turns MEASURE 1.2 from a one-time check into a standing review that keeps AI metrics matched to the risks they track and confirms that existing controls still work. Each step produces a record that an internal reviewer or independent assessor can inspect.

Inventory metrics and controls tied to mapped risks
Start from the risks recorded during MAP and list every metric and control that the measurement program uses to track them. Assign each item an owner from the team that built or operates the system, and note the trustworthiness characteristic it supports. The output is a register that links each metric and control to one or more mapped risks, so a reviewer can see which risks have no measurement behind them. Confirm the register is complete by checking it against the AI inventory described under GOVERN 1.
Set review triggers and cadence
Decide which events force a review, such as a model or prompt change, a new data source, a change in deployment context, or a rise in reported errors, and set a regular review interval for the rest. Record the cadence and triggers in the review procedure, and name the role accountable for starting each review. The output is an approved procedure with the trigger list and interval, which lets the organization show that reviews happen on a defined schedule rather than when someone remembers.
Route error and community impact reports into review
Connect the channels that carry error reports, end user complaints and reports of impact on affected communities to the review queue. Include software quality measures such as bug rates, severity, time to response and time to repair, since the Playbook suggests tracking them. Each report should be logged with its date, the metric or control it relates to, and a triage decision. The output is a log showing that every report was received, classified and either linked to a review item or closed with a reason.
Assess external validity and third-party inputs
For each metric, ask whether it still measures the real-world condition it was designed for, using the operating population, data and context as they are now. Review the third-party models, datasets and monitoring tools that feed the metrics, and confirm the vendor has not changed them without notice. Document the assumptions that were tested and any gaps found. The output is a validity note for each metric and a list of third-party dependencies with their current status, which the organization can share with a reviewer during an assessment.
Add new metrics where existing ones fall short
When the register, reports or validity checks show that a risk has no adequate measure, draft a new metric or test method. Define what it measures, how it is calculated, what threshold triggers action, and who reviews its results. Test the metric against known good and known bad cases before relying on it, and document any risk that still cannot be measured so it is recorded as a limitation under MEASURE 1.1. The output is a proposed metric specification with a test record and an approval decision.
Record review decisions and update the register
Close each review with a written decision for every metric and control reviewed: keep, adjust, replace, or retire, with the reasons and the evidence used. Update the register, the trigger list and the metric specifications to match the decisions, and carry any open items into the next cycle. The output is a dated review record that shows what was checked, what changed and who approved it. Ask an independent assessor, under MEASURE 1.3, to sample these records during a periodic check.

What evidence shows metrics and controls are still appropriate

An assessor separates a living review from a one-time sign-off by asking for dated records that repeat over the life of the system. MEASURE 1.2 expects the appropriateness of AI metrics and the effectiveness of existing controls to be assessed and updated on a regular basis, so a single approval memo from launch day does not show the outcome. The strongest evidence is a revision history for each metric definition: the version number, the date it changed, who approved the change, the reason, and the review that triggered it. Ask your team to show a metric whose threshold or data source changed at least once since deployment. If every metric still carries its launch-day definition, the reviews either did not happen or did not change anything, and an assessor will ask which.

Review records should tie each metric and control to a date, a reviewer and an outcome. Look for entries that name the control or metric, the evidence reviewed, whether the metric was kept, changed, or retired, and the follow-up owner. Error reports are the second half of the proof. Each report of a wrong output, a harmful result or a complaint from an affected individual should link to the metric or control it tested and to the review that considered it. Software quality measures, such as bug rates, defect severity, time to response and time to repair, give the review a measurable baseline. An assessor will check whether these figures were pulled on a schedule and whether a rising trend led to a documented decision.

The most convincing evidence is a record that a metric was changed or retired because a review found it no longer worked. Examples include a retired accuracy measure that did not reflect real user harm, or a new metric added after error reports showed a gap the existing measures missed. Illustrative only: a team reviews a quarterly report, finds that its error metric ignored a category of complaints, adds a measure for that category, and logs the change with the date and the reason. Confirm with your assessor which review cadence and record format they expect for your risk tolerance, since the framework does not set one. Storing these records in a system that keeps history and timestamps, instead of a shared document that gets overwritten, makes the living review easy to prove.

Evidence artifacts that satisfy MEASURE 1.2

MEASURE 1.2 asks whether the organization regularly checks that its AI metrics still fit the system and that its controls still work, including errors reported by affected communities. The artifacts below are the records that show those checks happened, who made the decisions, and what changed as a result.

Metric register with version history

A register listing every metric in use, its definition, the risk it measures from MAP, its data source, its owner and its threshold. Each change to a definition or threshold should carry a version number, a date and the reason for the change. An assessor will check whether the register matches what the monitoring tools actually compute. Keep it current by reviewing it at each model, data or scope change and on a fixed review calendar set in the governance policy.

Control effectiveness review records

Dated records showing that each control placed on an AI risk, such as access restrictions, human review steps or output filters, was tested for whether it still works. Each record names the tester, the test method, the result and any corrective action. Testers should not be the people who built the control. Keep the records current by logging each test run, including failed tests, rather than only the scheduled reviews that passed.

Error and impact report log

A log of reported errors, complaints and potential impacts on affected individuals and communities, with the source of each report, its severity, the metrics it touched and the decision taken. Each entry should link to the metric or control change it triggered, or record why no change was made. Keep the log open to reports from outside the AI team and review it on a set cadence so that unresolved entries are visible.

Software quality metric dashboards

Dashboards and exports that track the software quality measures the Playbook suggests for this subcategory, including bug rates, bug severity, time to response and time to repair. Confirm that each metric has a defined source and that the dashboard keeps historical values, since an assessor will look for trends rather than a single snapshot. Keep them current by setting automatic exports at a fixed interval and retaining the raw data.

Third-party input monitoring records

Records showing that inputs from vendor models, datasets and APIs are watched for changes that could affect metric validity, such as version updates, data drift notices or changed terms. Each record notes the input, the change observed, whether the metrics were rechecked and who signed off. Keep these current by assigning a named owner for each third-party input and logging every vendor release note reviewed, including those that required no action.

New-metric approval records

Approval records for metrics added when existing ones proved insufficient, showing the gap that prompted the new metric, its validity rationale, the reviewers and the approval date. Confirm that the approval came from someone who did not design the metric, and that the metric was added to the register. Keep the records current by refusing to put a metric into production reporting until its approval record exists.

What meeting MEASURE 1.2 costs in money and effort

This table lists the cost components of keeping AI metrics appropriate and controls effective under MEASURE 1.2, who is paid for each, and whether it is one-time or recurring. Use it to build a quote request, since no price or duration for this work is published in the framework itself.

Cost component	Who is paid or who spends time	One-time or recurring	What drives it up or down
Tooling and licenses	Software vendors for commercial evaluation, monitoring or audit platforms; open-source tools carry no license fee	Recurring for subscriptions, one-time for setup	Number of AI systems in scope and whether tools are commercial or self-hosted open source, such as garak or Langfuse, which are published under permissive licenses
Engineering time to instrument metrics	Internal platform or ML engineers, or an implementation partner under a fixed-scope agreement	One-time for initial instrumentation, recurring for changes	Count of metrics to add, how many data sources must be connected, and whether tracing already exists
Reviewer time from TEVV and impact assessment staff	Internal staff, who are often not front-line developers	Recurring, tied to review cycles	Review frequency, number of systems per reviewer, and how often metrics are found inappropriate and must be redesigned
Software quality metric collection	Engineering and support teams tracking bug rates, severity, time to response and time to repair	Recurring	Volume of incidents and whether ticketing data is already structured enough to report on without manual effort
Evidence upkeep	Compliance or GRC staff, sometimes supported by a GRC platform	Recurring	Number of controls and metrics, change frequency, and how much of the evidence is generated automatically versus written by hand
External review and expert consultation	Independent reviewers or domain experts engaged for specific reviews	Recurring or per engagement	Risk tolerance, the number of high-impact systems, and whether reviews are scoped in writing before work begins
Assessor time for independent assessment	An independent auditor or assessor, since NIST does not certify AI RMF use	Per assessment cycle	Scope of systems and controls covered, the quality of evidence provided, and the number of follow-up requests raised
Third-party dependency monitoring	Procurement, vendor management and engineering staff tracking external inputs	Recurring	Number of third-party models, datasets or APIs feeding metrics, and how often those vendors change their outputs

Who owns the metric review, who can help, and where the auditor stops

Your organization owns the MEASURE 1.2 outcome. The people accountable for the AI system decide which metrics are still appropriate, when a control has stopped working, and whether a reported error or a possible impact on affected communities justifies changing the system, its metrics or its controls. The NIST AI RMF Playbook suggests reviewing metric and control effectiveness across the lifecycle, so this review needs a standing owner and a schedule, not a one-time exercise.

Metric decisions: Decide which metrics stay, which are retired and when new metrics are needed because existing ones no longer capture the risk.
Error and impact intake: Own the channels that receive reports of errors and potential impacts, and record how each report was triaged.
Affected community engagement: Decide how feedback from affected individuals and communities reaches the review, in line with your risk tolerance.
Third-party inputs: Monitor vendor models, data and software for changes that affect your metrics.
Sign-off records: Keep dated minutes showing who reviewed each metric and control, what changed and why.

Check that each metric has a named owner and that the review cycle is written into your governance documents.

Where organizations fall short on metric and control reviews

MEASURE 1.2 asks whether the metrics and controls an organization relies on still fit the system and its risks. The gaps below are where that review usually breaks down, and each one has a closing action that an internal team or an implementation partner can put in place.

Metrics chosen at launch

Many teams pick accuracy, latency or refusal rates before launch and never revisit them. A metric that suited the original use case can stop reflecting real harm once users, data or the task change. Check each metric against the current intended purpose, and record the date and reason for its last review. Close the gap by assigning a named owner to each metric and setting a review point tied to each release, not a calendar date alone.

Error reports that stall

Errors reported by support staff, users or internal testers often sit in a ticket queue and never reach the person who owns the metric. The result is a dashboard that stays green while the same failure keeps recurring. Route every error report to the metric owner, and log whether the report led to a new test, a changed threshold or a documented decision to accept the risk.

No review after change

A new model version, a retrained classifier, a changed retrieval index or a shift in input data can all make earlier control results invalid. Many organizations treat these as routine releases and skip the metric review. Set a trigger list that includes model, prompt, data source and tool changes, and require a documented check of metric and control effectiveness before each change goes live.

Third-party inputs unwatched

Pre-trained models, hosted APIs, vendor safety filters and open-source libraries can change behavior without notice. When these inputs are not monitored, the organization's metrics may measure the wrong thing after an upstream update. Keep a register of third-party components with their versions, subscribe to vendor change notices, and re-run a defined set of tests when a component changes.

No record of community impact

Reports from affected individuals or communities about harms, such as wrongful denials or biased outputs, are often handled as customer complaints and not as inputs to control review. Without a record, no one can show that potential impacts were weighed when the controls were judged effective. Create a log of impact reports with the affected group, the alleged harm, the metric or control involved, and the outcome of the review.

Questions buyers ask about MEASURE 1.2

How much effort does a recurring MEASURE 1.2 review take?

The effort depends on how many AI systems are in scope, how often they change, and how many metrics and controls each one depends on. NIST's Playbook suggests reviewing metric and control effectiveness throughout the lifecycle, so the work is recurring rather than a one-time exercise. Each cycle usually includes checking whether current metrics still reflect the risks mapped under MAP, reviewing error and incident reports, and recording decisions about changing or retiring metrics. A system that is retrained often or serves a growing user base will need more frequent cycles than a stable internal tool. When requesting a quote, ask for the review cadence, the number of systems covered, and which roles attend each cycle, and confirm whether the implementation partner will also build the dashboards that feed the review.

Can we run MEASURE 1.2 reviews in-house?

Yes, many organizations run these reviews internally, provided they have people who can judge whether a metric still measures what matters. The organization owns its AI risk decisions and its controls, so it is responsible for the review whether or not an outside party helps. Internal teams usually handle the recurring cycle well once the metrics, data sources and review records exist. Outside help tends to be most useful for the initial setup, for building the instrumentation that collects metrics, and for testing whether the review process works as written. Confirm with your own risk owners which decisions must stay internal, and ask any partner to document a handover so that the review can continue without them.

What does the compliance team need from engineering for this review?

Compliance needs a clear record of which metrics exist, why each was chosen, how often it is checked, and what happened when it was found inadequate. Engineering needs to supply the raw data, the logic behind each metric, and the change history for models, prompts and controls. In practice, engineering should provide access to logs and traces, documented definitions of each metric, and tickets showing how errors and user reports were handled. Compliance should define the review questions, the sign-off roles and the retention period for records. Both teams should agree on a single register of metrics and controls so that each review compares the same items over time. Ask a partner to show how that register and its supporting logs would be produced in your environment.

How long does setup take, and what makes it longer?

Setup duration depends mainly on the state of your existing monitoring, the number of AI systems in scope, and how well documented the current metrics are. Organizations that already log inputs, outputs and errors usually need less time than those that must add instrumentation first. Other drivers include the number of data sources, whether metrics must be defined for generative systems that need different measures, and how many approvals the review process requires. Because no duration is fixed by NIST, you should ask for a written schedule as part of a fixed-scope proposal, with milestones for the metric register, instrumentation, first review cycle and handover. Compare any estimate against the scope items it depends on.

Will an assessor accept our MEASURE 1.2 evidence?

An independent assessor decides whether the evidence is sufficient for your purpose, and the answer depends on whether the records show a real review process rather than a single document. Assessors typically look for a defined metric register, dated review records, evidence that metrics changed when they stopped being fit for purpose, and records of errors and their potential impacts on affected communities. Logs and traces from tools such as Langfuse can support those records, but they only count as evidence when they are tied to a documented decision. Confirm the assessment criteria and scope with the assessor before work begins, because no certification body exists for the AI RMF and acceptance depends on the engagement terms.

Does NIST certify organizations against MEASURE 1.2 or the AI RMF?

No. NIST does not certify, accredit or validate any organization or its use of the AI RMF, and it does not operate a conformity assessment program for it. NIST describes the framework as voluntary, so any statement that an organization is "NIST certified" for MEASURE 1.2 should be treated with caution. What organizations can obtain is an independent assessment or review performed by an outside assessor, whose report describes the scope and findings. Ask any vendor or consultant to explain precisely who issues the report and what it covers, and confirm that the assessor, not the implementation partner, is the party making that statement. DeployOpen prepares systems, controls and evidence but does not issue audit reports or certifications.

Which open-source tools can produce evidence for these reviews?

Several open-source tools can generate records that feed a MEASURE 1.2 review, although none of them performs the review on its own. Langfuse, licensed under MIT, captures traces of each request, including LLM calls, retrieval steps, tool executions, timing, inputs, outputs, token usage and metadata, which can support error analysis and trend reviews. garak, licensed under Apache-2.0, probes language models for issues such as hallucination, data leakage, prompt injection and toxicity, and writes a JSONL report and a hit log that can document testing results. CISO Assistant, licensed under AGPL-3.0-only, can track assessments and control evidence across frameworks. Confirm each tool's capabilities against your deployment before relying on it, and ask any partner to explain how the outputs are linked to documented decisions.

How DeployOpen prepares metric review evidence

DeployOpen helps organizations set up the systems, metric records and control review records that MEASURE 1.2 asks them to keep, inside their own infrastructure. The organization owns its metrics and controls, and only an independent assessor can issue an assessment opinion.

How we scope

Fixed-scope engagements list which AI systems, metrics and controls are covered before any work starts.
The written scope names the review cadence, the data sources for each metric and who in the organization signs off on changes.
The scope records which MEASURE 1.2 suggested actions the organization adopts and which it declines, with reasons.

How we build

We configure tracing and logging for the AI application so that inputs, outputs, errors and tool calls can be reviewed later.
We set up probe and test reports, such as garak JSONL output, so each evaluation attempt and finding is retained.
We connect defect, bug severity and time-to-repair data from the existing ticketing system to the metric review record.

How we hand over

Handover includes a metric register with definitions, owners, thresholds and the date each metric was last reviewed.
A control effectiveness log records each review, its findings, the change decided and the evidence attached.
A reporting procedure documents how errors and potential impacts on affected communities are logged, triaged and escalated, with runbooks for the team.

How we support

Ongoing support and continuous testing are available as retainers after the initial build is handed over.
Retainers can cover periodic metric reviews, tooling upgrades and refreshed test runs when the AI system changes.
Before an assessment, we help the organization prepare evidence packs for its independent assessor, who reviews them and issues any report.