NIST AI RMF MEASURE 1.1: metric selection and evidence

What NIST AI RMF MEASURE 1.1 asks you to do before you pick a metric

MEASURE 1.1 asks an organization to select the approaches and metrics it will use to measure the AI risks already listed in the MAP function, starting with the most significant ones, and to document any risk or trustworthiness characteristic it will not or cannot measure. In practice, the outcome has two halves. The first is a ranked, justified set of metrics tied to specific mapped risks. The second is a written record of what is left unmeasured and why. Both halves need to exist before a team builds dashboards or runs tests, because the choice of metric decides what the organization will later be able to see. The reader should start by pulling the MAP outputs for the system in scope and ranking each risk by significance, so that measurement effort follows the risks that matter most. The framework is voluntary. NIST's AI RMF FAQ says neither private nor public sector organizations are required to use it, and there is no AI RMF certification. NIST does not validate or endorse any organization's use of the framework, and no conformity assessment program exists. This means no external body will issue a MEASURE 1.1 certificate. What an assessor, customer or regulator will test is whether your metric selection is traceable to your risks and your documented reasoning. Any claim that a system is "NIST AI RMF certified" should be treated as a red flag and questioned. Teams often struggle here for three reasons: mapped risks are described in qualitative terms that do not translate directly into numbers, the same metric is reused across systems without asking whether it fits, and the team never records the metrics it considered and rejected. The NIST AI RMF Playbook suggests documenting selection criteria, including metrics considered but not used, and justifying every mapped risk that will not be measured. Before selecting, confirm with the system owner which risks are in scope, who signs off on the risk ranking, and which domain experts will review whether each metric reflects real harm. Those decisions, recorded in advance, are what make the later choices defensible.

Which AI systems, risks and teams fall under NIST AI RMF MEASURE 1.1

NIST AI RMF is voluntary, so the organization decides where MEASURE 1.1 applies. The decisions below set the boundary of the work: which systems are measured, which MAP outputs drive metric selection, which generative AI risks are in play, and who must contribute.

Systems and components in scope

Start by listing every AI system and each component it depends on, including pre-trained models, third-party APIs, retrieval sources and fine-tuning data. MEASURE 1.1 asks which approaches and metrics are selected for implementation, so a component left off the inventory is one that will never be measured. Confirm the boundary with the system owner and record which components are owned in-house and which are supplied by vendors.

MAP outputs that drive selection

Metric selection starts from the risks enumerated in MAP, not from whatever metrics a tool offers by default. Take the MAP 5 impact characterization, with its likelihood and magnitude estimates, and rank the risks so the most significant are measured first. Check that each high-ranked risk has at least one candidate metric, or a documented reason it has none.

Generative AI risks from NIST AI 600-1

Systems that generate content need the risk categories in the Generative AI Profile, not only the core framework. NIST AI 600-1 ties its suggested MEASURE 1.1 actions to risks such as CBRN information or capabilities, harmful bias and homogenization, information integrity, and human-AI configuration. Decide which of these apply to each system, then record which can be measured quantitatively and which must be tracked qualitatively with the reasons.

AI Development team contributions

The AI Development group, tagged to MEASURE 1.1 in the Playbook, owns the model design choices that determine what can be measured. Its members should explain training data limits, model assumptions and known failure modes, which are the inputs needed to choose metrics that fit the design. Ask this group to propose candidate metrics and the performance limits they believe are reachable.

TEVV and domain expert input

Test, evaluation, verification and validation staff turn metric choices into procedures that can be run and repeated. Domain experts confirm whether a metric reflects real harm in the setting where the system operates. Both groups should sign off on the metric list, including the metrics considered but rejected and the reasons, because that record is what a later reviewer will ask for first.

Documented unmeasured risks

Some risks cannot be measured with available methods, and MEASURE 1.1 requires those gaps to be written down rather than hidden. Set up a register for each system that lists every mapped risk or trustworthiness characteristic without a metric, the reason, and the interim control that covers it. Ask the team to review this register at the same cadence as the metric review so gaps do not stay open silently.

How an assessor tests MEASURE 1.1 and what evidence they request

This table walks through each artifact an independent assessor would review for MEASURE 1.1, what they check in it, and where findings usually arise. Read it as a pre-assessment checklist: each row names the evidence to have ready and the gap that tends to draw a finding. The assessor issues any findings, and DeployOpen prepares the records but never issues a report or attestation.

Evidence item	What the assessor checks	Common finding
MAP risk register	Each risk named in MAP is listed with its context, likelihood and magnitude, and the ranking used to decide which risks are most significant	Risk ranking is missing or undated, so the assessor cannot confirm measurement started with the most significant risks
Metric selection criteria	Written rules explain why each chosen metric fits a specific mapped risk and the trustworthiness characteristic it serves	Metrics are listed without any stated link to a mapped risk
Metrics considered but not used	A record shows alternatives that were evaluated, with the reason each was dropped	No record exists, so the assessor cannot tell whether selection was deliberate
Acceptable performance limits	Thresholds are defined for each selected metric, with the course-correction step triggered when a limit is crossed	Limits are absent, set after results were known, or have no defined response
Register of unmeasured risks	Each mapped risk or trustworthiness characteristic not measured is listed, with the reason it cannot be measured and any compensating review	Unmeasured risks have no entry or no justification
Test procedures and results	Test plans show the system is checked for fitness for purpose, and results match the metrics and limits documented	Metrics are reported but no test procedure ties them to the system's intended use
Ownership and sign-off	Named roles, such as AI development, testing and evaluation staff and domain experts, approved the metric set and its updates	Approvals are unsigned or the owners cannot be identified

How to implement MEASURE 1.1 step by step

This sequence turns the MAP risk list into a documented set of metrics, performance limits and justified gaps, so that an assessor can see which risks are measured, how, and why others are not. Each step produces a record that stands on its own.

Confirm the prioritized MAP risk list
Start from the risks recorded under MAP 1 through MAP 5, including impact characterization for likelihood and magnitude. The AI development team and domain experts agree the ranking, with the most significant risks at the top, because MEASURE 1.1 asks that measurement begin there. The output is a signed-off risk register with a priority order and the owner of each risk. The step is done when every mapped risk has a priority and an owner, and nothing is left unranked.
Define fit-for-purpose testing procedures
For each top-ranked risk, the development team and TEVV staff write the test procedure that shows whether the system is fit for its intended purpose and trustworthy in its deployment context. Domain experts review that the tests reflect real use, not only benchmark conditions. The output is a test procedure record per risk, naming the test data, the environment, the pass conditions and the person who runs it. The step is done when each procedure can be repeated by someone other than its author.
Select metrics and tooling for each risk
Choose the metrics that the test procedures will produce and the tools that will collect them. For generative AI systems, the Generative AI Profile suggests tracking risks that cannot be measured quantitatively, so teams should also decide where structured human feedback such as red-teaming is most useful. Open-source tools such as garak can produce attempt-level reports for language model probes, and tracing tools such as Langfuse can record inputs, outputs and token usage. Confirm what each tool actually measures before relying on it. The output is a metric selection record that links each metric to its risk and its data source.
Set acceptable limits and course correction triggers
For each selected metric, the risk owner and the AI development team set an acceptable performance limit and state what happens when the system crosses it. The trigger should name the action, such as pausing a release, adding a human review step, or rolling back a model version, and the person authorized to take it. Accountable executives or the governance body approve the limits, since they set risk tolerance. The output is a limits and triggers table reviewed against the risk tolerance recorded under GOVERN 1 and MAP 1. The step is done when every metric has a limit and an action.
Document metric selection criteria and rejected metrics
Write down why each metric was chosen, including the criteria used, such as validity for the risk, availability of data, and cost of collection. Equally important, record the metrics that were considered and not used, with the reason for each rejection. Assessors look for this record because it shows the selection was reasoned rather than convenient. The output is a metric selection rationale document, versioned and stored with the risk register. The step is done when a reviewer who did not take part can explain why each metric is in or out.
Record justifications for unmeasured risks
Some mapped risks or trustworthiness characteristics cannot be measured with current methods. For each one, the team writes a register entry that states the risk, the reason it cannot be measured, the compensating controls or monitoring that cover it in the meantime, and the owner who will revisit the decision. The register is reviewed with the governance body, because accepting an unmeasured risk is a risk decision. The output is an unmeasured risk register. The step is done when every mapped risk is either linked to a metric or recorded here with a justification.
Hand the records to the next review cycle
Package the risk register, test procedures, metric selection record, limits and triggers, rationales and unmeasured risk register as one version-controlled set. Confirm the owner of each record and the date of the next review, which MEASURE 1.2 and MEASURE 1.3 then take over for effectiveness checks and independent assessment. Ask whether the organization's own review cadence is documented, since no external schedule is set by the framework. The output is an approved MEASURE 1.1 package. The step is done when the package can be produced to an assessor on request.

Which evidence satisfies MEASURE 1.1 and how to keep it current

MEASURE 1.1 is satisfied by a set of linked records rather than a single document: the risks chosen for measurement, the metrics that measure them, and the risks left unmeasured with reasons. Each artifact below should be reviewed on a schedule and whenever the system, its data or its use changes. The Playbook suggests what to record, but it is a suggestion source, not a checklist, so your own risk tolerance should set the depth.

Prioritized risk-to-metric map

This map links each risk enumerated in MAP to the metric or test that measures it, ordered by significance so the most serious risks are measured first. Assessors check that the ordering reflects a documented significance rating, not whatever the team found easiest to test. Refresh it after each MAP update, after material changes to the system's context, and when a new incident or user report changes the risk ranking. Ask the owner to record who approved the ordering and when.

Metric definitions with thresholds

Each metric needs a written definition, the data it uses, how it is calculated, and the acceptable performance limit that triggers course correction. A threshold without a stated response action is weak evidence, so pair every limit with a named owner and the action taken when it is breached. Review definitions when training data, prompts, model versions or user populations change, because a threshold set for one population may not hold for another.

Metrics-considered log

This log records candidate metrics that were evaluated but not adopted, with the reason for each decision. It shows that selection was deliberate and helps explain why a commonly expected measure is absent. Keep entries even when a candidate was rejected for cost or data availability. Update the log whenever a metric is retired or replaced, so the reasons stay attached to the decision that caused them.

Unmeasured risk register

Any mapped risk or trustworthiness characteristic that cannot be measured must appear here, with the reason it cannot be measured and the compensating controls or monitoring that cover it. Vague entries such as no suitable tool do not satisfy the outcome; state what data, method or expertise is missing and who will close the gap. Reassess each entry at every review cycle, because a new method or data source may make a previously unmeasurable risk testable.

Test procedure records

Test procedure records document how each metric was produced: the test dataset, the environment, the version of the system under test, the results and the person who reviewed them. They turn a metric into reproducible evidence. Store them with the system version identifier so results can be traced after a model or prompt change. Re-run and re-record tests when the system changes materially, and keep prior records rather than overwriting them.

Red-teaming scope records for MS-1.1-008 and MS-1.1-009

For generative AI, NIST AI 600-1 suggests tracking risks that cannot be measured quantitatively, with reasons, and defining where structured human feedback such as red-teaming is most useful. The scope record names the risks targeted, the attack categories, the testers and their independence from the build team, the tools used, and the findings routed to the unmeasured register. Revise the scope after each significant model, prompt or tool change, and after any incident that exposes a gap the previous scope missed.

What meeting MEASURE 1.1 costs in money and effort

This table breaks down the cost components of selecting and documenting risk metrics under MEASURE 1.1, who is paid for each, and whether it is one-time or recurring. Use it to shape a written scope and to ask each provider for a line-item quote, since NIST publishes no prices or durations for this work.

Cost component	Who is paid	One-time or recurring	What drives it up or down
Metric selection and documentation	Internal AI, risk and product owners, or an implementation partner	One-time, then revised when the system or risk map changes	Number of mapped risks, number of models, and how many metrics were considered before selection
Engineering time to define and instrument metrics	Internal platform and ML engineers, or a partner preparing the telemetry	One-time build, with recurring changes as metrics evolve	Logging maturity, number of pipelines, and whether metrics already exist in current tooling
Tooling and licenses	Commercial vendors for evaluation or observability platforms	Recurring, usually by seat, volume or deployment	Choice of commercial versus self-hosted tools, data volume and retention needs
Open-source tooling in self-hosted form	Infrastructure and staff time to run it, with no license fee for the software itself	One-time setup, then recurring hosting and upkeep	Hosting footprint, upgrade cadence and the skills available internally. Confirm licenses such as Apache-2.0, MIT and AGPL-3.0-only with counsel before use
Generative AI test and red-team exercises	Internal red teams, testing firms or a DeployOpen-style security testing engagement	Recurring when run per release or per scope change	Generative AI scope, number of GAI risks targeted, and whether structured human feedback is scheduled
Evidence upkeep	Internal compliance and engineering staff	Recurring, tied to each review cycle and model update	Frequency of model changes, retention of metric reports, and how well evidence is indexed
Independent assessor time	External auditors, assessors or consultants you engage	Recurring if you commission periodic reviews	Scope of review, number of metrics assessed and readiness of documentation before the assessor arrives
Documentation of unmeasured risks	Internal risk owners, with counsel where legal exposure is involved	Recurring review, one-time initial write-up	Number of risks that cannot be measured quantitatively and how often those decisions are revisited

Which self-hosted open-source tools can produce MEASURE 1.1 evidence

MEASURE 1.1 asks for selected metrics and documented gaps, and each tool below produces a different kind of record for that work. Testing output, runtime traces and framework tracking answer different assessor questions, so a program usually needs more than one. Confirm each tool's license terms and operational fit before adopting it.

Probing language models with garak

garak (Apache-2.0) probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. Its JSONL report records every attempt and evaluation, and its hit log lists the attempts that found a vulnerability. For MEASURE 1.1, these files show which generative AI risks were tested, which probes were chosen, and where results crossed the acceptable limits you set. Check that the probes you run match the risks named in your MAP records, and that the test scope covers your deployed model configuration.

Tracing runtime behavior with Langfuse

Langfuse (MIT) records traces of each request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. For MEASURE 1.1, traces give the measurement evidence that a metric was computed on real production or staging behavior, not on a single test prompt. Confirm that your data retention and redaction settings meet your privacy obligations, because traces can capture personal data in inputs and outputs.

Tracking frameworks with CISO Assistant

CISO Assistant (AGPL-3.0-only) ships more than 200 frameworks, including NIST-aligned and EU frameworks, and runs assessments against them with automatic mapping between frameworks. It can hold your metric selections, owners and assessment status in one register. Its mapping does not replace your own MAP and MEASURE records, because a framework link does not show that a risk was analyzed for your system. Confirm that a tool-generated mapping is reviewed and signed off by your team before you rely on it in an assessment.

Checking license fit before adoption

Licenses differ across these tools: garak is Apache-2.0, Langfuse is MIT and CISO Assistant is AGPL-3.0-only. Legal teams should confirm whether each license permits your intended use, including internal modification, hosting for users outside the organization and redistribution in products. The AGPL terms can apply when software is offered over a network, so a hosted service built on CISO Assistant needs a specific review. Record the license review as part of your AI inventory.

Confirming environment fit and gaps

Each tool has operating requirements, data flows and integration points that depend on your environment. Confirm how each tool reaches your models and data, where its logs are stored, who can read them and how long they are kept. Document any MEASURE 1.1 risk that none of these tools can measure, with the reason, so the gap is visible to assessors. Ask your implementation partner for a written list of the tools' data paths and access controls before deployment.

Who does what for MEASURE 1.1: your team, a partner and the auditor

Your organization owns MEASURE 1.1. The outcome asks that metrics are chosen for the risks identified in MAP, starting with the most significant ones, and that unmeasured risks are documented. Those choices are business and risk decisions, so they cannot be delegated to a vendor. AI development staff know how the system behaves, TEVV staff know how to test it, domain experts know what harm looks like in context, and risk owners decide what is acceptable.

AI development: names the system components, data sources and model versions that each metric must cover.
TEVV staff: proposes testing procedures, acceptable performance limits and the course-correction steps when a limit is breached.
Domain experts: judges whether a metric captures real-world harm for the users and context involved.
Risk owners: approve the metric set, the acceptable limits and the written justification for any risk that will not be measured.
Documentation owner: keeps metrics considered but not used, with the reasons for each choice.

Before you engage anyone, confirm that these roles have names, that the risk owner has signed off the metric selection criteria, and that the list of unmeasured risks is reviewed on a set schedule.

Where organizations fall short on MEASURE 1.1 and how to close each gap

MEASURE 1.1 asks organizations to choose metrics for the mapped AI risks, starting with the most significant, and to record what they cannot measure. The gaps below tend to appear together: metrics drift away from the risks they were meant to track, and the reasons behind those choices never get written down. Each card names the gap and the action that closes it.

Metrics not traced to mapped risks

Teams often adopt accuracy, latency or uptime dashboards because they are easy to collect, then cannot say which MAP risk each one measures. MEASURE 1.1 starts from the risks enumerated in MAP, so every metric should point back to a named risk, its owner and the trustworthiness characteristic it serves. Check that each risk in the MAP register has at least one metric, or a documented reason it has none. Metrics without a traceable risk are candidates for removal or relabeling.

No record of metrics rejected

The Playbook suggests documenting the criteria used to select metrics, including metrics considered but not used. Without that record, an assessor or a new team member cannot tell whether a weak metric was chosen deliberately or simply overlooked. Keep a short selection log for each risk that lists candidate metrics, the reason each was accepted or dropped, and who approved the decision. Review the log when the system or its use changes.

Limits set without course correction

Acceptable performance limits are only useful if a breach triggers a defined response. Many teams set thresholds for accuracy, toxicity or error rates but never write down who reviews a breach, what gets paused and how the system returns to service. Pair each limit with a named owner, an escalation path and a corrective action such as retraining, rollback or narrowing scope. Test the path once before production so the first breach is not the first rehearsal.

Unmeasured risks without written reasons

Some trustworthiness characteristics, such as fairness in an open-ended use or explainability of a third-party model, cannot be measured with available methods. MEASURE 1.1 expects these to be documented rather than silently dropped. For each unmeasured risk, record why measurement is not feasible, what proxy or qualitative check is used instead, and when the decision will be reviewed. Ask the risk owner to sign the entry so the gap is an accepted decision rather than an omission.

Generative AI risks left for red teaming

NIST AI 600-1 lists suggested actions under MEASURE 1.1 that track generative AI risks that cannot be measured quantitatively and define where structured human feedback, such as red teaming, would be most useful. Many programs run automated benchmarks only and never decide where adversarial testing belongs. Identify the GAI risks that apply to the system, such as information integrity or harmful bias, then schedule red-team exercises for those with the highest impact. Record the scope, testers and findings alongside the metrics.

Probe results kept outside the metric record

Automated probing of language models produces large volumes of attempt and evaluation data, but that output often sits in a scratch folder rather than the measurement record. Tools such as garak write a JSONL report of every attempt and a hit log of successful attacks, and tracing platforms such as Langfuse record inputs, outputs and token usage per request. Decide which of these outputs feed each metric, store them with the risk entry they support, and confirm retention meets your own policy. Confirm exact output formats against the current tool versions in your environment.

Questions buyers ask about MEASURE 1.1 before hiring outside help

What does outside help for MEASURE 1.1 cost, and what drives the price?

The price depends on the written scope, not on a standard rate, and DeployOpen does not publish a price list. The main cost drivers are how many AI systems are in scope, how many risks are mapped to metrics, how much testing infrastructure must be built, and how many evidence sources need to be connected. Costs fall into one-time work, such as selecting metrics, documenting selection criteria and building the first test procedures, and recurring work, such as reviewing metrics as systems change. Ask any partner for a quote that separates these two groups, lists which systems and risks are covered, and states who holds the metric documentation after handover. A quote follows from the written scope, so share your inventory of AI systems and the MAP outputs before asking for one.

How long does MEASURE 1.1 work take for a single AI system?

Duration depends mainly on how many risks came out of MAP, how many of those are significant enough to measure first, and whether the metrics need new instrumentation or can use existing logs. A single system with a short list of mapped risks and existing logging usually takes less effort than a fleet of systems that need new evaluation harnesses. NIST's Playbook asks organizations to start with the most significant risks, so the first phase is often a ranking exercise before any build work. Timelines should appear in the written scope, along with the dependencies that could delay them, such as access to data, staff availability and sign-off from domain experts. Ask for a schedule that names each milestone and the party responsible for it.

What does the internal team need to contribute to MEASURE 1.1?

The internal team must own the decisions that measurement depends on. NIST's Playbook tags MEASURE 1.1 with three AI actor groups: AI Development, TEVV and Domain Experts. Your team needs people who know the system's intended purpose, its users, its data and the consequences of error. Expect to provide access to logs and test environments, time from domain experts to set acceptable performance limits, and a named owner who can accept or reject metrics. Setting limits and deciding what is an acceptable risk remain organizational decisions. An outside partner can draft metric definitions and procedures, but it cannot decide on your behalf what level of performance is tolerable for your users.

Can one person in our organization handle MEASURE 1.1 alone?

One person can usually coordinate MEASURE 1.1, but rarely can carry all of it alone. The work spans AI development, testing and evaluation, and domain judgment, which NIST's Playbook treats as separate actor groups. A capable coordinator with access to engineers, a test owner and a domain expert can often run the process for a small number of systems. As the number of systems, risks and metrics grows, the workload becomes a team effort. Before deciding, check whether your team can write measurement procedures, document metrics considered but not used, and justify any mapped risk that will not be measured. Any gaps in those skills are the areas where outside help is most often requested.

Can an outside partner prepare MEASURE 1.1 evidence without becoming the assessor?

Yes, provided the roles are kept separate. DeployOpen plans, implements, tests and documents systems and evidence inside your environment, but it does not issue audit reports, attestations or certifications. Independent assessors and certification bodies do that work. For MEASURE 1.1, a partner can help select metrics, build test procedures, document selection criteria and record the risks that cannot be measured. Your organization owns the controls and the decisions behind them. Where an independent assessment is needed, MEASURE 1.3 covers the role of internal non-developer reviewers and external assessors, and the assessor should be selected separately from the team that built the evidence.

Does the NIST AI RMF have a certification I can ask a partner to obtain?

No. The NIST AI RMF is a voluntary framework, and NIST does not offer a certification, accreditation or conformity assessment program for it. NIST's use cases page also states that NIST does not validate or endorse any individual organization or its approach to using the framework. Any claim of AI RMF certification should be treated with caution and checked before signing. What organizations can do is document how they apply the subcategories, measure their AI risks, and show evidence to customers or regulators. If a contract asks for an independent opinion, ask the assessor which standard or report type they will issue, because that is a separate question from the framework itself.

What does a MEASURE 1.1 engagement produce that we keep after it ends?

A well-scoped engagement should leave your organization with documentation you can maintain without the partner. Typical outputs include a ranked list of mapped risks, the selected approaches and metrics for each, the acceptable performance limits and course-correction steps, a record of metrics considered but not used, and a list of risks that will not or cannot be measured with the reasons. Ask the partner to name each deliverable in the written scope and to confirm who holds the source files, test scripts and logs. Ownership of code and documentation should be settled before work starts, not at handover.

How DeployOpen works on this

DeployOpen helps organizations choose, build and document the metrics and test approaches behind MEASURE 1.1, using their own infrastructure and self-hosted open-source tooling. We prepare the systems and evidence, while the organization owns its risk decisions and any independent assessor issues the findings.

How we scope

Agree the in-scope AI systems and the MAP risks that the metrics must cover, in writing before work starts.
Record which risks or trustworthiness characteristics are expected to be measured, and which will be documented as not measurable.
Confirm which internal owners, AI development, TEVV and domain experts, hold each metric decision.

How we build

Set up test procedures and metric definitions for the highest-priority mapped risks, with stated acceptable performance limits.
Deploy open-source evaluation and red-teaming tools such as garak in the client's own environment, configured to the agreed scope.
Instrument tracing with Langfuse so that inputs, outputs, tool calls and token usage can support the metrics chosen.

How we hand over

Document metric selection criteria, including metrics considered but not used and the reasons for each decision.
Deliver a register of risks that cannot be measured quantitatively, with justifications and planned human feedback exercises.
Provide evidence packs, run records and configuration files so the organization or its independent assessor can review them.

How we support

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
Ongoing support, upgrades and continuous testing are available as a retainer, so metrics can be reviewed as systems change.
Confirm with DeployOpen how revisions to NIST AI RMF 1.0 or the Generative AI Profile will be reflected in the agreed scope.