MEASURE 3.1 tracking AI risks: requirements and evidence

What NIST AI RMF MEASURE 3.1 asks for after an AI system goes live

MEASURE 3.1 asks whether your organization still knows how a deployed AI system is behaving, and whether it can show that knowledge. The pre-launch risk assessment was a snapshot. After go-live, the system meets real users, new data, changed upstream models and unexpected inputs, so its intended performance and its actual performance can drift apart. The subcategory expects approaches, personnel and documentation to be in place to regularly identify and track existing risks, risks nobody anticipated and emergent risks, using factors such as intended versus actual performance in deployment. The AI RMF is voluntary and NIST does not certify anyone against it, so there is no pass mark. What a customer, regulator or independent assessor will look for is a running record: which measures are tracked, who reviews them, how often, and what happened when a measure moved.

Three problems make this hard in practice. The first is no baseline: if the pre-launch testing results were never stored in a form that production data can be compared against, nobody can tell whether performance has changed. The second is scattered logs: model outputs, retrieval results, tool calls, support tickets and infrastructure alerts sit in separate systems with different retention rules. The third is no owner for feedback: support and incident response staff see problems first but have no defined path into the risk register. The Playbook suggests actions such as comparing system risks with simpler models, human baselines and manual benchmarks, comparing user feedback with internal performance measures, measuring how quickly errors are answered, and asking support staff what information and staffing they need. These are prompts, not a checklist. Choose the ones that fit your system's risk, and record why you skipped the rest.

Before you buy outside help, confirm three things internally: which named role (for example, the system owner, a TEVV lead or an operations lead) reviews tracked risks and on what cadence; where the baseline metrics from pre-deployment testing are stored; and which monitoring tools already capture traces and alerts. Those answers define the scope an implementation partner would prepare and the evidence an assessor would request. An implementation partner can set up monitoring, logging and documentation in your environment, but the risk decisions, the accountable owner and any conclusions about the system remain yours. Only an independent auditor or assessor can test and report on the result.

Which AI systems, data, teams and third parties MEASURE 3.1 covers

MEASURE 3.1 asks for approaches, people and documentation that keep tracking risk after deployment, so the first decisions are about what sits in scope. The cards below cover the choices that change how much monitoring work an organization takes on and who owns it.

Production systems versus pilots

MEASURE 3.1 is about risk in deployed contexts, so a system that only runs in a sandbox sits outside its core purpose until it reaches users. Decide where the line falls, for example at first production traffic or at the first decision that affects a person. Pilots that touch real data or real users should carry the same tracking as production, because intended and actual performance can diverge early. Check that the inventory from GOVERN 1 records each system's deployment status so that scope is not decided ad hoc.

Generative versus predictive models

Generative systems can produce errors that predictive scoring models do not, including fabricated outputs, leaked data and unsafe content. The tracking method for each type differs, so the scope should name which model family each monitored system uses. NIST AI 600-1 lists no suggested actions for MEASURE 3.1, so generative systems need the organization's own tracking design, which should be documented as such. Ask whether the monitoring plan covers both model types or only the one that the team builds.

Third-party and pre-trained models

Many organizations operate models they did not train, such as vendor APIs or open-weight models fine-tuned in-house. Performance can change when a vendor updates a model, even though the organization's code has not changed. Confirm which third-party components feed each system, who receives vendor change notices, and whether the organization can observe outputs independently of the vendor's own reporting. MAP 4 and MANAGE 3 cover the supply chain side of these components.

Data sources and feedback inputs

Tracking depends on what data the organization can see after deployment. Inputs may include prompts and outputs, retrieval sources, user complaints, support tickets and incident records. Each input raises privacy and retention questions, so the scope should list which data is logged, where it is stored and who may read it. Feedback from end users and impacted communities also belongs here, because MEASURE 3.3 relies on the same channels for reporting and appeals.

TEVV team ownership

The Playbook tags MEASURE 3.1 with the TEVV actor category, which covers testing, evaluation, verification and validation. Under AI 100-1 Appendix A, this group is responsible for measurement design and results, so it needs access to production signals, not only pre-release test sets. Decide who in the TEVV group owns the tracking metrics, how often results are reviewed and who signs off on changes to thresholds.

Impact assessment and operations ownership

The Playbook also tags MEASURE 3.1 with the AI Impact Assessment and Operation and Monitoring categories. Impact assessment staff judge whether unanticipated effects are emerging, while operations staff run the alerts, incident response and support queues that surface problems first. Name a person or team for each function, define how their findings reach one another, and confirm that someone holds the authority to pause a system when tracked risk crosses an agreed threshold.

How an assessor tests MEASURE 3.1 in practice

This table sets out how an assessor would trace a deployed AI system through MEASURE 3.1, what evidence they would request for each step, and the gaps they most often find. AI RMF reviews are not formal certification, so every conclusion rests on the organization's own records and an independent review of them.

Test area	Evidence requested	Common findings
Trace intended performance to monitoring	Documented intended purpose and performance expectations from MAP and MEASURE 1, the monitoring plan, and recent results for each tracked metric	Intended performance is written at a high level with no measurable thresholds, so monitoring results cannot be compared against anything
Confirm actual deployed performance is compared with intended performance	Dashboards or reports showing deployed metrics over time, with the comparison method and the date each review was held	Metrics are collected but never compared with the documented baseline, or reviews are undated and cannot show a regular cadence
Sample tracked risks for owners and responses	A sample of tracked risks with the named owner, the response taken, the decision record and the closure or re-review date	Risks are logged without an owner, or responses are described as planned with no record that they were carried out
Check detection of unanticipated and emergent risks	Incident and issue logs, alert rules covering model behavior and data drift, and records of risks added after deployment	Monitoring covers only the risks named in the original assessment, so new failure modes appear first in incident reports
Review user and support feedback	Feedback intake records, triage notes, and minutes showing that support and incident response input reached the risk review	Feedback is collected in a support queue that no one reviews against model performance measures
Check error response times are measured	Defined response time targets, timestamps from detection to triage to closure, and reports on how often targets were met	Response times are not recorded, or targets exist in a policy document but no report shows performance against them
Check staffing and information needs are reviewed	Records of feedback from support and incident response staff on the information they need and whether staffing matches the load	Operations staff are not asked about monitoring gaps, so alerts go unanswered when staffing is thin
Check documentation is current	Versioned monitoring procedures, change records for the system, and evidence that monitoring was updated after material changes	Monitoring documentation still describes an earlier model version or configuration after the system changed

How to set up risk tracking for a deployed AI system

This sequence turns MEASURE 3.1 into a working routine: named owners, agreed baselines, production telemetry, comparisons and feedback loops that feed a single register of tracked risks. Each step ends with a record an assessor can inspect.

Name an owner for each deployed system
Assign a accountable owner and a named operator for every AI system in production, drawn from the AI inventory produced under GOVERN 1. The owner decides who reviews tracking results and who can accept or escalate a risk, and the operator runs the daily checks. Record both roles in a tracking charter per system, listing the people, their backups and the decisions each may make. The step is done when every system in the inventory has a charter and no system lacks a named owner.
Define baselines and intended-performance measures
Take the context and risk priorities from the MAP and MEASURE 1 work and convert them into measurable targets for deployment. Specify the intended performance for each significant risk, the thresholds that signal drift or unexpected harm, and the metrics that will be compared over time. Document each measure with its definition, data source, owner and the reason it was chosen. The output is a measurement plan with a baseline record for each system, showing the values observed at launch.
Instrument production telemetry
Configure logging and tracing so that the measures in the plan can be computed from live operation. For language-model applications, tracing tools such as Langfuse record each request with its inputs, outputs, timing, token usage and tool calls, which lets the team compare actual behavior with intended behavior. Grafana Alerting can then build rules over the metrics and logs together. Confirm retention periods, access controls and how personal data in traces is handled with the privacy team before turning logging on. The step is done when a test request appears in the trace store and the alert rules fire on a deliberately faulty test input.
Set comparisons against simpler models and human baselines
The Playbook suggests comparing AI system risks with simpler models, human baselines and manual benchmarks, so the team can tell whether the system performs better than the alternative it replaced. Choose a comparison method for each system, run it on a fixed sample at regular intervals, and record the difference in error rates and harm indicators. The comparison also shows when a simpler approach matches the AI system, which is a legitimate finding for MANAGE 2. The output is a comparison report per system, dated and signed off by the owner.
Connect user and community feedback to performance measures
Route complaints, support tickets, incident reports and community feedback into the same system as the telemetry, so each item can be matched to a metric or a risk. Gather input from support and incident response staff on what information they need and how many staff they need to act on it, as the Playbook suggests. Measure how long the organization takes to respond to reported errors, and track that response time against the target set in the plan. The output is a feedback log linked to metric records, with response times that the owner reviews.
Set the review cadence and escalation path
Define how often the owner reviews the metrics, comparison reports and feedback log, and what happens when a threshold is crossed. The escalation path should name who is notified, who can suspend or roll back the system, and which decisions go to senior management, consistent with the accountability structure set under GOVERN 2. Agree on the cadence and the escalation criteria in writing before go-live, and rehearse one escalation with a test event. The output is a review calendar and an escalation procedure approved by the system owner.
Record each tracked risk and its status
Open a record for every risk that the monitoring surfaces, whether it was anticipated in the MAP work or emerged after deployment. Each record should state the evidence, the affected system, the severity as judged by the owner, the current status (open, monitoring, mitigated or accepted) and the next review date. Link each record to the metric, comparison or feedback item that raised it. The step is done when the register is populated for every system and an assessor could trace any entry back to its source data.

Which evidence shows MEASURE 3.1 is in place and current

MEASURE 3.1 is easiest to defend when each artifact is dated, owned and linked to the deployed system it describes. The cards below cover the records an assessor or customer is likely to ask for, and how to keep each one current between reviews. None of these artifacts is a certificate: the organization owns them, and an independent reviewer decides whether they satisfy a contract or regulator.

Intended-performance baseline per system

This is the documented description of what each deployed system is supposed to do, the performance levels and error types the team accepted, and the conditions under which those levels apply. Without it, actual performance has nothing to be compared against. Check that each baseline names a version of the system, carries an owner and a sign-off date, and is re-approved after material model, data or scope changes. Ask whether the baseline was written before deployment or reconstructed afterward.

Monitoring dashboards and alert rules

Export the dashboard definitions and alert rules that watch each system's outputs, latency and error signals, and store them in version control so every change has an author, a reason and a date. Assessors often ask whether an alert that was tuned or silenced left a record. Confirm that the export includes the data sources each rule queries, and that the change history can be shown for the full review period your policy sets.

Tracked-risk register with owners and dates

A register lists each existing, unanticipated or emergent risk observed in deployment, its severity, the named owner, the date it was logged, its status and the date it was last reviewed. For generative systems, entries can reference the trace or test result that surfaced the issue. Check that closed items keep their history rather than disappearing, and that new entries show when they were first detected, not when they were entered.

Comparison reports against baselines and humans

The Playbook suggests comparing system risk against simpler models, human baselines and manual benchmarks. Keep dated reports that show the system's measured behavior against the baseline and against those comparators, with the test set, the evaluator and the method recorded. Ask whether the reports were produced on a schedule or only after an incident, and whether results that were unfavorable led to a documented decision.

Records of user and support feedback reviewed

Feedback from users, support staff and incident responders only counts as MEASURE 3.1 evidence when it is logged, triaged and compared with internal performance measures. Keep the intake records, the triage decisions and the links to any risk register entries they produced. Confirm that the review cadence is written down, that reviewers are named, and that feedback that was dismissed has a recorded reason. Illustrative example: a support ticket about wrong answers that is linked to a new test case and a register entry.

Error response time records

Measure how long it takes from detection of an AI error or risk to the first response and to containment, and keep the timestamps from ticketing, alerting and incident tools. These records show whether the organization acts on tracked risks, which is the point of the subcategory. Check that the timestamps come from systems that cannot be edited without a trace, and that the export covers the same period as the register and the dashboards so the artifacts can be cross-checked.

What tracking AI risks in deployment costs to meet

This table lists the cost components of running MEASURE 3.1 monitoring, who is paid for each, whether it is one-time or recurring, and what drives it up or down. Use it to build a quote request, since NIST sets no fee and DeployOpen does not publish prices.

Cost component	Who is paid	One-time or recurring	What drives it up
Tooling or licenses	Software vendors or open-source support providers	Recurring if commercial; open-source core often has no license fee	Number of deployed systems, log and trace volume, number of data sources
Engineering time to instrument systems	Internal platform or ML engineers, or an implementation partner	One-time per system, plus rework after each model change	Number of model endpoints, retrieval and tool integrations, custom metrics required
Analyst time for risk review	Internal risk, TEVV or operations staff	Recurring	Feedback volume, number of alert rules, severity of deployed use cases
Feedback and incident intake	Support, incident response and product teams	Recurring	End user report volume, number of channels, response time commitments set internally
Evidence upkeep	Internal compliance staff or a readiness partner	Recurring, tied to review cycles and audits	Number of systems in scope, frequency of documentation updates, number of frameworks mapped
External testing or review	Independent testers, red teams or assessors	Recurring at the frequency the organization sets	Model change frequency, third-party dependencies, the breadth of the test scope
Auditor sampling	Independent auditor, where an audit requires it	Recurring per audit cycle	Number of evidence samples requested, number of systems covered, auditor's fee basis
Hosting and infrastructure	Cloud provider or data center operator	Recurring	Retention period for traces and logs, storage volume, redundancy requirements

Self-hosted open-source tools that can produce MEASURE 3.1 evidence

MEASURE 3.1 asks for a regular way to spot existing, unanticipated and emergent AI risks after deployment. The four tools below cover different parts of that job: adversarial probing, request-level tracing, assessment records and alerting. Each produces records that help show tracking happened, but none of them decides which risks matter.

garak for adversarial probing

garak is an Apache-2.0 scanner that probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. For MEASURE 3.1, its JSONL report records every attempt and evaluation, and its hit log lists attempts that found a vulnerability, which gives dated evidence that probing repeats after deployment. Confirm that the probe set matches the risks your system actually carries, and that results feed a documented review rather than sitting in a log.

Langfuse traces for live behavior

Langfuse is an MIT-licensed observability tool. Each trace captures every operation in a request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. These traces let a team compare intended and actual performance in deployment, spot drift in outputs or cost, and trace an incident back to a specific request. Confirm your data retention rules and how inputs containing personal data are masked before storage.

CISO Assistant for assessment records

CISO Assistant is an AGPL-3.0-only tool that ships more than 200 frameworks, including NIST CSF, ISO 27001 and the EU AI Act family of obligations, and runs audits and compliance assessments against them with automatic mapping between frameworks. For MEASURE 3.1, it can hold the register of identified risks, their owners, review dates and assessment results, so reviews are dated and traceable. It records what your team decided. It does not generate risk findings, so the risk judgment stays with your people.

Grafana Alerting for threshold breaches

Grafana Alerting is AGPL-3.0-only and builds alert rules from queries and expressions across multiple data sources. It can alert on metrics and logs together and lets teams create, manage and act on alerts from a single view. For MEASURE 3.1, alert rules can flag drops in quality signals, rising error rates or slow response times, and the alert history supports the claim that emergent problems are noticed. Confirm which signals are meaningful for your system, who receives alerts, and who is accountable for acting on them.

Who owns MEASURE 3.1 and what a partner can take on

Your organization owns the outcome of MEASURE 3.1. No partner or auditor can decide which deployed AI risks matter to your business, accept a residual risk, or determine how a complaint about an automated decision is resolved. The Playbook suggestions for this subcategory point toward people who are close to the system in use, including operation and monitoring staff and the teams that handle support and incident response, so these roles must be named before work begins.

Risk ownership: Name an accountable owner for each deployed system who signs off on the risk tracking approach, the metrics chosen and the thresholds that trigger review.
Decisions: Record who decides when an emergent risk leads to a change, a restriction or a shutdown, and how that decision is documented.
Feedback handling: Assign staff to triage user and community feedback, compare it with internal performance measures, and route it to the right owner.
Response measurement: Track how long it takes to acknowledge and act on reported errors, and review that data on a regular schedule.
Staffing input: Gather feedback from support and incident response staff on the information and headcount they need.

Confirm that these owners have the authority and time to act before a partner begins configuration work.

Questions buyers ask about MEASURE 3.1 cost, effort and outsourcing

How much effort does MEASURE 3.1 take to implement?

The effort depends on how many AI systems are in production, how much each one changes, and how much monitoring already exists. MEASURE 3.1 asks for approaches, personnel and documentation that regularly identify and track existing, unanticipated and emergent risks after deployment, so the work is ongoing rather than a single project. Most of the effort lands in three places: defining which performance and risk signals to track for each system, wiring those signals into logging and alerting, and keeping the documentation current as systems and their use change. NIST AI 100-1 is voluntary and does not set a required level of effort. Ask any partner to show how the work scales with the number of systems and the number of data sources, and to separate one-time setup from recurring review.

Can an in-house team implement MEASURE 3.1 without outside help?

Yes, an in-house team can implement MEASURE 3.1 if it has people who understand the deployed systems, the monitoring stack and the affected users. The NIST AI RMF Playbook tags MEASURE 3.1 with the TEVV, AI Impact Assessment and Operation and Monitoring actor categories, so several roles need to contribute. The main gaps tend to be independent testing skills, time to maintain documentation, and experience with the incident response and support processes that feed the feedback loop. An outside partner can prepare systems and evidence, but the organization keeps ownership of the risk decisions and the tracking program. Confirm internally which roles cover each actor category before deciding.

What does an implementation partner need from our organization?

A partner needs access to the people and systems that show how AI behaves in production. Typical inputs include an inventory of deployed AI systems, their intended purpose and known limits, the data sources and logs that exist today, current incident and support workflows, and named owners for each system. The partner will also need access to the monitoring platform, permission to add instrumentation where needed, and time with operations, support and incident response staff. Expect requests for existing risk documentation and any prior evaluation results. Clear ownership on the client side shortens the work more than any technical choice.

How long does setup usually take, and what drives that timeline?

No fixed duration applies to MEASURE 3.1, so treat any single number with caution and ask for one based on your written scope. The timeline is driven mainly by the number of deployed systems, how much telemetry already exists, the number of data sources that need to be connected, and how quickly stakeholders can review and approve tracking thresholds. Systems with generative components usually need additional evaluation design, and the NIST AI 600-1 Generative AI Profile does not list suggested actions for MEASURE 3.1, so the team must define those checks itself. Ask the partner to split the timeline into phases with dependencies, so you can see which steps wait on your teams.

Is MEASURE 3.1 required by law?

No. NIST's AI RMF FAQ states that neither private nor public sector organizations are required to use the framework, and NIST describes it as voluntary. Some customers, contracts or state laws may name the AI RMF or its Generative AI Profile, so a specific obligation can still apply to you even though the framework itself is optional. Check the contract or law that cites it and confirm exactly what it expects. Also note that NIST expects a formal review no later than 2028, and the White House AI Action Plan recommends a revision, so the text of MEASURE 3.1 may change. Confirm the version your contract references.

What should a written scope for MEASURE 3.1 include?

A written scope should name the AI systems in and out of scope, the deployed contexts to be monitored, the risks and performance factors to track, the data sources and tools involved, and the documentation to be delivered. It should also state who owns each decision, how feedback from users and support staff enters the process, and how alerts are handled. Ask for deliverables described in concrete terms, such as a tracking plan, monitoring configuration, runbooks and an evidence index, along with the dates on which each is due. For fixed-scope engagements, DeployOpen agrees scope, deliverables and timelines in writing before work starts, and any quote follows from that written scope. Ask every bidder to state what it will not do.

How DeployOpen works on this

DeployOpen helps organizations set up the systems, monitoring and evidence that MEASURE 3.1 asks for, inside the client's own infrastructure. The organization keeps ownership of its risk decisions, and DeployOpen prepares the operational record that shows how deployed AI risks are tracked over time.

How we scope

Engagements are fixed-scope, with the MEASURE 3.1 deliverables and systems in scope agreed in writing before work starts.
The written scope lists which AI systems, deployed contexts and risk indicators will be tracked for existing and emergent risks.
Scope names the people who review tracking results, such as TEVV, impact assessment and operations staff, and their responsibilities.

How we build

We set up monitoring and logging for deployed AI systems, including traces of inputs, outputs, tool calls and timing where the platform supports them.
We configure alert rules that combine metrics and logs, so drift, error spikes and unexpected behavior reach a named response owner.
We test probes and evaluation tooling against the system before go-live, and record what each test covers and what it does not cover.

How we hand over

We document the tracking approach, data sources, alert thresholds and escalation paths in a form the client's own teams can maintain.
Handover includes records of comparisons against simpler models, manual baselines and user feedback, where those comparisons are in scope.
Evidence packs are organized for review by the client and, where the client chooses, by an independent auditor or assessor, who issues any formal report.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer, so tracking keeps pace with system and framework changes.
Retainer work can refresh probes, dashboards and alert rules when the AI system, its data or its deployed context changes.
Ask us to confirm which parts of the tracking record your internal teams own after handover, and which parts stay with a retainer.