MEASURE 2.4 Production monitoring: requirements and evidence

What MEASURE 2.4 production monitoring asks for in plain terms

MEASURE 2.4 asks that the functionality and behavior of a deployed AI system and its components, as identified in MAP, are monitored once the system is in production. If your AI system is live, this subcategory is the point where the work of checking it shifts from pre-launch testing to watching what it actually does with real inputs. The outcome sits in MEASURE 2 of the Measure function in *NIST AI 100-1* (Table 3). The framework is voluntary and has no certification, so no one issues a MEASURE 2.4 certificate. An independent assessor or your own internal audit may test whether the monitoring exists and works, but the evidence is what you produce.

The risk this addresses is performance decay after launch. Input data shifts as customer behavior, upstream systems or markets change, and outputs can degrade without any error being raised. This is silent failure: the system keeps answering, dashboards stay green, and the model is simply wrong more often. Monitoring is hard for three reasons. Production data is messy and rarely matches the test set. Ground truth often arrives late, sometimes weeks later, so accuracy cannot be measured in real time. And monitoring is frequently owned by no one: the data science team hands off to operations, operations watches uptime rather than model behavior, and the risk owner never sees the signals. Before you plan any tooling, confirm in writing who owns each monitored signal, who receives the alerts and who can pause the system.

Which AI systems, data and teams fall under production monitoring

MEASURE 2.4 only works if the organization first decides what is in scope. The decisions below connect the systems identified in MAP to the monitoring work, the data flows that need watching and the teams that own the results. Settle them in writing before tooling is chosen, because each one changes the metrics, alerts and review queues that follow.

Deployed systems identified in MAP

Start from the inventory of AI systems that MAP 1 and MAP 2 recorded as deployed or about to be deployed, not from whatever the platform team happens to instrument. Each system with a documented intended purpose, known limits and user population should have an owner and a monitoring scope. Systems that were mapped but never released still need a decision: either they are excluded with a reason, or they carry pre-deployment checks forward. Ask for the inventory version and the date it was last reviewed.

Components, not only the model

MEASURE 2.4 asks about the functionality and behavior of the system and its components. For a typical deployment that means the prompt templates, classifiers, feature pipelines, ranking logic, rules and the serving layer, not just the model artifact. A drop in output quality can come from a changed retrieval index or a rule update while the model stays the same. Record each component in scope and the version or configuration hash that identifies it.

Generative AI features

Decide whether generative features are monitored at the same depth as predictive models. Monitoring usually needs to cover prompts and completions, token usage, tool calls and refusal rates, because failures such as hallucination, prompt injection or toxic output do not show up in standard accuracy dashboards. The NIST Generative AI Profile adds no suggested actions for MEASURE 2.4, so the team must define these metrics itself. Confirm the scope with the risk owner and document why each feature is included or excluded.

Third-party models and APIs

Where a vendor hosts the model or supplies an API, the organization still owns the production behavior its users see. Monitoring scope should include the vendor version identifier, latency, error rates and output distributions observed at the organization's boundary, even when the vendor's internal metrics are not available. The MANAGE 3 pages cover the contractual side. For MEASURE 2.4, confirm what telemetry the vendor exposes and what the organization can capture itself.

Inputs, outputs and retrieval sources

Monitoring for distribution shift needs a defined set of data flows: the inputs the system receives, the outputs it produces and, for retrieval-augmented systems, the documents and indexes it draws from. Each flow needs a baseline taken from pre-deployment testing so that later comparisons mean something. Confirm which flows contain personal data, since storing traces of them has privacy consequences that MEASURE 2.10 also covers. Ask who approves retention for each flow.

Shared ownership by deployment and TEVV teams

The Playbook tags MEASURE 2.4 with the AI Deployment and TEVV actor categories. Deployment staff usually run the telemetry and respond to alerts, while TEVV staff define the test baselines, ground truth sampling and thresholds for drift. Name one accountable owner for each monitored system and record who reviews unreliable outputs and who can pause the system. Without that split written down, alerts tend to reach a dashboard that no one is assigned to read.

How an assessor tests production monitoring

This table sets out each step an assessor or internal auditor takes to test MEASURE 2.4, the evidence requested at each step and the findings that commonly appear. Read it as a test plan: each row is a check you should be able to show before a review, and the last column names where programs usually fail.

Assessor test	What the assessor checks	Evidence requested	Common finding
Scope against the MAP inventory	Monitored components match the system components identified in MAP, including third-party models, data feeds and retrieval sources	MAP inventory, monitoring configuration export, list of monitored data sources	Third-party components or vendor-hosted models with no monitoring attached
Production versus pre-deployment metrics	Production metrics are compared with the baseline set in pre-deployment testing under the same metric definitions	Pre-deployment test report, production metric dashboards for a sampled period, comparison notes	Baselines recorded in a slide deck but not in the monitoring system, so no like-for-like comparison is possible
Distribution shift tests	Input and output distributions are tested for shift, and the method and reference window are documented	Drift test configuration, reference dataset definition, sample drift reports	Drift is measured on one feature only, or the reference window was never defined
Thresholds and their rationale	Anomaly limits, control limits and alert thresholds are reasoned, approved and reviewed on a set cycle	Threshold register with rationale, approval record, review log with dates	Thresholds set at deployment and never reviewed, or default values with no documented reason
Alert routing and ownership	Each alert rule routes to a named owner and an escalation path, and firing is tested	Alert rule export, notification routing configuration, test alert records	Alerts route to a shared channel with no named owner, or rules were never tested
Response records	Each fired alert has a recorded triage, action or decision, including closure	Incident or ticket records linked to alerts, closure notes, sampled timeline	Alerts with no response record, or acknowledgements without any action taken
Human review of flagged outputs	Outputs flagged as unreliable are reviewed by a qualified person, and the review outcome is recorded	Review queue logs, reviewer roles, sampled review decisions, feedback into the system	Review queue exists but reviewers are not defined, or reviews are not logged
Ground truth comparison	Outputs are assessed against newly collected ground truth on a documented cadence	Ground truth sampling plan, labeled samples, comparison results	No ground truth collection after launch, so accuracy in production is unknown

How to implement production monitoring in five steps

This sequence turns MEASURE 2.4 from a statement of intent into a working monitoring program, starting with the risks identified in MAP and ending with a review loop that tests outputs against new ground truth. Each step produces a named artifact that an assessor or internal reviewer can inspect.

Define monitored signals from MAP risks
Start from the risks recorded under MAP, and list the functions and behaviors of the AI system and each of its components that would reveal those risks in production. Record each signal in a signal register with its source, owner, the MAP risk it tracks and the data it depends on, such as model inputs, outputs, confidence scores, retrieval results or tool calls. The step is complete when every high-priority MAP risk maps to at least one signal and the AI deployment and TEVV owners have signed off on the register.
Establish pre-deployment baselines
Before go-live, record how the system performs in testing so production results have something to be compared against. Capture input and output distributions, error and refusal rates, latency and any quality scores from the deployment-like testing described under MEASURE 2.3, and store them in a baseline report with the test sets and conditions used. The NIST Playbook suggests comparing production metrics with pre-deployment testing, so the baseline must be stored in a form that monitoring tools can query later. The step is done when the baseline report is versioned and the TEVV lead has accepted it.
Instrument the system and its components
Add telemetry to the system and to each component it relies on, including models, data pipelines, retrieval steps and third-party services. Metrics collection can use a pull model such as Prometheus, and request-level traces can capture inputs, outputs, timing, token usage and tool executions in a tool such as Langfuse, depending on the architecture. Confirm with the platform team what each tool captures, how long data is retained, and where personal data is masked before storage. The step is done when a test request produces the expected metrics and traces in each system and the monitoring configuration is recorded in the signal register.
Set drift and anomaly thresholds with alerting and escalation
Define the thresholds that matter for each signal, using control limits, confidence intervals or a machine learning method where simple limits do not fit, and configure alert rules that fire when drift or anomalies cross them. Grafana Alerting can build rules from queries across several data sources and combine metrics with logs, so one view can show related alerts. Document who receives each alert, the expected response time agreed internally, and when an alert must go to the accountable owner for a decision. The step is complete when alert rules are tested with simulated drift, and the alert rules and escalation paths are recorded in a monitoring runbook.
Run a review loop against newly collected ground truth
On a schedule the organization sets and documents, sample production outputs, compare them with newly collected ground truth, and review the unreliable outputs that human reviewers flag. Each review produces an entry in a review log that records the sample, the findings, any drift or anomaly investigated, the decision taken and whether the baseline or thresholds need revising. Findings that point to a new or changed risk go back to the MAP and MEASURE records so that the signal register stays current. The step is in place when the review log shows completed cycles and a named owner signs off on each one.

How drift, anomalies and unreliable outputs are detected in practice

Detecting problems in production under MEASURE 2.4 comes down to four distinct checks, and each needs its own signal. The first is distribution shift in inputs and outputs: comparing the data the system sees and produces now against the baseline captured before deployment. The second is anomaly detection, where a metric is judged against control limits, confidence intervals or machine learning methods, so that an unusual value raises a flag without someone watching a dashboard. The third is alerting, which turns drift and anomaly signals into a notification with an owner. The fourth is assessment of outputs against newly collected ground truth, which is the only check that shows whether the system is still correct rather than merely behaving differently. The NIST AI RMF Playbook suggests these actions for MEASURE 2.4, but NIST states that the Playbook is neither a checklist nor a set of steps. Organizations should select the methods that fit the system's risk, document why, and confirm with their assessor which combination they expect to see.

Generative AI systems add monitoring surfaces that a classic predictive model does not have. Each request may pass through retrieval steps, tool executions and several model calls, and each step can fail or drift independently. Tracing tools such as Langfuse record the timing, inputs, outputs, token usage and metadata of each operation in a trace, which lets a team see whether a bad answer came from a retrieval miss, a tool error or the model itself. Token usage also matters as a behavioral signal, since sudden growth can indicate prompt changes, looping agents or unusual user behavior. The trace must link each output to the retrieval and tool steps behind it so that unreliable outputs can be reviewed in context.

Before choosing tools, confirm three things with the team that owns the system: which production metrics were compared against pre-deployment testing, where the baseline distributions are stored, and who receives each alert and is authorized to pull a system from service. Metrics collection can be handled by Prometheus, which scrapes instrumented jobs and stores time series, and alert rules can be built in Grafana Alerting across metrics and logs. Those tools are generic, so the confirmation step matters: an alert rule that nobody reviews produces no MEASURE 2.4 evidence. Keep the alert history, the review decisions on flagged outputs and the ground-truth samples as records, because an assessor will ask to see them. The components and settings should be confirmed against each tool's documentation in your own environment rather than assumed.

Which evidence shows production monitoring works

An assessor testing MEASURE 2.4 wants to see that production monitoring was planned, configured, run and acted on, not just installed. The cards below cover the records that connect a monitored signal to its baseline, its threshold, its alert and the human decision that followed.

Monitored signal register

This register lists each production signal the system is monitored on, such as input distribution, output confidence, error rates, latency and refusal rates, and ties each one back to the risk or impact identified in MAP. It is the starting point for the assessor, because it shows whether monitoring covers what the organization said could go wrong. Review it at each material change to the system and whenever MAP results are updated, and record who approved each change.

Pre-deployment baseline reports

The baseline report records the metric values measured in pre-deployment testing, under conditions close to production. Production metrics only mean something when they are compared with this reference, so the baseline must be stored with its test data, configuration and date. Confirm that the baseline is frozen, versioned and linked to the model version it describes, so that a later drift finding can be traced to a specific reference point.

Threshold and confidence interval documentation

This document explains how each alert threshold, control limit and confidence interval was chosen, who approved it, and when it was last reviewed. Assessors look for a reason behind each number, such as a statistical method or a risk tolerance from GOVERN 1, rather than a default value copied from a tool. Keep the document in version control next to the alert rules so that every threshold change has a documented justification.

Alert configurations and change records

Exports of the alert rules, notification routes and silence settings show what the monitoring system will actually flag and who it notifies. Schedule an export of these configurations, for example weekly, and store each export with a timestamp. Pair every change to a rule with a change ticket that names the requester, the reviewer and the reason, since an undocumented threshold change is one of the most common findings in monitoring reviews.

Alert history and drift reports

Alert history shows which alerts fired, when they were acknowledged and how long they stayed open, while drift reports show the statistical comparison of production inputs and outputs against the baseline. Together they demonstrate that the monitoring produced findings and that the findings were examined. Generate drift reports on a fixed schedule and keep them with the alert log, so that a gap in either record is visible during the assessment period.

Ground-truth comparison and human review records

Ground-truth comparisons record outputs that were later checked against newly collected correct answers, and human review logs record who examined unreliable or flagged outputs, what they decided and why. These records show that the organization tests whether the system is still right, not just whether its metrics look stable. Confirm that reviewers are independent of the team that operates the system where the risk warrants it, and keep corrective action records that show how each confirmed problem was fixed or escalated.

Which open-source tools produce monitoring evidence

Prometheus is the usual starting point for numeric production signals. It collects metrics from instrumented jobs through a pull model over HTTP and stores them as time series identified by metric name and labels, on local storage that does not depend on external systems. The team queries that history with PromQL, which is where a comparison against pre-deployment test results becomes possible, provided the same metrics were captured during testing.

Pull-based collection: Each instrumented job exposes an HTTP endpoint that Prometheus scrapes, so the scrape configuration itself documents which components report.
Labeled time series: Labels such as model version or environment let an assessor separate results by release, which supports the comparison MEASURE 2.4 asks for.
PromQL queries: Saved queries for input and output distributions and error rates can be re-run during an assessment to show how the numbers changed over time.
Local storage limits: Retention and capacity depend on the local disk and configuration, so the team must confirm how long evidence is kept and whether it is archived elsewhere.

Confirm that your instrumentation actually emits the distribution and quality metrics you need, because Prometheus only stores what the application sends it.

What meeting MEASURE 2.4 costs in money and effort

This table breaks down the cost components of running production monitoring for MEASURE 2.4, showing who is paid, whether each cost is one-time or recurring, and what moves it up or down. Use it to structure a written quote, since the figures depend on your own system, traffic and staffing rather than on any published price.

Cost component	Who is paid or who carries it	One-time or recurring	What drives it up or down
Instrumentation engineering	Internal platform or ML engineers, or an implementation partner for the metrics, traces and logs	One-time, with updates when the system changes	Number of model components, pipelines and tools to instrument; reusing a common telemetry library lowers effort
Alert and dashboard build	Engineers or SRE staff building rules for drift, anomalies and failures	One-time, then ongoing tuning	Number of monitored signals and alert routes; a short list tied to MAP risks keeps the build smaller
Baseline and threshold setting	Data scientists or TEVV staff comparing production metrics with pre-deployment test results	One-time per model version, repeated after major changes	Volume of historical data, number of input and output distributions, and how often the model is retrained
Ongoing monitoring operations and review time	Internal operations, model owners and risk staff reviewing alerts and dashboards	Recurring	Alert volume and false-positive rate, review cadence, and whether review roles are documented clearly
Ground-truth labeling	Internal domain experts or a labeling vendor assessing sampled outputs against newly collected ground truth	Recurring	Sample size, complexity of correct answers, and whether labeling can use existing business outcomes
External testing or review (optional)	Independent testers or assessors who review monitoring design or run deployment-like tests	Optional, per review cycle	Scope, system risk level and whether the work is done once or on a retainer; ask for a fixed-scope proposal
Auditor sampling effort	Independent auditor or assessor, billed under the audit engagement	Per audit	Number of monitoring records the auditor samples and how well alert history and review records are kept
Open-source licenses	No license fee for Prometheus, Grafana and Langfuse under their open-source licenses; hosting and staff time remain	Mostly recurring infrastructure	Hosting footprint, data retention period and storage volume; confirm current license terms with each project

Who does what for production monitoring

Your organization owns the monitoring program itself. MEASURE 2.4 asks that the system's functionality and behavior are monitored in production, but NIST does not prescribe who sets the limits or who acts on a signal. Those choices belong to the accountable owner of the AI system and to the teams that understand its use. Your team decides which production metrics matter, sets the thresholds that count as drift or anomaly, and decides who responds when an alert fires.

Thresholds and tolerances: Set drift and anomaly limits from your documented risk tolerance and from the pre-deployment baseline, and record who approved them and when.
Alert response: Name the on-call role, the escalation path and the criteria for pausing, rolling back or deactivating the system.
Human review decisions: Decide which outputs are sampled for review, who reviews them, and how reviewers record that an output was unreliable.
Ground truth collection: Own the process that gathers newly labeled outcomes so outputs can be assessed against them.
Documented ownership: Keep a current record of these decisions, including changes to limits after a incident.

Check that every alert has a named owner and a recorded response path before an assessor asks for it.",

Where production monitoring programs usually fall short

Most production monitoring programs watch the servers well and the AI outputs poorly. The gaps below cluster around what gets measured, who acts on it and whether results reach the people who decide whether the system keeps running. Each card names one gap and the check that closes it.

Uptime without output quality

Dashboards often show latency, error rates and container health while nothing tracks whether the model's answers are still correct. A system can return responses on time and still be wrong, stale or unsafe. For MEASURE 2.4, check that monitoring includes output-level signals such as sampled quality scores, refusal rates, and the behavior the MAP function identified as important. Ask which output metrics exist today and who reviews them.

Thresholds set once and never reviewed

Alert limits are often copied from a pre-deployment test and left in place for the life of the system. Input populations change, user behavior shifts and the model ages, so a threshold that was sensible at launch can hide real drift or fire constantly. Confirm that each threshold has a documented rationale, a baseline from pre-deployment testing, and a scheduled review date owned by a named person.

Third-party models and APIs out of scope

Teams frequently monitor their own service but treat calls to a hosted model API or a vendor-supplied model as someone else's problem. When the vendor changes a model version or its behavior, the organization still answers for the outputs. Include every external model, API and data feed in the monitoring inventory, record the version or identifier in use, and track vendor change notices as monitoring inputs.

Alerts with no named owner

An alert that routes to a shared channel with no accountable responder often gets acknowledged and forgotten. For MEASURE 2.4, every alert rule should name the role that receives it, the response time expected, and the escalation path if no one acts. Test the routing before go-live, because a silent alert is a monitoring gap that looks like a healthy system on the dashboard.

No ground-truth loop

Drift detection tells you that inputs or outputs have moved, but not whether the system is now wrong. Without a process to collect newly labeled examples and compare outputs against them, the program cannot confirm performance. Establish a sampling plan, a labeling procedure with reviewers who understand the domain, and a schedule for comparing results with the pre-deployment test set documented under MEASURE 2.1.

Monitoring results that stop before MANAGE

Monitoring that produces charts but no decisions does not manage risk. Results should flow into defined actions: rollback, retraining, restriction of use, human review or a documented decision to continue, which are the kinds of responses the MANAGE function addresses. Check that each alert class maps to a decision owner, a response procedure and a record of what was done, so monitoring evidence connects to governance outcomes.

Questions buyers ask about MEASURE 2.4 monitoring

Does NIST certify MEASURE 2.4 production monitoring?

No. The AI RMF has no certification, accreditation or conformity assessment program, and NIST does not validate or endorse any organization's use of the framework. MEASURE 2.4 is an outcome in the framework that your organization can show it achieves. An independent auditor or assessor may test that evidence against your own stated approach, but no one issues a NIST certificate for it. Ask any vendor or consultant who claims otherwise to show the basis for that claim in writing.

How much effort does production monitoring for an AI system take?

The effort depends on how many AI systems are in production, how many components each one has, and how much of the monitoring already exists in your logging and metrics stack. The main cost drivers are the number of data sources to instrument, the baselines you must establish from pre-deployment testing, the number of alert rules and their tuning, and who reviews flagged outputs. Setup is largely one-time work, while alert tuning, baseline refresh and human review recur for as long as the system runs. Ask any quote to break out instrumentation, alert design, review staffing and ongoing operations separately.

Can open-source tools handle AI production monitoring?

Open-source tools can cover much of the technical work. Prometheus collects metrics over HTTP and stores them as time series that you can query with PromQL. Grafana Alerting builds alert rules from queries across multiple data sources and handles metrics and logs together. Langfuse captures traces of LLM calls, retrieval steps and tool executions with inputs, outputs and token usage, which supports generative AI monitoring. Tools for drift statistics or ground-truth comparison may need custom code or other components. Confirm each tool's coverage against your own system before relying on it.

What must the team provide to monitor AI systems in production?

The team must provide the pre-deployment baselines, the list of AI components identified during MAP, the metrics to monitor, the alert thresholds and owners, and a process for human review of unreliable outputs. Engineers need to instrument the system so that inputs, outputs and component behavior are recorded. Operations staff need an escalation path and a record of what happened when an alert fired. Without documented ownership for each alert, the monitoring produces data but no evidence of response.

Will an auditor accept production monitoring as MEASURE 2.4 evidence?

An auditor or assessor decides whether your evidence is sufficient for the criteria your engagement covers, and no auditor issues a MEASURE 2.4 certificate. Assessors usually look for documented monitoring plans, metric definitions, baseline comparisons, alert configurations and records of alerts and reviews. Evidence is stronger when it links back to the risks identified in MAP and shows what happened after an alert. Confirm the assessor's scope and expectations in writing before you collect evidence.

How does generative AI change production monitoring?

Generative AI adds risks that numeric drift metrics do not capture well, such as hallucination, data leakage, prompt injection and toxic output. NIST AI 600-1 lists no suggested actions for MEASURE 2.4, so your team has to choose how to address these risks itself. Monitoring usually adds trace-level logging of prompts, retrieval results and tool calls, plus sampled human review of outputs. Garak can probe language models for several of these weaknesses and writes a report of attempts and vulnerabilities, which can support pre-production and periodic testing. Because prompt and output logs may contain personal or confidential data, confirm your data retention and access rules first.

When should an organization bring in an implementation partner for monitoring?

Consider a partner when your team lacks the time or skills to instrument systems, design alerts and build evidence packs, or when you need an independent view before an assessment. DeployOpen prepares systems, controls and evidence for compliance assessments, deploys self-hosted open-source platforms inside your own infrastructure, and runs security testing. It does not issue audit reports or certifications. Most engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer. Ask for a written scope that names who owns each control and each piece of evidence.

How DeployOpen scopes and prepares MEASURE 2.4 evidence

DeployOpen starts MEASURE 2.4 work with a fixed scope agreed in writing before any build begins. The scope should name each AI system in production, the risks identified in MAP that monitoring must cover, the telemetry sources to instrument (model gateways, retrieval services, tool calls, application logs), the alert recipients and the human reviewer for unreliable outputs. Deliverables and timelines follow from that written scope, and a quote follows from the same document. Before agreeing scope, confirm with your own team who owns the production system, who receives drift alerts, and whether pre-deployment baselines from MEASURE 2.3 exist in a form the monitoring stack can compare against.

Instrumentation and monitoring are then built inside your own cloud account, data center or air-gapped network. A typical build uses Prometheus to collect metrics over HTTP and store them as time series, Grafana Alerting to define rules across metrics and logs, and Langfuse to capture traces of LLM calls, retrieval steps and tool executions with inputs, outputs and token usage. Where a generative system is in scope, garak can be run against a test endpoint to produce a JSONL report of every attempt and a hit log of successful probes. Your security and platform teams keep administrative control of these tools, and the configuration, alert thresholds and retention settings should be reviewed and approved by your accountable owner.

Evidence is packaged for your assessor, which may be an internal audit function, a customer or a regulator. DeployOpen prepares the monitoring configuration, alert history, drift reports, review logs and the documented decisions taken on alerts, mapped to the MEASURE 2.4 suggested actions in the Playbook. DeployOpen can also run optional security testing of AI systems and agents and offer a retainer for ongoing review of drift and alert trends. DeployOpen prepares systems and evidence but does not issue audit reports, attestations or certifications. Those come from an independent auditor or assessor, and no AI RMF certification exists, so confirm with your reviewer which form of evidence they will accept.

MEASURE 2.4 Production monitoring