MEASURE 4.3 Measurable performance changes: NIST AI RMF

What NIST AI RMF MEASURE 4.3 asks you to document after an AI system changes

MEASURE 4.3 asks you to identify and document whether an AI system's performance has measurably improved or declined after a change. Picture a team that swaps a model version, rewrites a system prompt, reindexes a retrieval source or reroutes a workflow. Before anyone calls the update a success, the team has to show whether trustworthiness got better or worse for the people the system affects. MEASURE 4.3 is an outcome in the Measure function of NIST AI 100-1 (Table 3), sitting under MEASURE 4, which covers feedback on measurement efficacy. The outcome expects those changes to be based on consultation with relevant AI actors, including affected communities, and on field data about context-relevant risks and trustworthiness. The risk this addresses is silent degradation. A change can keep uptime and latency steady while answers become less accurate for one user group, or while a retrieval update quietly lets sensitive content reach users who should not see it. Teams often monitor availability and cost but not trustworthiness, and they frequently lack a baseline to compare against. Without a documented pre-change measurement, nobody can say what "worse" looked like, so regressions go unnoticed until a complaint or incident surfaces them. Confirm early whether your current dashboards record any trustworthiness measure at all, and which version of each measure was in effect before the change.

NIST AI RMF is voluntary and has no certification, so no body will stamp a change as compliant. Passing means producing evidence that an internal reviewer or a customer assessor can examine and reproduce. In practice, that evidence is a before-and-after record: the baseline quantitative measures for each trustworthiness characteristic, the qualitative findings that complement them, the sensitivity analysis run after the update, and the decision that followed. The Playbook suggests performing and documenting sensitivity analysis of performance variance after system or procedural updates, which is the step most often skipped. Decide who signs off on the comparison, who from affected communities or domain experts was consulted, and where the record is stored. The NIST AI RMF Playbook lists suggested documentation for MEASURE 4.3, and the Playbook itself is not a checklist, so treat its prompts as a starting point for your own records.

Which AI systems, updates and teams fall under MEASURE 4.3

MEASURE 4.3 only works if the organization first decides what counts as a system, which changes force a new measurement, who must be consulted and which trustworthiness characteristics get a baseline. Each decision below changes the evidence you need, so settle them in writing before the first measurement run.

Systems in scope, including vendor-hosted models

Decide whether the unit of scope is the deployed AI system, the underlying model or the business workflow that uses it. Vendor-hosted models, third-party APIs and pre-trained models still shape the system's behavior, so they belong in scope even when you cannot see their training data or internal changes. Record which parts you control, which the vendor controls, and what the vendor notifies you about. Confirm with each vendor how and when it announces model changes, since your baseline depends on that notice.

Model version and provider changes

A new model version, a silent provider update or a change of hosting region can move performance without any change on your side. Treat each one as a trigger for a documented measurement against your baseline. Keep the exact model identifier, version and deployment date in your records so you can show which behavior a measurement describes. Ask vendors whether they publish version histories and deprecation dates.

Prompt, policy and retrieval corpus changes

Prompts, system instructions, guardrail policies and retrieval corpora change more often than models do, and each can shift outputs. Decide which edits count as material, for example a change to refusal rules or to the documents a retrieval step can surface. Log each edit with its author and date, then rerun the affected measurements. The NIST Playbook suggests a sensitivity analysis after such updates, so plan for it before the change ships.

Tool access and operating population

Expanding the tools an AI system can call, or the users and populations it serves, changes the risks that matter. A system that moves from drafting text to executing actions, or from internal staff to members of the public, needs new measurement even if the model is unchanged. Define the operating population in writing, including new customer segments, languages and geographies. Review tool permissions with the people who own the downstream systems.

Actor categories that must contribute

The Playbook tags MEASURE 4.3 with TEVV, AI Deployment, Operation and Monitoring, End-Users, and Affected Individuals and Communities. Name an accountable owner for each category, since each contributes different evidence: testers measure, deployers record configuration, operators report field behavior, and end users and affected communities describe harms that metrics miss. Document how each group was consulted, when, and what changed as a result. Appendix A of NIST AI 100-1 describes the tasks of each category.

Trustworthiness characteristics to baseline

AI 100-1 lists seven trustworthy characteristics: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. Baseline the characteristics tied to the risks you mapped as most significant, not all seven by default. Pair each chosen characteristic with a quantitative measure and a qualitative check, and record why the others were deferred so an assessor can follow the reasoning.

How to set up performance change documentation step by step

This sequence turns MEASURE 4.3 into a working routine: it sets baselines, adds qualitative input from people affected by the system, runs sensitivity analysis after each change, and records what moved and what was decided. The Playbook suggests these actions, and the organization chooses how deep to go.

Define the trustworthy characteristics in scope
Start by listing which of the seven trustworthy characteristics matter for this system, such as valid and reliable, safe, secure and resilient, or fair with harmful bias managed. The system owner and the TEVV lead agree on this list, using the risks already identified under MAP and MEASURE 1. The output is a short scope record that names each characteristic, the context-relevant risk it addresses, and the person accountable for its measurement. The step is done when every mapped high-priority risk has at least one characteristic linked to it.
Establish baseline quantitative measures
For each characteristic in scope, the TEVV team sets a baseline quantitative measure taken from the system as it operates in its deployment context, not only from benchmark data. Record the metric definition, the data used, the test conditions, the date and the software or model version. Metrics can be collected with tools such as Prometheus, which stores time series by metric name and labels, or Langfuse traces for generative AI applications. The output is a baseline register that someone outside the build team could reproduce.
Add qualitative approaches alongside metrics
Numbers rarely show how a change feels to the people using the system, so pair each baseline metric with a qualitative method. Options include structured expert review of sampled outputs, annotated case reviews, and reviews of complaints or overrides logged by operators. The domain experts who validate results under MEASURE 4.2 can run these reviews. The output is a qualitative review protocol with its sampling method and a rating scale, stored with the baseline records so both views can be compared later.
Set up consultation with affected communities
Name who gathers input from affected communities and end users, how often, and through which channels, such as feedback forms, user panels or community advisory sessions. Decide how field data on context-relevant risks is collected, including incident reports from operators and end-user problem reports. Record which concerns were raised and how the team responded, since MEASURE 3 expects the same reporting channels after deployment. Confirm the approach with your legal and privacy team before collecting personal data. The output is a consultation plan with named owners and a log template.
Run sensitivity analysis after each update
After every system or procedural update, such as a model swap, a prompt change, a retraining run or a new retrieval source, rerun the baseline metrics under the same test conditions. The Playbook suggests performing and documenting sensitivity analysis of performance variance, so compare the new results with the baseline and identify which inputs or conditions drive the difference. Store the raw results, such as the JSONL report produced by garak for language model probes, alongside the summary. The output is a dated analysis showing the variance for each characteristic in scope.
Write the change record and decision
Close each update with a change record that states the measured direction of change for each characteristic, whether it improved, declined or stayed within the expected range, and cites the baseline, the analysis and the qualitative review. Record the decision taken, such as accepting the change, adding a monitoring alert, restricting use, or rolling back. Set alert thresholds in a tool such as Grafana Alerting so declines are visible. An independent assessor may review these records, but only the organization owns the decision. The output is a signed change record linked to the version it describes.

How an assessor tests MEASURE 4.3 evidence

This table compares five areas an assessor examines for MEASURE 4.3, the approach used for each, the evidence requested and the findings that usually come up. Read each row as a test an independent reviewer can repeat, and use the findings column to check your own records before a review.

Test area	Test approach and evidence requested	Common findings
Baseline definition	Assessor asks for the baseline quantitative measures for each trustworthy characteristic in scope, the date they were fixed, the data and test set used, and who approved them. They check that the same baseline is reused across later releases.	Baselines were never frozen, so each release was compared against a different reference. Baselines exist for accuracy but not for fairness, safety or other characteristics in scope.
Link between consultation and recorded changes	Assessor samples consultation records (affected community input, domain expert notes, end-user reports) and traces each item to a specific recorded change, metric movement or decision. They check that the trace runs in both directions.	Stakeholder input is logged but never tied to a change. Performance changes are described only in release notes with no reference to the feedback that prompted them.
Sensitivity analysis after updates	Assessor selects recent model, prompt, data or procedure updates, not only the initial release, and asks for before and after results, the variance analysis performed and the documented interpretation of any decline.	Only the initial release was tested. Updates to prompts, retrieval sources or model versions shipped without a documented variance check, so declines went unrecorded.
Field data on context-relevant risks	Assessor asks for production monitoring outputs (for example, traces, error logs, alert history and user reports) for the deployment context, and checks whether those signals feed back into the measured metrics.	Monitoring collects data that no one reviews against the baseline. Alerts fire but there is no record of whether the measured performance changed as a result.
Decision record	Assessor reviews the documented decision for each recorded change: whether it was accepted, rolled back or treated, who decided, and what evidence supported the decision. They confirm the record exists for the sampled changes.	Decisions are made in meetings or chat threads with no retained record. The record does not state the measured change that justified the decision.

Evidence that shows a documented performance change

MEASURE 4.3 asks for performance changes to be identified and documented, drawing on stakeholder consultation and field data. The six artifacts below form one evidence set, and each one should be refreshed when a specific event occurs rather than on a fixed calendar, since the framework does not set a review cadence for them.

Frozen baseline record

The baseline is the set of quantitative measures recorded for each trustworthy characteristic before a change, with metric definitions, test data versions and the conditions under which they were captured. Assessors use it as the reference point for any claimed improvement or decline. Refresh it when the system is updated, when it is redeployed into a new context, or when a metric definition changes, and never overwrite the prior version. Confirm that each metric can be reproduced from the stored definition.

Sensitivity analysis report per update

The Playbook suggests performing and documenting sensitivity analysis of performance variance after system or procedural updates. Each report should state which inputs were varied, the resulting range in each measure and which changes were judged material. Produce a new report for every update to the model, prompts, retrieval sources or operating procedure, and check whether the results still hold for each affected user group.

Stakeholder consultation log linked to changes

This log records who was consulted, including affected communities and end users, what they raised and how the input was weighed. The point of the log is traceability: every consultation should link to the measurement change it informed. Add entries when a consultation occurs, which is usually triggered by an update, a new deployment context or a reported incident. Ask whether the log captures dissenting views as well as agreement.

Field data summary on context risks

This summary condenses operational data about the risks and trustworthiness characteristics that matter in the deployment setting, such as error patterns, complaints, overrides and unexpected outputs. It shows whether measured performance holds up outside the test environment. Refresh it when the deployment context changes, when a new user population is onboarded, or after an incident, and state the data window and collection method on each version.

Change record with the decision taken

Each change record names the performance improvement or decline identified, the evidence behind it and the decision made, such as accepting the change, rolling it back, restricting use or escalating to governance. It connects the baseline, sensitivity analysis, consultation and field data to a named owner. Create one for every material change and reopen it when a later incident shows the decision was wrong. Confirm that the approver is the accountable role defined under GOVERN 2.

Monitoring dashboard export and metric history

Exports from monitoring tools provide the time series that show whether performance moved after a change. Tools such as Prometheus store metrics as time series queried with PromQL, and Grafana Alerting can alert on metrics and logs together. Export the relevant panels and the query definitions with each change record so the trend can be reproduced. Confirm retention periods for each store, because a metric that has aged out cannot support a later review.

Why generative AI systems need their own baselines for MEASURE 4.3

NIST AI 600-1, the Generative AI Profile (July 2024), lists no suggested actions for MEASURE 4.3, and it states that not every AI RMF subcategory is included. A team running a language model therefore cannot copy a profile action and call MEASURE 4.3 covered. It has to design its own baselines for the trustworthiness characteristics that matter in its deployment, document how those baselines were chosen, and show how they are compared over time. The Playbook suggestions for MEASURE 4.3 remain the starting point, but they are general, and the team must decide which measures apply to its model, prompts, retrieval sources and tools.

Generative outputs usually need both quantitative measures and qualitative review. A numeric score can show that a test set still passes, but it may miss a fabricated citation, an answer that is technically correct yet misleading for the user, or a change in tone that affects affected communities. Pair automated scores with structured human review of a sample, and record who performed the review and against which criteria. Keep the raw evaluation outputs as well as the summaries. Tools such as garak can write a JSONL report of every attempt and evaluation, and a hit log of attempts that found a vulnerability, which gives the team a record to compare across runs.

A prompt, system instruction, retrieval index or tool configuration can change behavior without any change to the model weights, so the baseline must be re-run after those changes, not only after model upgrades. Decide which risks the organization baselines, for example hallucination, prompt injection, toxicity or data leakage, and write down the test set, metric and threshold for each. Define in advance what a meaningful decline means for that system, and who is accountable for acting on it. Confirm with your own evaluation and monitoring owners how traces from tools such as Langfuse, or metrics from Prometheus, will link runs to specific configuration versions.

Which self-hosted open-source tools can produce MEASURE 4.3 evidence

MEASURE 4.3 asks for measurable performance changes to be identified and documented. The four tools below each produce a different slice of that evidence: adversarial test results, application traces, alert history and metric trends. Together they give a team data to document, but the organization still decides what a change means and records that decision.

garak probe reports

garak is an Apache-2.0 scanner that probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. It writes a JSONL report of every attempt and evaluation, plus a hit log of attempts that found a vulnerability. Run the same probe set before and after a model, prompt or configuration update, then compare the reports. Check that the probe set matches the risks mapped for your context, since garak does not decide which risks matter to your system.

Langfuse traces for runtime behavior

Langfuse is an MIT-licensed observability tool. Each trace captures every operation in a request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. Traces let a team compare behavior before and after a change, such as slower retrieval or higher token use after a prompt revision. Confirm that your logging policy allows capturing inputs and outputs, especially where personal data is involved.

Grafana alert rules across metrics and logs

Grafana Alerting builds alert rules from queries and expressions across multiple data sources, and it alerts on metrics and logs together from a single view. Use it to set thresholds for performance declines and to keep a record of when those thresholds fired. Grafana is licensed under AGPL-3.0-only. Before you distribute modified Grafana code or offer it as a network service to others, have counsel confirm that the license fits your distribution model.

Prometheus metric history

Prometheus is an Apache-2.0 system that collects metrics from instrumented jobs through a pull model over HTTP. It stores them as time series identified by metric name and labels on local storage, and queries them with PromQL. Use PromQL to compare accuracy, latency or error-rate series across a release boundary and to check for sustained shifts rather than one-off spikes. Confirm your retention settings, because local storage limits how far back a team can compare.

What makes MEASURE 4.3 cost more or less

A single internal model with few updates is usually the lowest-effort case for MEASURE 4.3. The work centers on setting a baseline for each trustworthy characteristic the organization cares about, recording it, and repeating a sensitivity check when the model or its procedure changes. Because updates are rare, the number of comparisons stays small and the team can review each one by hand.

Baseline setup: One set of quantitative measures plus a qualitative review is usually enough to start, and it is a one-time effort that recurs only when the measures change.
Update frequency: Few update types mean few sensitivity analyses per year, so review time stays predictable.
Log quality: Prometheus can collect metrics from the serving stack, and Grafana Alerting can flag drift, but the team still reviews the output and writes the findings down.
Field data: A small internal user group may provide labeled feedback, which keeps labeling cost low but needs a named owner.
License spend: Open-source monitoring removes license fees, while engineering setup and documentation time remain.

Confirm with your team how many update types you expect each year and who signs off on the documented results.

What meeting MEASURE 4.3 costs in money and effort

This table lists the cost components of building and keeping up MEASURE 4.3 evidence, who is paid for each, whether it is one-time or recurring, and what drives it up or down. No price or duration is given because none has been supplied for this topic, so use the drivers and quote checklist below to compare written proposals. Open-source tools remove license spend but not the engineering and review effort that the evidence requires.

Cost component	One-time or recurring	What drives it	How to keep it down
Tooling or licenses	Recurring for commercial tools, none for the open-source options named on this page	Self-hosted tools such as Prometheus, Grafana, Langfuse and garak remove license fees, but hosting, storage and upgrades still need staff time	Standardize on tools already in the environment and confirm which components a partner will support after handover
Engineering time to instrument baselines and logs	One-time build, with recurring changes when systems change	Number of systems, number of trustworthy characteristics measured, and how many metrics lack existing instrumentation	Start with the highest-impact systems and reuse one metric and logging pattern across them
Stakeholder consultation time for operators and affected communities	Recurring, tied to each review cycle	Number of affected groups, whether consultation is interviews, surveys or panels, and the translation or accessibility work needed	Use a fixed consultation method with a documented question set and reuse feedback categories across reviews
Sensitivity analysis effort per update	Recurring, once for each system or procedural update	Frequency of model, prompt, data or configuration changes and the number of variables tested	Define which update types trigger a full analysis and which need only a baseline comparison
Evidence upkeep between audits	Recurring	Retention period for measurement records, volume of logs and traces, and the effort to keep the documentation linked to each change	Automate report generation from stored metrics and agree a retention period that matches legal and contractual needs
External testing or review	Recurring or per engagement	Scope of red teaming or validation, depth of review of consultation results, and whether a third party checks the performance comparisons	Agree a written scope with fixed deliverables and test only the systems that carry the highest risk
Auditor or assessor sampling	Recurring, set by the assessor's plan	Number of systems and updates in the period, and how well records link each performance change to its trigger and outcome	Ask the assessor to confirm the sampling approach early and prepare an indexed evidence set before fieldwork

Who does what: your team, a partner and the independent assessor

Your organization owns MEASURE 4.3. The outcome requires that performance improvements or declines are identified and documented, based on consultation with relevant AI actors, including affected communities, and on field data about context-relevant risks and trustworthiness. Those judgments belong to people accountable for the AI system, not to the people who built the dashboards. NIST does not validate or endorse any individual organization or its approach to using the AI RMF, so the record of what your team decided and why is the core evidence.

Defining meaningful change: Set the thresholds that count as a material improvement or decline for each trustworthiness characteristic, and approve them before results arrive.
Running consultation: Hold and document the conversations with end users, operators and affected communities that inform what the metrics should capture.
Deciding on baselines and sensitivity analysis: Approve the baseline quantitative measures and the sensitivity analysis performed after system or procedural updates.
Signing decision records: Record each identified change, the evidence behind it, and the response, with a named accountable owner.

Before any outside help starts, confirm which of these decisions your governance policy already assigns to a named role.

Common questions about MEASURE 4.3 cost, effort and evidence

Is MEASURE 4.3 a certification requirement?

No. MEASURE 4.3 is one outcome in the NIST AI Risk Management Framework, which is a voluntary framework. NIST does not certify organizations against it, does not validate or endorse any individual organization's use of it, and describes no accreditation or conformity assessment program for it. An organization can be asked by a customer, contract or regulator to show this outcome is met, but no certificate exists for it. Any independent assessment is a review of your own evidence, and only an independent auditor or assessor issues a report on it. DeployOpen prepares systems and evidence for that review but does not issue reports or certificates.

How much effort does MEASURE 4.3 take, and what drives it?

Effort depends mainly on how many AI systems are in scope, how many trustworthiness characteristics you measure, and how mature your logging and monitoring already are. The outcome asks you to identify and document measurable changes in performance, which means you need baselines, a record of each update, and a way to compare results before and after changes. Effort rises when models or prompts change often, when logs are scattered across tools, or when consultation with affected communities has to be arranged from scratch. It falls when baselines already exist and changes pass through a documented release process. No effort figure is supplied for this outcome, so ask any partner to size the work against your actual system inventory.

How long does MEASURE 4.3 implementation take?

Timing depends on scope, the number of systems, the state of existing monitoring, and how quickly your teams can complete consultation with relevant AI actors. No duration is supplied for this work, so none is given here. Ask for a timeline in a written plan before signing, and check that it lists the systems covered, the milestones for baselines and sensitivity analysis, the consultation steps, and the dates on which evidence will be handed over. A fixed-scope agreement should state these items explicitly so that both sides can see what is included.

What does the work need from our team?

The work needs named owners, access and consultation time. Assign an accountable owner for each in-scope system, a technical owner who can explain its updates, and a contact who can arrange feedback sessions with domain experts, end users and affected communities. Engineers must give read access to monitoring data, traces and logs, and the team must keep a record of every model, prompt or procedural change. Consultation time is often the constraint, so agree on how many sessions are needed and who attends them before the work starts.

Can we do MEASURE 4.3 in-house?

Yes, an organization can do this work in-house, because the framework is voluntary and the outcome describes practices rather than a fixed toolset. In-house teams usually need people who understand the AI system, monitoring engineers who can build baselines, and staff who can run consultation with affected parties. An outside partner may help where the team lacks monitoring or testing skills, where the timeline is tight, or where an independent reviewer is needed to check the work. Whichever route you choose, the organization owns the controls and the decisions, and an independent assessor is the only party who issues any report.

What will an assessor sample when reviewing MEASURE 4.3?

An assessor typically samples a set of in-scope systems and checks whether documented baselines exist, whether changes were measured, and whether results were reviewed with the right people. Expect requests for baseline metric definitions, records of updates with before-and-after results, sensitivity analysis reports where performance was tested after a change, consultation records showing who was asked and what they said, and the decisions taken in response. The exact sample and procedures depend on the assessor and the requesting party, so confirm them in advance. Keep evidence organized by system and by date so that each sampled change can be traced end to end.

What should we ask a partner before signing a fixed-scope agreement?

Ask for a written scope that lists each system, the outcomes covered, and the deliverables, plus a timeline with milestones. Confirm which party owns each control and each piece of evidence, and what the partner will not do, such as issuing reports or certificates. Ask how the partner will access logs and monitoring data, how consultation will be arranged, and how changes after handover are handled. Request the rules for scope changes, acceptance criteria for each deliverable, and the terms for data handling and confidentiality. Ask which open-source tools will be installed in your environment and what they will record, and confirm each claim against the vendor documentation before you rely on it.

How DeployOpen scopes and prepares MEASURE 4.3 evidence

DeployOpen helps organizations set up the measurement systems, baselines and documentation that MEASURE 4.3 asks them to keep, inside their own infrastructure. The organization owns its AI risk decisions and its conclusions about performance changes, and an independent assessor is the party that tests and reports on them.

How we scope

Agree in writing which AI systems, versions and deployment contexts the MEASURE 4.3 work covers before any build starts.
Identify the trustworthiness characteristics, metrics and data sources that the organization will track for each in-scope system.
Confirm which AI actor groups, including affected communities where relevant, will be consulted and how those consultations will be recorded.

How we build

Deploy self-hosted monitoring for metrics, traces and alerts, such as Prometheus, Grafana Alerting and Langfuse, in the client's environment.
Configure baseline quantitative measures and qualitative review routines, then capture pre-update and post-update results for sensitivity comparison.
Where the system is a language model, set up garak runs and retain the JSONL reports and hit logs as dated test evidence.

How we hand over

Deliver written runbooks that explain how each baseline was set, how updates are compared and who reviews a performance change.
Provide a documentation pack with metric definitions, consultation records, sensitivity analysis results and the decision log for measurable changes.
Walk the client's team through the dashboards, alert rules and evidence locations so they can produce the record without us.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover, sized to the written scope.
Review measurement outputs with the client's team at agreed intervals, so changes in performance are documented as they occur.
Confirm with your chosen independent assessor which evidence formats and retention periods they expect before the build begins.