What MEASURE 2.13 asks you to prove about your AI metrics
When an auditor, customer questionnaire or internal review asks whether your AI measurement is trustworthy, MEASURE 2.13 is the subcategory that checks the measurement itself. Its outcome is that the effectiveness of the TEVV metrics and processes used in the MEASURE function is evaluated and documented. In plain terms, you must show that the tests, metrics and test, evaluation, verification and validation (TEVV) processes you rely on still work: they should keep sustaining improvements, stay useful to the people who read them, and actually detect the risks they were chosen for. The risk being addressed is a dashboard that looks healthy while it misses real failures, drift or harm. A metric can be stable for months and still measure the wrong thing.
This is hard because it is a meta-check. You are not only producing results, you are producing evidence that the results can be trusted. The Playbook suggests asking whether metrics can sustain error removal, whether they are useful enough to justify their cost, and whether end users and affected communities accept them. Your evidence might include a record of metric reviews, comparisons between metric results and confirmed incidents or user complaints, and notes on why a metric was retired or simplified. The Playbook is voluntary and its audit log shows its last content update in August 2023, so confirm with NIST's current AI RMF Playbook how its Measure actions are presented before you cite them.
For generative AI, the Generative AI Profile in NIST AI 600-1 adds one suggested action for MEASURE 2.13, MS-2.13-001, tied to confabulation, harmful bias and homogenization, and information integrity. It asks for measurement error models for pre-deployment metrics that demonstrate the construct validity of each metric. Practically, that means showing that a metric truly measures the property it claims to, such as hallucination, not a proxy that correlates with it. NIST AI 100-1 is voluntary and there is no AI RMF certification, so no body will issue a pass for this subcategory. Your assessor, customer or regulator will judge the documented evidence you bring, and you should confirm with them which evidence format they expect.
Which metrics, systems and teams fall under MEASURE 2.13
MEASURE 2.13 asks whether the TEVV metrics and processes used across the Measure function are themselves working. Before judging effectiveness, an organization needs a clear boundary: which systems, metric sets, TEVV processes, vendor inputs and actor categories the review covers. The cards below set that boundary and show who owns each part.
In-scope AI systems and metric sets
Start with an inventory of the AI systems whose metrics feed MEASURE decisions, then list the metric sets each system relies on for validity, safety, security, fairness and other characteristics. Record which metrics were chosen for the most significant risks identified in MAP. Metrics outside that inventory, such as ad hoc dashboards that no one reviews, are the most common gap. Confirm that every metric set has a named owner and a documented purpose before the effectiveness review begins.
TEVV processes that count
Decide which test, evaluation, verification and validation activities fall under the review: pre-deployment testing, deployment-like testing, red-team exercises, human subject evaluations and production monitoring. The question for MEASURE 2.13 is whether each process still identifies risk, not only whether it runs. Document the process steps and the triggers for rerunning them, so the reviewer can tell whether a missed risk came from a metric or from a process that was skipped.
Third-party models and vendor metrics
Decide whether metrics supplied by model providers, data vendors or security tool suppliers are in scope. Vendor benchmark results are inputs, not proof that a metric works in your context. Ask each vendor which metrics it reports, how they were measured and what changed between versions, then record whether your organization has validated them against your own deployment conditions. Contract terms that give access to test documentation should be confirmed before relying on vendor figures.
TEVV team ownership
The AI actor category TEVV owns the design, execution and documentation of the metrics. NIST's Playbook tags MEASURE 2.13 to TEVV alongside AI Deployment and Operation and Monitoring, and AI 100-1 Appendix A describes the tasks of each category. Assign the TEVV team to test whether metrics sustain improvement and identify errors, and confirm that the team has authority to recommend retiring or replacing a metric.
Deployment and operation ownership
AI Deployment and Operation personnel own whether metrics still match how the system is used in production. They can report whether end users find a metric understandable and acceptable, and whether affected communities have raised concerns that the metrics miss. Confirm a written route by which operators send these observations to the TEVV team, so that metric problems found in daily use reach the review cycle rather than staying with the people who noticed them.
Monitoring ownership and generative AI metrics
Monitoring personnel own the signals that show whether metrics still identify risk after launch, such as drift, anomalies and alerts. For generative AI, NIST AI 600-1 lists one suggested action under MEASURE 2.13 (MS-2.13-001) and asks for measurement error models that demonstrate the construct validity of pre-deployment metrics, tied to confabulation, harmful bias and homogenization, and information integrity. Confirm which trace data and test reports the monitoring team retains so that validity questions can be answered later.
How an assessor tests MEASURE 2.13
This table sets out what an assessor typically checks for each common question about MEASURE 2.13, the evidence they request and the findings that usually signal weak metric governance. Read it as a guide to what an independent reviewer looks for, not as a certification test, since the AI RMF has no assessment program and the organization owns its own conclusions.
| Assessor question | What the assessor typically looks for | Evidence usually requested |
|---|---|---|
| Sampling of metrics | A sample of the metrics used in MEASURE for the most significant mapped risks, checked for whether each is still in use and tied to a documented risk | Metric register, mapping from MAP risks to metrics, list of metrics retired or added in the review period |
| Metric revision history | Dated changes to thresholds, formulas, test sets and datasets, with reasons, so the assessor can see whether metrics improved or were quietly relaxed | Version history in the metric repository, change tickets, approval records for each revision |
| Error and drift handling | Proof that errors found by metrics led to corrective action and that the metrics themselves were adjusted when they stopped catching problems | Incident or defect records linked to metric results, drift reports, before and after measurements after a fix |
| Utility and acceptability review | A documented check that each metric is useful, understandable to decision makers and acceptable to end users and affected communities | Utility review memos, stakeholder feedback records, decisions to simplify or replace complex metrics |
| Independent challenge | Whether reviewers outside the team that built the metrics questioned their validity, coverage and construct fit | Review minutes, independent reviewer findings, responses and closure status of each finding |
| Post-deployment relevance | Evidence that metrics were revisited after deployment and still identify the risks they were chosen for | Scheduled review records, production monitoring results compared with pre-deployment metric results |
| Typical finding: no revisits | Metrics selected before launch with no review record afterward, even though the system or its data has changed | Absence of review records is itself the evidence gap the assessor reports |
| Typical finding: no utility review | Metrics documented as adequate with no written check of utility or of end user acceptance | Missing utility review, no stakeholder sign-off, no documented decision on overly complex metrics |
How to implement MEASURE 2.13 in sequence
This sequence turns MEASURE 2.13 into a repeatable review of the TEVV metrics and processes behind the Measure function, so that each metric is shown to sustain improvement, remove errors, stay useful and be acceptable to the people it affects. The output is a documented record that an internal reviewer or independent assessor can follow from the metric to the decision it informed.
Inventory every metric and TEVV process
Start by listing each metric, test set, evaluation method and TEVV tool used in the Measure function for the AI system, including the outputs of MEASURE 1 and the risks selected for measurement. The TEVV team owns the list, with input from the deployment and operations team, since they know which metrics actually drive monitoring. The step is complete when every metric has an owner, a stated purpose, the risk it addresses and the place where its results are recorded. Gaps usually appear here, such as metrics that run in a notebook but feed no governance record.
Define effectiveness criteria for each metric
For each inventoried metric, write down how its effectiveness will be judged against the four questions in the Playbook suggestion: whether it sustains improvement and removal of errors, whether it has utility for the decision it supports, and whether end users and affected communities find it acceptable. The TEVV lead drafts the criteria and the accountable owner from the governance function approves them before results are collected. The output is a criteria table with a pass condition and a named reviewer for each metric, which prevents the team from choosing success measures after seeing the results.
Build measurement error models for pre-deployment metrics
For pre-deployment metrics, and especially for generative AI systems, build a measurement error model that shows how much of each metric's reading comes from the construct it claims to measure and how much comes from noise, test set choice or evaluator variation. NIST AI 600-1 ties this work to MS-2.13-001 and to demonstrating the construct validity of each metric. The TEVV team performs the modeling and a second reviewer checks the assumptions. The output is a short model note per metric that states its known error sources and the conditions under which the metric should not be trusted.
Test each metric against its stated criteria
Run the TEVV processes and collect results against the criteria defined in the earlier step, including checks for whether improvements hold across repeated runs and whether the metric separates good and poor outputs. Where a metric depends on human judgment or user feedback, record who was asked and how the feedback was adjudicated. The output is a results log per metric that records the test date, the version of the system and metric, and a clear pass, fail or inconclusive outcome against each criterion. Results that fail should not be overwritten, because they are evidence of the metric's limits.
Review and retire overly complex metrics
Where a metric is hard to explain, hard to reproduce or does not change decisions, decide whether to replace it with a more descriptive measure that operators and affected parties can read. The review is led by the TEVV lead with the operations team, who must confirm that the simpler measure still tracks the risk. Retirement needs a recorded reason, a note on any gap in coverage and a plan for keeping historical results comparable. The output is a decision entry for each metric kept, simplified or retired, together with the replacement where one is chosen.
Check acceptability with end users and affected communities
Confirm that the metrics the organization keeps are understood and accepted by the people who use the system and by communities it affects, using the feedback channels already set up under GOVERN 5 and MAP 5. Document what the users and communities said, including objections to particular measures, and whether the team changed the metric, the threshold or the reporting in response. The output is a feedback record that links each accepted or changed metric to the input that shaped it, so an assessor can see that acceptability was tested rather than assumed.
Record decisions and schedule re-evaluation
Bring the outcomes of the previous steps into one governance record that states, for each metric, its effectiveness assessment, the decision taken and the accountable approver. Set the conditions that will trigger a new review, such as a change in the system, a new risk identified in production or a revision of the AI RMF or Playbook, and link the record to the production monitoring evidence used under MEASURE 2.4. The organization owns this record and its approvals. Where an independent assessor later tests the program, the record is the starting point for the evidence trail, and the implementation partner can prepare it but does not sign off on its conclusions.
Evidence that shows metrics are effective and kept current
MEASURE 2.13 asks whether the metrics and TEVV processes themselves work, not only whether the system scores well. The six artifacts below form one record: the definitions, the reviews that test them, the error and construct validity work behind them, the change history, the production evidence that they catch real problems, and the calendar that keeps the cycle running.
Metric definitions and rationale
Keep a register for every metric used in MEASURE 2 testing. Each entry should state what the metric measures, the risk it is meant to detect, the test set and tool that produce it, the threshold that triggers action, and why this metric was chosen over alternatives. Without the rationale, a reviewer cannot tell whether a metric was selected for the risk or simply because it was easy to compute. Update the entry whenever the test set or tool changes, and link it to the MAP risk it covers.
Utility and acceptability reviews
For each metric, record a dated review that asks three questions: can the metric sustain improvement and the removal of errors, is it useful enough to justify its cost, and do end users and affected communities accept it as a fair description of the system. The Playbook suggests preferring descriptive approaches to overly complex ones, so note where a simpler metric would give the same decision. Ask the people who receive the output, not only the team that built the metric, and keep their responses in the record.
Measurement error model outputs
For generative AI systems, NIST AI 600-1 suggests creating measurement error models for pre-deployment metrics to demonstrate construct validity. Store the model assumptions, the error estimates, and the conclusion on whether each metric measures what its definition claims. Re-run the model when the test set, the model version or the prompts change, since a model built on last year's inputs no longer describes today's measurement. Confirm the approach with your AI actors and your assessor before you rely on it.
Metric change logs and review minutes
Version every metric definition and threshold, and log each change with the reason, the approver and the date. Review minutes should show who examined the metric, what evidence they considered, and what they decided to keep, retire or replace. Auditors test the sequence: a threshold that moved after an incident without a recorded reason is a finding. Store logs in a system that prevents silent edits, and back them up with the rest of the TEVV evidence.
Production signals that confirm detection
A metric proves its worth when it flags problems that matter after deployment. Compare pre-deployment metrics with production monitoring data, and record every case where a production incident, user complaint or human review found an issue the metric missed. Each miss is evidence that the metric or its threshold needs work. Tracing tools such as Langfuse can record the inputs, outputs and token usage behind each production event, which gives reviewers the material to judge whether a metric would have caught it.
Re-review schedule between audits
Set a calendar that re-reviews every metric on a fixed cycle and after defined triggers, such as a new model version, a material incident, a change in intended use or a new regulation. Assign an owner for each metric and record each completed review. NIST says the AI RMF itself will be revised, and the Playbook will be updated afterward, so schedule a check against the current NIST publications as well. Ask your assessor which cycle they expect to see in the evidence.
What meeting MEASURE 2.13 costs in money and effort
This table breaks down the cost components of meeting MEASURE 2.13, showing who is paid for each, whether it is one-time or recurring, and what drives it up or down. Use it to build a quote request that separates build work from ongoing review, and to see where open-source tooling reduces spend without removing the need for judgment.
| Cost component | Who is paid or who does the work | One-time or recurring | What drives it up or down |
|---|---|---|---|
| Engineering time to build review processes | Internal platform or ML engineers, or an implementation partner preparing the review workflow and records | One-time to set up, then recurring as reviews run | Number of systems in scope, number of metrics, and whether TEVV records already exist in a usable form |
| Metric utility and acceptability review | Internal TEVV staff and product owners reviewing whether metrics stay useful and understandable | Recurring at each review cycle | How many metrics are retained, how complex they are, and how often the system or its use changes |
| Stakeholder and affected-community engagement | Internal risk, legal and communications staff, plus any external facilitators the organization hires | Recurring, with an initial round at setup | Number of affected groups, the need for accessible channels, and whether feedback already flows through GOVERN 5 and MAP 5 processes |
| Tooling for evidence | Open-source options such as garak (Apache-2.0) and Langfuse (MIT) carry no license fee; cost falls on hosting, setup and staff time | One-time integration, then recurring hosting and upkeep | Whether the platform runs in the client's own infrastructure, log retention needs, and the volume of traces or probe runs |
| Independent external review | External assessors or reviewers engaged by the organization. Only an independent party issues findings or reports | Usually recurring at a set interval, with extra work after major changes | Scope of the review, depth of sampling, and whether the organization's own evidence is complete when reviewers arrive |
| Recurring re-review after changes | Internal teams, with optional support from an implementation partner for re-running tests and updating records | Recurring, triggered by model, data, use or metric changes | Frequency of model updates, changes in the regulatory context, and how well earlier decisions were documented |
| Revision-related rework | Internal policy and compliance staff, and any consultant retained to track changes | Occasional, tied to external changes | NIST says AI RMF 1.0 is being revised and has not published a revised version as of October 2026, so plan for updates to subcategory language |
Which open-source tools produce MEASURE 2.13 evidence
garak, an Apache-2.0 licensed scanner, probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. For MEASURE 2.13, its output is useful as raw material for asking whether a chosen metric actually detects the failure it claims to detect. The run writes a JSONL report covering every attempt and evaluation, plus a hit log of the attempts that found a vulnerability. Comparing hit log entries against the metric scores your team reported for the same behavior shows whether the metric agreed with the probe result.
- Attempt and evaluation records: The JSONL report lets you count how many attempts each probe made and how each was scored, which supports a documented review of metric coverage.
- Hit log for misalignment checks: Attempts that the probe marked as successful can be compared with whether the production or test metric flagged the same output. Disagreements are the main signal that a metric is not identifying risk.
- Hallucination and misinformation probes: These map to the confabulation and information integrity concerns named in NIST AI 600-1 under MS-2.13-001, so they can support construct validity discussion for the metrics involved.
- Probe scope limits: A probe run proves only that the listed probes were executed against the tested model and configuration. It does not prove that the probes cover every risk the system faces.
- Version and configuration capture: Record the garak version, probe selection, model identifier and any generator settings with each run, since the same test repeated on a different configuration is not comparable evidence.
Confirm with your security and TEVV teams which probes match the risks identified in MAP, and whether the hit log format fits the review workflow used for your metrics.
Langfuse, an MIT licensed observability platform, records traces of each request to a language model application. A trace captures every operation in the request, including LLM calls, retrieval steps, tool executions and custom logic, along with timing, inputs, outputs, token usage and metadata. For MEASURE 2.13, the trace data supports a review of whether production metrics still reflect what users experience and whether the metrics remain useful after deployment. It is most valuable where the team already tracks quality scores and needs to check whether those scores track the traced behavior.
- Timing and token usage: Latency and token counts per trace help test whether a cost or responsiveness metric reflects the actual workload, which matters when a metric is kept for operational reasons.
- Input and output records: Stored inputs and outputs allow reviewers to sample traces, read them and judge whether a metric flagged the problems a human reviewer would flag.
- Retrieval and tool steps: Per-step records show which component produced a bad output, so a metric can be checked against the step it is meant to measure.
- Privacy and retention decisions: Traces can contain personal or sensitive data, so retention, masking and access controls must be set before the traces are used for metric review.
- Coverage limits: Traces show what was logged. Operations that were not instrumented do not appear, so gaps in instrumentation become gaps in metric evidence.
Before relying on trace data, confirm the deployment model (self-hosted or cloud), the data retention and access settings, and whether instrumentation covers every operation that a metric depends on.
Neither tool decides whether a metric is effective. Each produces records that a person must interpret against a defined question: does this metric still sustain improvement, is it useful to the people who rely on it, and does it identify the risk it targets? Assessors will expect the organization to show its reasoning, not just the tool output. Treating a tool report as a conclusion is the most common gap in this evidence.
- No pass or fail verdict: A probe report or trace does not state that a metric is acceptable to end users or affected communities. That judgment requires documented stakeholder input.
- No substitute for construct validity work: Showing that a probe triggered a behavior does not show that the metric measured the same construct. Pre-deployment measurement error models, as described for MS-2.13-001, need separate analysis.
- Sample and cadence gaps: A single probe run or a sample of traces does not show ongoing effectiveness. Record how often reviews happen and what triggers an extra review.
- Sign-off and decisions: Metric changes, retirements and risk acceptance decisions must be recorded by the accountable owner, with the evidence cited.
- Independent review: Where the organization needs an outside view, an independent assessor reviews the evidence and issues any report. An implementation partner can prepare the logs and review records but does not validate them.
Ask your assessor early which evidence they expect for construct validity and stakeholder acceptance, since tool output alone is unlikely to answer those questions.
Where organizations fall short on MEASURE 2.13
MEASURE 2.13 asks whether the metrics and TEVV processes used across the Measure function still work. The gaps below are the ones that most often leave an organization with dashboards that look complete but do not show whether measurement is effective. Each card names a gap and the check that closes it.
Metrics chosen once and never reviewed
Teams often select metrics at launch and then run the same measures for the life of the system. MEASURE 2.13 expects the organization to check whether selected metrics and TEVV processes can still sustain improvements and the removal of errors. Schedule a review tied to model changes, new data sources and incident results, and record what was kept, changed or retired and why.
No record of utility or acceptability checks
A metric can be accurate and still be too complex for the people who must act on it, or unacceptable to end users and affected communities. The Playbook suggests evaluating metrics for utility and preferring descriptive approaches over overly complex ones. Document who reviewed each metric, what they concluded, and which metrics were simplified after feedback.
No link between metrics and production drift
Pre-deployment scores often stop being informative once inputs shift in production. If MEASURE 2.13 results are not compared with production monitoring signals, the organization cannot tell whether its metrics still detect the risks that matter. Confirm that drift and anomaly alerts from production trigger a review of the metrics that were supposed to catch them.
Generative AI metrics without construct validity evidence
For generative AI, NIST AI 600-1 suggests creating measurement error models for pre-deployment metrics to demonstrate the construct validity of each metric (MS-2.13-001). Many teams report scores for hallucination or harmful output without showing that the score measures the risk it is named for. Ask for a written error model and test results that show each metric tracks the risk it claims to measure.
No review by people outside the AI team
Teams that build and score a system rarely test whether their metrics would convince the TEVV, deployment and operations actors NIST tags to MEASURE 2.13. Without an outside review, blind spots persist. Ask a separate reviewer, such as an internal audit or risk function, to challenge the metric set and record the questions raised and how each was resolved.
How long and how much effort does MEASURE 2.13 take?
What drives the cost of working on MEASURE 2.13?
The cost depends mostly on how many AI systems are in scope, how many TEVV metrics and processes already exist, and how much of the review must be rebuilt rather than checked. MEASURE 2.13 asks you to evaluate and document whether your metrics and TEVV processes are effective, so the work scales with the number of metrics, pipelines and deployed models you need to examine. Other drivers include the number of AI actor groups involved, whether generative AI systems need the Generative AI Profile actions, and how much historical test data is available for review. Ask any provider for a quote that lists each system, each metric family and each process reviewed, plus the evidence they expect you to supply.
Can we evaluate MEASURE 2.13 in-house?
Yes, the AI RMF is voluntary and NIST does not require an outside party to assess your use of it, so an internal team can carry out the review. The NIST Playbook suggests actions and documentation prompts but states it is neither a checklist nor a set of steps, so your team must decide how to apply it. Teams that lack experience with metric validation, or that need an independent view of whether metrics identify real risk, often bring in an implementation partner for part of the work. Your organization still owns the controls and the conclusions.
What does the work need from engineering and governance staff?
Engineering staff typically need to provide metric definitions, test pipelines, evaluation logs and records of how each metric was calculated and changed. Governance staff typically need to confirm which risks the metrics are meant to detect, who accepts the metrics as useful, and how findings feed into go or no-go decisions. The Playbook tags MEASURE 2.13 to the TEVV and AI Deployment and Operation and Monitoring actor categories, so both groups should expect to contribute. Plan time for review meetings where end users or affected communities can confirm that the metrics are acceptable to them.
How does MEASURE 2.13 relate to the Generative AI Profile action MS-2.13-001?
NIST AI 600-1 adds one suggested action for MEASURE 2.13, MS-2.13-001, tagged to the generative AI risks of confabulation, harmful bias and homogenization, and information integrity. It asks for measurement error models for pre-deployment metrics that demonstrate the construct validity of each metric. If you run generative AI, this action adds specific work on top of the general MEASURE 2.13 review, so expect separate effort for construct validity analysis. Confirm the scope of that analysis with your assessor or internal audit team before you start.
Is there an audit or certificate for MEASURE 2.13 or the AI RMF?
No. The AI RMF has no certification, accreditation or conformity assessment program, and NIST states that it does not validate or endorse any organization's approach to using the framework. Any claim of AI RMF certification should be treated with caution. Some customers or regulators may ask for evidence of alignment, and in that case the evidence is your own documentation, reviewed by whoever your contract or regulator names. Only an independent auditor or certification body issues reports or certificates for other standards, so check which party a given requirement actually accepts.
What should we confirm with the auditor or customer before starting?
Confirm which assessor or reviewer will examine the work and what form the evidence must take. Ask whether they expect a specific scope of AI systems, whether they want the Generative AI Profile actions addressed, and how they will judge that a metric is effective. Ask which version of the framework they reference, since NIST says AI RMF 1.0 is being revised and has not published a revised version as of October 2026. Also confirm how often they expect the metric review to repeat, and whether they accept internal review records.
What does a fixed-scope engagement cover?
A fixed-scope engagement with DeployOpen covers the systems, metrics, TEVV processes and deliverables agreed in writing before work starts, along with the timeline for each. For MEASURE 2.13, that can include preparing evaluation records, reviewing metric utility and documenting how metric effectiveness was checked. DeployOpen prepares systems, controls and evidence for assessment, but it does not issue audit reports, attestations or certifications, and it does not decide whether your metrics meet your obligations. Ongoing monitoring and re-testing are available as retainers, and pricing follows from the written scope.
How DeployOpen prepares MEASURE 2.13 evidence
DeployOpen's part in MEASURE 2.13 is preparation. Before any work starts, the scope is agreed in writing, and that document should name the MEASURE 2.13 evidence to be produced: the metric register, the review records for each TEVV metric and process, the utility and end user acceptance checks, and the decision log showing what changed as a result. DeployOpen then implements the self-hosted open-source platforms that generate this evidence inside the client's own cloud accounts, data centers or air-gapped networks, so the records stay under the client's control. Where a metric depends on a tool such as garak or Langfuse, the engagement covers configuring its outputs, retention and access so the records can be reproduced later.
Security testing supports the same evidence. DeployOpen can run application, API, infrastructure and AI system testing to check whether the metrics you rely on would actually detect the failures they are meant to catch. A metric that never fires against a seeded defect is a finding under MEASURE 2.13, and the test results become part of the documented review. Ongoing support and continuous testing are available as retainers, which keeps the review cycle running after the initial evidence set is delivered. The client still owns the decisions about which metrics to retire, change or keep.
DeployOpen does not issue audit reports, attestations or certifications, and nothing in this work should be read as a conformity finding. The AI RMF has no certification program, so the question for your organization is what an independent assessor, internal audit or a customer's reviewer will accept as evidence. Confirm that expectation with that party before scoping, and ask them which review cadence, sign-off roles and documentation format they expect. Independent assessors and certification bodies are the parties that issue reports or certificates where a contract or regulator requires one.
