What MEASURE 4.1 requires for NIST AI RMF measurement
MEASURE 4.1 asks that the measurement approaches an organization uses to identify AI risks are tied to the context where the system is deployed, that domain experts and end users inform those approaches, and that the approaches are documented. A team that can show metric values but cannot explain why those metrics reflect how people actually use the system does not yet meet the intent of this subcategory. The risk it targets is familiar: a model scores well on a curated test set, the metrics look healthy in a review meeting, and the failures that matter (a misread case, a workaround operators rely on, a question the model answers confidently but wrongly) never appear in the numbers because nobody measured them.
The AI RMF is voluntary, and NIST does not validate or endorse any organization's use of it. No certificate exists, so no outside body will confirm MEASURE 4.1 is met. The evidence is internal, and an independent assessor, if you bring one in, will test whether your documentation and records hold up. Your readers should confirm which internal owner signs off on the measurement approach and which contract or regulator, if any, asks for it, because those demands set how formal the record must be.
This is hard for two reasons. Context is specific to each system: the same classifier behaves differently in a claims queue, a triage desk and a customer chat. The people who know that context, such as frontline operators, practitioners in the business unit, and affected users, usually sit outside the AI team and may not feel safe disagreeing with a vendor's or a data scientist's view. Start by listing the deployment settings, the user groups in each, and the feedback channels that already exist, then record which measurements were chosen with those people and which were not. The Playbook suggestions for this subcategory, covered in the Playbook guidance, give a starting set of prompts.
Which AI systems and measurement methods MEASURE 4.1 covers
MEASURE 4.1 only works if the measurement boundary is drawn before tests are chosen. The cards below cover the decisions that set that boundary: which systems and deployments are included, which measurement methods count, who must be involved, and where third-party components sit. The organization owns each choice, and the subcategories it selects under the framework determine how far the scope reaches.
AI systems and deployments in scope
List every AI system and each deployment context in which it runs, including internal pilots, customer-facing features and workflows where staff act on model output. MEASURE 4.1 asks that measurement be tied to the deployment context, so a model tested in a sandbox is a different scope from the same model used in a claims or hiring workflow. Record the intended purpose and the users for each deployment, and confirm the list against the AI inventory kept under GOVERN 1.
Automated tests and benchmarks
Decide which automated evaluations count toward MEASURE 4.1, such as scripted benchmarks, regression suites and adversarial probes. Automated tools like garak produce a JSONL report of every attempt and a hit log of vulnerabilities, which can serve as records of the approach. Confirm that each test is linked to a risk identified in the context-specific mapping, because a generic benchmark score alone does not show that measurement reflects the deployment.
Human evaluations and elicitation
Define which human-centered methods fall inside the boundary, such as structured interviews, red-team sessions, expert reviews and feedback sessions with operators. The Playbook suggests designing elicitation with human-factors experts and creating settings where people can openly share doubts. Document who conducted each method, how participants were selected and how results were recorded, so that the approach can be reviewed later.
Production monitoring and telemetry
Set which runtime signals are measured after launch, such as output quality samples, error rates, user complaints and drift indicators. Tracing tools like Langfuse capture inputs, outputs, timing, token usage and tool executions for each request, which can support this measurement if the trace fields are chosen deliberately. Confirm retention periods, access controls and privacy handling for any stored prompts or outputs before the monitoring scope is approved.
Teams and actors who must participate
Name the people who contribute to measurement in each category the Playbook tags for MEASURE 4.1: TEVV staff, deployment and operations teams, end users, and affected individuals and communities. Each group sees different failures, so a scope that includes only the development team will miss context-specific harms. Record each role, the task it performs and how its input reaches the measurement design. Appendix A of AI 100-1 describes the tasks for each actor category.
Third-party models and services
Decide whether externally hosted models, APIs, retrieval sources and plug-ins sit inside the measured boundary. If a third-party model changes its behavior, your deployment can change even when your code does not. Include third-party components where your organization controls their inputs or consumes their outputs, and note the vendor-side measurements you must request under the supply chain governance in place. Confirm what the vendor will disclose about testing, updates and versioning.
How an assessor tests MEASURE 4.1 measurement approaches
This table sets out five review steps an assessor or internal auditor would take for MEASURE 4.1, the test approach, the evidence requested and the gaps that show up most often. No assessor issues a NIST AI RMF certificate, so each row tests the organization's own documented measurement approach against the deployment context it claims to serve.
| Test approach | Evidence requested | Common findings |
|---|---|---|
| Confirm measurement approaches are documented and linked to a specific deployment context | Measurement plan naming the system, its deployment setting, the risks each metric covers and the version in use | Metrics documented in general terms with no link to a named deployment or user population |
| Sample consultation records with domain experts and end users | Meeting notes, workshop records, survey instruments and a list of who was consulted and in what role | Consultation happened once at design time and was never repeated after deployment changes |
| Check that end users, operators and practitioners could raise doubts | Feedback channels, anonymous reporting options, samples of raised concerns and how each was answered | Feedback forms exist but reviewers never see them, so concerns are not captured or acted on |
| Check whether internal measurements were compared with end-user feedback | Side-by-side comparison reports, discrepancy logs and the decisions taken on each discrepancy | Internal metrics reported as healthy while user feedback points elsewhere, with no comparison recorded |
| Review how elicitation techniques were designed and evaluated | Design rationale, input from human-factors experts, pilot results and a satisfaction or quality check on the technique | Elicitation questions written by the AI team alone with no record of testing or expert review |
Why deployment context changes what good measurement looks like
The same model needs different measurement depending on where it runs. In a claims triage workflow, the risks that matter most are wrong routing decisions, missed fraud signals and unfair treatment of particular claimant groups, and the people who can judge output quality are experienced adjusters and claims auditors. In a clinical summary tool, the central concern is a plausible but incorrect statement that a clinician might copy into a record, so judgment belongs to clinicians and documentation reviewers. In an internal help desk, the main risks are wrong policy answers and data exposure through overly broad retrieval, and the judges are service desk staff and the employees who ask the questions. A single benchmark score cannot cover these differences, because it does not show which failure a given user will notice first.
For each deployment, the MEASURE 4.1 work starts by documenting who the end users and affected people are, which failure modes they can actually see, and which experts can tell a good answer from a bad one. The NIST AI RMF Playbook suggests capturing feedback from end users, operators and practitioners in settings where they can openly share doubts, and designing elicitation with human-factors experts. It also suggests comparing internal measurements with end-user feedback, so that a gap between what the team's test suite reports and what users experience becomes a recorded finding instead of a surprise. Teams should confirm which of these suggested actions fit their own setting and record why any were left out.
The Generative AI Profile, NIST AI 600-1, lists no suggested actions for MEASURE 4.1, and it says subcategories it does not address may be added later. Organizations running generative AI therefore have to derive their own context-specific actions, such as the feedback channels, elicitation questions, expert panels and comparison method for each deployment, and then document them. A practical first step is to write a short measurement plan per use case that names the owner, the users consulted, the expert reviewers and the review cadence. Ask any implementation partner to show how that plan will be produced and kept current, because the organization, not the partner, must own the approach.
Implementing MEASURE 4.1 step by step
This sequence connects the risk measurement approaches an organization already uses to the real deployment context, using structured input from domain experts and end users, and leaves a documented record that an assessor or internal reviewer can follow. The Playbook is a voluntary companion for this work, not a checklist.
Confirm the deployment context and mapped risks
Start from the context and risk register produced under MAP, including the intended purpose, the settings where the system operates, the people it affects and the risks already identified for each. Name the accountable owner for the AI system and the team that will run the measurement work. The step is complete when a short context statement exists, signed off by that owner, and every mapped risk it references can be traced back to a MAP output. Confirm the current status of these outputs with your own AI governance team before you rely on them.
Identify domain experts and end-user groups
List the domain experts who understand the business or sector use of the system and the end-user and operator groups who use its outputs day to day. Include affected individuals and communities where the system's decisions reach them. Record why each group is included and which risks it can speak to, so the selection is defensible later. The output is a stakeholder table with roles, representation and the specific risks each group is positioned to inform. Ask your legal or compliance team to confirm any consultation obligations that apply in your jurisdiction.
Design elicitation with human-factors input
Work with human-factors specialists to design how feedback will be gathered: interview guides, structured surveys, observation protocols or review sessions. The design should ask about doubts, workarounds and unexpected outcomes, not only satisfaction. Decide how feedback will be recorded and how participants will know their input is used. The output is a written elicitation plan that names the method for each stakeholder group and the questions each method asks. If your organization lacks human-factors expertise in-house, confirm with your advisor who will perform this design.
Run consultations where people can share doubts
Hold the sessions in settings where participants can openly question the system without fear of blame, for example with facilitators who are independent of the delivery team and with assurances about how comments will be attributed. Record the sessions with consent, capture both the feedback and the reasons behind it, and note any concerns that participants raise about the system's limits. The output is a dated consultation record for each group, including attendance, the questions asked and the main issues raised. Confirm with your HR and works council or equivalent bodies what consent and recording rules apply.
Link each measurement approach to a context statement
For every metric, test or monitoring approach in use, write a short statement that explains which deployment context and which mapped risk it addresses, and which stakeholder feedback informed it. Flag approaches that do not connect to any context or risk, since they may measure the wrong thing. The output is a linkage table that maps each measurement approach to its context statement, risk and input source. Where a generative AI system is involved, the NIST AI 600-1 profile lists no suggested actions for this subcategory, so document your own rationale for any approach you choose.
Compare internal metrics with feedback
Set the internal measurement results beside the feedback gathered in the consultations and note where they agree, where they conflict and where feedback reveals a risk the metrics do not capture. Investigate each gap, whether the metric is measuring the wrong thing, the feedback reflects a misunderstanding or a new risk has appeared. Record each finding with its evidence. The output is a comparison log with a disposition for each gap, which feeds MEASURE 4.2 and MEASURE 4.3. A useful test is whether a reviewer can trace each conclusion back to the raw consultation notes and the metric data.
Record approaches and changes in a versioned document
Assemble the context statements, stakeholder table, elicitation plan, consultation records, linkage table and comparison log into one controlled document. Give it a version number, an owner and a change history, and record every change to a measurement approach with the reason for it. Store it where the people who need it can find it and where it can be retained as evidence. The step is complete when the document is approved by the accountable owner and a reviewer outside the delivery team has confirmed it can be followed. Ask your assessor or internal audit team which format and retention period they expect.
Evidence that shows measurement is connected to deployment context
MEASURE 4.1 asks whether measurement approaches for AI risks are tied to the real deployment context and informed by domain experts and end users, with those approaches documented. The six artifacts below link the context, the measurement design, the consultation record and the changes over time, so an assessor can trace each measurement back to the setting it was meant to test.
Documented context statement per system
A short statement for each AI system that records its intended purpose, the people who use its output, the populations it affects and the settings where it runs. This is the anchor for every other artifact, because a measurement approach can only be shown to fit a context that is written down. Check that the statement names the deployment populations and the operators, not only the development team's view. Review it when the system's use or user group changes, and record the date and approver of each revision.
Measurement approach linked to context
A document that lists each measurement method used for the system's identified risks and states why it fits the context statement. An assessor will look for a clear line from each risk to the metric, the test population and the conditions under which the metric was run. Confirm that gaps are recorded, such as risks with no suitable metric yet. Set review triggers for model replacement, prompt or retrieval changes, and new deployment populations, since any of these can make an earlier approach stop matching the setting.
Consultation plans and session records
Plans for consulting domain experts and end users, with attendance lists, agendas, notes and the feedback each session produced. The Playbook suggests capturing feedback in settings where people can openly share doubts, so record how those sessions were run as well as who attended. Keep a log of which feedback changed a measurement decision and which did not, with a reason. Refresh the plan at each deployment change or when a new user group is onboarded.
Elicitation design and human-factors review
The questionnaires, interview guides, task scripts and rating scales used to gather end user input, plus a record of review by human-factors specialists. The Playbook suggests designing elicitation with human-factors experts, so keep the review comments and the revisions they caused. Ask whether the design tests what users actually do with the output, not only what they say they do. Re-review the design when the interface, the user task or the population changes.
Internal metrics compared with user feedback
A comparison between the system's internal measurements and what end users, operators and practitioners report. Gaps between the two are often the most useful evidence of a context mismatch, for example a benchmark score that looks sound while users report errors on a specific task. Record each comparison with its date, data sources and the follow-up decision. Repeat the comparison after changes to the model, the data or the user base, and when production monitoring shows a shift in reported problems.
Change log for measurement approaches
A dated record of each change to a measurement approach, the trigger for the change, the person who approved it and the evidence that supported it. Assessors use this log to confirm that approaches were updated when the context moved, rather than left unchanged since the first deployment. Tie each entry to a source such as a model version, a ticket, a consultation session or a monitoring alert. Review the log at each audit cycle and at every release that alters the system's behavior or users.
What meeting MEASURE 4.1 costs in effort and tooling
This table compares the main cost components of meeting MEASURE 4.1, who is paid for each, whether it is one-time or recurring, and what moves it up or down. Because the AI RMF is voluntary and has no certification fee, every cost comes from internal effort, outside help and tooling, so the rows below are where a quote will be built. Use it to decide what to ask for in a written, fixed scope.
| Cost component | One-time or recurring | What drives it | How to keep it down |
|---|---|---|---|
| Internal engineering time to build measurement approaches | One-time build, then recurring upkeep | Number of deployed AI systems, number of context-specific measures, and how many metrics must link to documented risks | Start with the highest-impact system, reuse one documented approach template across systems, and record the approach in a single register |
| Domain expert and end user time for consultation | Recurring, tied to review cycles | How many expert groups and user populations are involved, how many sessions are held, and how hard it is to recruit participants | Schedule fewer, well-prepared sessions, combine groups where the context is similar, and record outcomes so later rounds build on them |
| Human-factors design support | One-time for the elicitation design, occasional for redesign | Complexity of the user population, the need to design feedback methods that let people openly share doubts, and whether satisfaction with the methods is measured | Ask whether a human-factors specialist is needed for the design only, and have internal staff run routine sessions using the agreed method |
| Evidence upkeep as deployments change | Recurring | Model updates, new data sources, changes in use context, and how often the documented approach is reviewed | Tie documentation updates to existing change management, and set a review trigger for each material change rather than reviewing on an open schedule |
| External review or independent testing | Periodic, often scoped per system | Whether an independent party checks the approach, the depth of testing, and whether results must be reviewed by experts outside the team | Define the review questions in advance, limit external work to validation of the documented approach, and reuse results across systems with similar context |
| Tooling or licenses | Recurring for commercial licenses, one-time for setup with open-source tools | Choice of commercial or open-source tools for testing and tracing, hosting requirements, and volume of logged data | Self-hosted open-source tools such as garak (Apache-2.0) and Langfuse (MIT) remove license spend, but they add setup, hosting and maintenance effort that should be costed internally |
How garak and Langfuse produce evidence for generative AI measurement
MEASURE 4.1 asks whether measurement is tied to the deployment context. garak produces evidence from controlled probing of a language model, while Langfuse records what the deployed system does in production. Used together, they can show what was tested and what happened in use, but the link back to the context statement must be built by the organization.
garak adversarial probing
garak is an Apache-2.0 tool that probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. Its output includes a JSONL report of every attempt and evaluation and a hit log of attempts that found a vulnerability. For MEASURE 4.1, map the probe families you run to the risks named in your context statement, and record which probes were chosen and why. Confirm that the probes match your model, language and use case before relying on the results.
garak reports as context evidence
Each garak run can be stored with the deployment context it tested, such as the model version, system prompt, tools and intended user group. Link hit log entries to the risks in your context statement so a reviewer can see which deployment assumptions were exercised. Illustrative example: a hit on a prompt injection probe for a customer-facing assistant with a tool connected to account data would point to a specific risk the context document needs to address. Confirm that your versioning captures every element that changes behavior.
Langfuse production traces
Langfuse is an MIT-licensed observability tool. Each trace captures every operation in a request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. Tag traces with the deployment environment, user group and feature so production behavior can be compared with the context the organization documented. Confirm how your team handles personal data in inputs and outputs before storing them.
Connecting traces to user reports
Traces become measurement evidence when they can be linked to end user problem reports, complaints and feedback. Store a trace identifier with each user report where your support process allows it, so reviewers can see the model path behind a reported failure. This supports the MEASURE 4.1 expectation that approaches are informed by end users. Confirm with your privacy and legal teams which identifiers and content you may retain, and for how long.
What neither tool provides
Neither garak nor Langfuse consults domain experts or end users. Their outputs show what was probed or what occurred, not whether the measurement approach fits the setting. The evidence of consultation, including meeting records, elicitation design, feedback summaries and decisions about which risks to measure, must come from the organization's own records. Keep these records alongside tool output so the documented approach can be traced from context to result.
Confirm fit before relying on either tool
Before relying on either tool for MEASURE 4.1, confirm how it runs in your environment: where logs are stored, who can access traces, how probes are configured for your model endpoints, and whether your deployment supports the integration points you need. Ask your implementation partner to document the setup and test it on a representative system. Neither tool is an assessment, and an independent assessor will decide whether the evidence is sufficient.
Who does what: your team, an implementation partner and the auditor
Your organization owns the measurement approach from end to end. Your team decides what the deployment context is, which domain experts and end users must be consulted, how their feedback is weighed against internal measurements, and what risk tolerance the measurements are judged against. These decisions shape every metric, so they cannot be delegated to a vendor or consultant. Your team also signs off on the final documented approach and keeps it current when the system, its users or its context change.
- Context decisions: Define the deployment setting, user groups, affected communities and the risks that matter most for this system.
- Consultation: Choose and run the sessions with domain experts and end users, and record who took part, what they raised and how it changed the approach.
- Approval and ownership: Name the accountable owner who approves the documented approach and its revisions.
- Internal records: Keep the minutes, feedback logs and comparison notes that prove consultation actually happened.
- Ongoing review: Trigger a review when the model, data, user population or deployment setting changes.
Before any partner starts work, confirm that you can name the owner of each decision above and that the approval trail is written down.
An implementation partner such as DeployOpen prepares the systems, processes and records that support your decisions. In a fixed-scope engagement, the partner can plan the consultation process, design the measurement records, configure self-hosted tools in your own cloud accounts or data centers and assemble the evidence pack. Tools such as garak, which is Apache-2.0 licensed and probes language models for issues including hallucination, data leakage and prompt injection, and Langfuse, which is MIT licensed and records traces of LLM calls, retrieval steps and tool executions, can produce the raw material that measurement records draw from. The partner does not decide your risk tolerance, does not replace your domain experts, and does not issue any report or attestation.
- Process design: Draft the consultation plan, session templates and feedback capture forms for your review.
- Record design: Structure the measurement documentation so each approach links to its context, owner and feedback.
- Tool configuration: Deploy and configure garak and Langfuse inside your environment, with access controls and retention settings your team approves.
- Evidence pack: Collect logs, reports and approval records into an organized package for the assessor.
- Handover: Train your staff to run the process and maintain the records after the engagement ends.
Confirm in writing which deliverables are included in the fixed scope and which tool capabilities you must verify yourselves before relying on them.
An independent auditor, assessor or certification body is the only party that tests the documented approach and issues an opinion on it. For MEASURE 4.1, the assessor typically checks whether the approach is tied to the deployment context, whether the consultation described in the documentation matches what occurred, and whether the approach itself is written down. Because the AI RMF is voluntary and has no certification, any formal opinion comes from the assessor you engage for your own purposes, not from NIST. Where your own records are the only proof that consultation took place, the assessor will rely on them, so their quality matters.
- Sampling: Select deployment contexts or consultation sessions and trace each back to records.
- Consistency checks: Compare the documented approach with system configurations, tool outputs and feedback logs.
- Gap identification: Flag consultations described in the documentation but not supported by any record.
- Reporting: Issue findings and any formal opinion under the scope you agreed with them.
- Independence: Keep the assessor separate from the team that designed the approach.
Ask any assessor to confirm in advance which samples they will select and which records they will accept as proof of consultation.
Questions buyers ask about MEASURE 4.1
Is MEASURE 4.1 mandatory, and is there a certificate for it?
No. The NIST AI RMF is voluntary, and NIST does not publish a certification, accreditation or conformity assessment program for it. NIST's FAQ says neither private nor public sector organizations are required to use the framework. MEASURE 4.1 describes an outcome: measurement approaches for AI risks should be tied to the deployment context, informed by domain experts and end users, and documented. Whether you must meet it depends on contracts, sector rules or state laws that name the framework, so confirm those requirements with your legal and compliance teams. Any assessment you commission will be a readiness review or an audit against a different framework, not an AI RMF certificate.
How much effort does MEASURE 4.1 take, and what drives it?
The effort depends mostly on how many AI systems are in scope, how many distinct user groups and deployment settings each one has, and how mature your existing monitoring is. The work breaks into three parts: documenting the deployment context, running structured feedback sessions with operators, end users and domain experts, and recording how measurement approaches were chosen and adjusted. Costs recur when systems change, because context shifts and the documentation must follow. Ask any provider for a quote that lists the systems in scope, the number of feedback channels, who runs the sessions, and what documentation they will produce.
How long does MEASURE 4.1 work take to complete?
The timeline is set by scope and by people's availability, not by the subcategory itself. Scheduling domain experts and end users, obtaining consent for feedback collection, and getting sign-off from business owners often take longer than the technical setup. Other drivers include the number of AI systems, whether a baseline measurement program already exists, and how many review cycles your governance process requires. The Playbook is a voluntary companion and its last content update was in August 2023, so plan for internal review of its suggested actions rather than expecting it to set a schedule. Ask for milestones in the written scope before work starts.
What must our team supply, such as access to experts and end users?
Your organization owns its controls, so it must supply the people and access the work depends on. That includes named domain experts who can judge whether results make sense in context, a sample of end users, operators and practitioners who can speak openly about doubts, and access to production logs or usage data. You also need a decision owner who approves the deployment context and the measurement approaches. Confirm with your HR, legal and privacy teams how feedback from end users is collected, stored and used, especially where personal data is involved. If any of these inputs are unavailable, the documented approach will show gaps that an assessor will ask about.
Can an in-house team do MEASURE 4.1 without an outside partner?
Yes, an in-house team can meet MEASURE 4.1 if it has the right skills and time. The subcategory asks for human-factors thinking in designing feedback collection, a way to compare internal measurements with end user feedback, and clear documentation. Teams often bring in outside help for specific tasks, such as designing elicitation methods, setting up test tooling, or preparing evidence packages for an independent review. DeployOpen plans, implements and documents self-hosted open-source platforms inside your own infrastructure and runs security testing, and it prepares systems and evidence. It does not issue audit reports or certifications. Ask any partner to define its role in writing, and confirm internally who will own the outcome.
How do garak and Langfuse fit into MEASURE 4.1?
Neither tool satisfies MEASURE 4.1 on its own, because the subcategory is about context and human input. garak is an Apache-2.0 scanner that probes language models for issues such as hallucination, data leakage, prompt injection and jailbreaks, and it writes a JSONL report of every attempt and a hit log of vulnerabilities. Langfuse is an MIT-licensed observability tool whose traces record LLM calls, retrieval steps, tool executions, timing, inputs, outputs, token usage and metadata. Used together, garak results can be compared with production traces and with end user feedback, which supports the 'compare internal measurements with end user feedback' action. Confirm each tool's current capabilities in its documentation before relying on them in an evidence file.
What will an assessor ask to see for MEASURE 4.1?
An assessor will look for evidence that your measurement approaches are documented, tied to the deployment context, and informed by the people who use or are affected by the system. Expect to show a written description of the deployment context, the list of domain experts and end users consulted, feedback session records, how feedback changed measurement choices, and the rationale for each approach. For a tool-based program, expect garak reports and Langfuse trace samples linked to the decisions they informed. Ask any assessor which evidence format they accept and whether they review documentation or test operating effectiveness, since that changes the preparation work.
What changes when NIST revises the AI RMF?
NIST says AI RMF 1.0 is being revised as part of the White House AI Action Plan and has not yet published a revised version. The July 2025 AI Action Plan recommends removing references to misinformation, diversity, equity and inclusion, and climate change. NIST also expects a formal review with community input no later than 2028, and its versioning uses a first number that changes only with major revisions and decimals for minor ones. MEASURE 4.1's core intent may stay, but subcategory wording, numbering and the Playbook could change. Keep your documentation tied to the version you used, record the date of each review, and confirm the current status with NIST's AI RMF pages before each review cycle.
How DeployOpen scopes and prepares MEASURE 4.1 evidence
DeployOpen prepares the measurement approaches, documentation and supporting systems that show how AI risk measurement is tied to a specific deployment context. Organizations keep ownership of the consultation with domain experts and end users, and of every decision about which approaches are adequate.
How we scope
- Engagements are fixed-scope, with the deployment context, AI systems and measurement approaches agreed in writing before work starts.
- The written scope names which stakeholder groups, such as operators and end users, will be included in feedback collection for MEASURE 4.1.
- Scope confirms which AI actor categories from the Playbook are assigned to each measurement approach and who inside the client owns each one.
How we build
- We set up measurement workflows that record how each approach was chosen and linked to the documented deployment context.
- Where a system uses generative AI, we can configure open-source test tooling such as garak to log probe attempts and evaluation results.
- Where logging is needed to connect operator feedback with system behavior, we can deploy open-source tracing such as Langfuse inside your environment.
How we hand over
- Deliverables include the documented measurement approaches, the context they were tied to and the records of consultation with experts and end users.
- Test outputs and traces are delivered in formats your team can retain, review and hand to an independent assessor.
- We walk your staff through the documentation so the owners can maintain it after handover without further DeployOpen involvement.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer for measurement tooling and documentation.
- Retainer work can revisit measurement approaches when the deployment context changes or when the revised NIST AI RMF is published.
- DeployOpen prepares evidence for assessment but does not issue audit reports, attestations or certifications, which come from independent assessors.
