MAP 5.1 impact likelihood and magnitude: evidence

What MAP 5.1 asks for and why it is hard to show

A customer questionnaire, a regulator's request or an internal risk committee asks one question: how does your organization rate the impacts of this AI system? MAP 5.1 in the NIST AI RMF expects every identified impact, beneficial or harmful, to carry a documented likelihood and magnitude, and the rating must rest on named inputs such as expected use, past uses in similar contexts, public incident reports, external feedback and other data. The framework is voluntary and has no certification, so no body issues a MAP 5.1 badge. An independent assessor tests the evidence: the impact register, the rating scale, the inputs behind each rating and the dates on which ratings were revisited.

The difficulty is rarely the rating itself. Impacts are often unanticipated, so the register starts with gaps that only appear once users, affected communities or incident data surface them. Teams also rate on different scales: one group uses a red-amber-green label, another uses a numeric score, and a third writes prose, which makes the portfolio impossible to compare. The Playbook suggests uniform impact assessment scales across the AI portfolio, qualitative or simulation-based, and an assessor will ask whether one scale is applied everywhere. Confirm which scale your organization has adopted and who signed off on it.

Ratings also go stale. A likelihood set at design time can be wrong after a model update, a change in user population or a new public incident involving a similar system. Ask for three things before any assessment: a single owner for each impact rating, a record of the inputs used and a trigger for re-rating after material change. Red-teaming that links system limitations to downstream harm, and testing and evaluation at key lifecycle stages, are the usual sources for the evidence. Confirm with your assessor, in writing, which artifacts they expect to sample.

Which AI systems, data and third parties fall under MAP 5.1

Before anyone rates the likelihood or magnitude of an impact, the organization has to decide what sits inside the rating exercise. The decisions below determine how many systems are rated, which outside parties must supply evidence, and whether generative AI needs extra actions. The organization owns these choices and should record them in writing before rating starts.

AI systems and versions in scope

Decide which AI systems, model versions and configurations need an impact rating. A rating done for version 2 of a credit model does not cover version 3 after its training data or decision thresholds change. Include systems that only inform human decisions, since MAP 5.1 covers impacts from outputs people act on. Ask each business owner to list production, pilot and retired systems that still affect people. Record the version identifier and the date each rating was made so later changes can be traced.

Vendor models and APIs

Decide whether third-party models, hosted APIs and vendor-embedded AI features are in scope. An organization that calls an external model still deploys it, so its users can be harmed by its outputs even when the organization did not train it. Confirm what the vendor documents about known limitations, incident history and testing, and what the contract lets you request. Where the vendor will not share that information, record the gap as a residual impact that needs a compensating control.

Relevant affected groups and AI actors

Decide which affected individuals, communities and end users count as relevant AI actors for each system. The Playbook tags MAP 5.1 with affected individuals and communities and end users, so the list should name groups such as customers, applicants, patients, employees or residents who can be affected by the output. For each group, state how their feedback reaches the rating team, and how groups that are hard to reach, such as minors or people with limited digital access, will be represented.

Generative AI and the MP-5.1 actions

Decide whether generative AI systems need the additional MP-5.1 suggested actions in NIST AI 600-1. These cover content provenance harms such as deepfakes or tampered content, and adversarial role-playing, red-teaming or chaos testing to find unforeseen failure modes. Any system that generates text, images, audio or code for users should be checked against these actions. The Generative AI Profile is voluntary, so the organization decides which actions apply and records the reason for each exclusion.

Data sources and external evidence

Decide which data sources feed the likelihood and magnitude estimates. MAP 5.1 names expected use, past uses in similar contexts, public incident reports, external feedback and other data. Set the sources in writing, such as internal incident logs, complaints, red-team results and public reports about similar systems, and note which ones are excluded and why. Data that comes from a third party, including vendor incident notices, should be recorded with its date and origin so the rating can be audited later.

How an assessor tests MAP 5.1 and what they ask to see

This table sets out how an assessor typically examines MAP 5.1 in an organization's own documentation, the evidence they request and the findings that come up most often. Read it as a pre-assessment checklist: each row is a question the assessor will ask, and the evidence column shows what should exist before they arrive. NIST does not certify or endorse any organization's use of the AI RMF, so any assessment is performed by a party the organization engages for that purpose.

Test area	What the assessor checks	Evidence requested	Common findings
Rating coverage	Every identified impact, beneficial or harmful, has a likelihood rating and a magnitude rating.	Impact register listing each impact with both ratings, the system it relates to and the review date.	Harmful impacts rated but beneficial or unanticipated impacts left out of the register.
Traceable inputs	Each rating links back to its basis: expected use, past uses in similar contexts, public incident reports, external feedback or other data.	Source notes or references attached to each rating, with the data set, incident or feedback item named.	Ratings stated without a basis, or inputs that cannot be found again after the fact.
Scale consistency	One impact assessment scale is applied across the AI portfolio, so ratings from different teams compare directly.	Written scale definitions (for example, qualitative red-amber-green or a defined numeric method), plus a sample of ratings from two or more systems.	Each team uses its own scale, or the scale is documented but scored differently in practice.
Reasoning visibility	The assessor can follow how a rating was reached, including assumptions and the people who agreed to it.	Rating rationale, assumptions log, reviewer names and sign-off records.	Final numbers recorded with no explanation, or reasoning held only in individual memory.
Revisit after change	Ratings are updated when the model, data, use case, user population or deployment context changes.	Change records linked to impact ratings, with dated re-rating entries and triggers defined in policy.	Ratings frozen at launch while the system has since been retrained, extended to new users or moved to a new setting.
Testing inputs	Test, evaluation, verification and validation results and red-teaming findings feed into the likelihood and magnitude judgments.	TEVV reports and red-team or adversarial test results mapped to specific impact entries.	Testing performed but results not connected to the impacts they were meant to inform.
Affected parties	Ratings reflect the perspective of affected individuals and communities and end-users, not only the build team.	Records of external feedback, user research or stakeholder input used in rating decisions.	Impact ratings written entirely by developers with no documented external input.
Generative AI actions	For generative systems, provenance harms such as deepfakes or tampered content are rated, and adversarial testing results are considered.	Ranked list of content provenance harms with likelihood and impact, plus adversarial role-play or red-team records.	Provenance and misuse risks absent from the impact register for systems that generate content.

How to implement MAP 5.1 in six steps

This sequence moves from naming each impact to ratings that have been tested and are reviewed as the system changes. Each step produces a record an assessor can read, so the likelihood and magnitude judgments can be traced back to their inputs.

Build the impact register
Start by listing every impact the AI system could have, both beneficial and harmful, for people inside and outside the organization. Draw on the context and intended purpose recorded under MAP 1 and the affected groups identified earlier in the Map function. Assign each impact an owner from the AI design or deployment team and a short description of who is affected and how. The step is done when the register covers each intended use and each affected group, and someone other than the author has reviewed it for gaps.
Set a uniform likelihood and magnitude scale
Define one scale for likelihood and one for magnitude, and apply it to every AI system in the portfolio so ratings can be compared. The Playbook lists options such as qualitative red-amber-green bands or simulation and econometric methods. Choose the option that your team can apply consistently and that your risk owners understand, and document the definition of each level. The output is a short scale document with approved sign-off from the accountable risk owner.
Compile the evidence inputs log
For each impact, record the evidence that supports the likelihood and magnitude judgment. Typical inputs named in MAP 5.1 include expected use, past uses in similar contexts, public incident reports, external feedback and other data. Log the source, date, and relevance of each input, and note where evidence is missing. Missing evidence should be flagged as an open item instead of filled with assumptions. The step is complete when every impact in the register links to at least one logged input or to a documented gap.
Rate each impact
Using the agreed scale and the evidence log, assign a likelihood rating and a magnitude rating to each impact in the register. Record the rationale in one or two sentences so a reviewer can follow the reasoning. Where ratings depend on assumptions about user behavior or operating conditions, state those assumptions next to the rating. The output is a ratings table that lists impact, likelihood, magnitude, rationale, rater, and date.
Test the ratings through TEVV and red teaming
Check whether the ratings hold up under testing. The Playbook suggests applying TEVV at key lifecycle stages and running red-teaming to connect system limitations to downstream impacts. For generative AI, NIST AI 600-1 suggests adversarial role-playing, red teaming, or chaos testing to surface failure modes that were not foreseen. Record which tests ran, what they found, and which ratings changed as a result. The record should be kept by a team that is independent of the system's developers where practical. Tools such as garak can produce test logs for language model probes, and Langfuse traces can show operational behavior that supports or challenges a rating.
Schedule reviews tied to lifecycle stages
Set a review schedule that is linked to events in the system's life, such as a new model version, a change in intended use, a new user group, a new data source, or a significant public incident. Confirm the review trigger list and the named owner for each review with your governance function, since MAP 5.1 does not set a fixed interval. Each review should update the register, the evidence log, and the ratings, and it should record what changed. The step is done when the schedule is approved, the triggers are written into change management, and the first review date is on the calendar.

Which evidence satisfies MAP 5.1 and how to keep it current

MAP 5.1 is met by records that show each impact was rated for likelihood and magnitude and that the ratings were tested and revisited. The artifacts below connect to one another, so a change in one should trigger a check of the others. Assessors will expect the set to be current, not just assembled once.

Impact register with ratings

The register lists each identified impact, whether beneficial or harmful, with a likelihood rating, a magnitude rating and the named system and context it applies to. Each row should show who rated it and when. For MAP 5.1, check that the register covers unanticipated and downstream impacts, not only the harms the team expected at design. Review the register at each release gate so that new features add rows rather than leaving the old ones untouched.

Scale definition and version history

The impact assessment scale defines what low, medium and high likelihood or magnitude mean for the organization. The Playbook suggests a uniform scale across the AI portfolio, which may be qualitative, such as red-amber-green, or based on simulation. Keep the scale under version control and record each change with its reason. Ratings made on different scale versions cannot be compared without a note explaining the difference.

Source inputs for each rating

Each rating should cite the inputs behind it: the expected use, past uses in similar contexts, public incident reports, external feedback and other data. Store these as linked references in the register, not as recollections in meeting notes. Ask for the source list when reviewing a rating. A rating that cites no input is hard to defend, and assessors will ask how it was reached.

Red-teaming and TEVV results

Red-teaming, adversarial testing and test, evaluation, verification and validation results show whether the ratings hold in practice. For generative AI, the Generative AI Profile calls for adversarial role-playing, red-teaming or chaos testing to find failure modes that were not foreseen. Keep test scope, the rating each test challenged, and any rating that changed as a result. Tools such as garak can generate JSONL reports and a hit log of successful attempts, which can be attached to the record.

Re-rating records after changes and incidents

Re-rating records show that the register was updated when conditions changed. Tie re-rating to release gates, so a model update, a new data source or a new user population triggers a review before deployment. Also tie it to incident reviews, so that a reported failure leads to a documented re-rating of the affected impact. Keep traces of production behavior, such as those from Langfuse, to show what the system did before and after the change.

Sign-off by the accountable owner

A named accountable owner signs off on each rating and each change to the scale, and that sign-off is recorded with a date. The owner should be identified in the governance records for the system. Ask whether the owner has authority to accept or escalate high-magnitude impacts. Re-confirm sign-off at each scheduled review, since a stale approval does not show that the current ratings were reviewed.

What meeting MAP 5.1 costs in money and effort

This table lists the cost components of building and keeping up a MAP 5.1 impact likelihood and magnitude register, who is paid for each, whether it is one-time or recurring, and what moves it up or down. No figures are given because none are supplied for this page, so use the drivers to scope a written quote.

Cost component	Who is paid	One-time or recurring	What drives it up or down
Tooling and licenses for the impact register and scoring	Software vendors, or the organization for internal tools	Mostly recurring (subscriptions, hosting, support)	Number of AI systems tracked, whether scales live in a spreadsheet or a governance platform, and whether open-source tools reduce license lines
Engineering time to build the impact register and uniform scales	Internal staff, or an implementation partner under a fixed scope	One-time build, then periodic updates	Number of AI systems in scope, how many impact categories the organization defines, and how many existing assessments can be reused
Evidence upkeep (incident reviews, external feedback logs, updated likelihood and magnitude ratings)	Risk, product and AI teams, sometimes a compliance lead	Recurring	How often systems change, how many generative AI use cases are live, and how quickly public incident reports and user feedback are reviewed
TEVV and red-teaming to test impact assumptions	Internal testers or an external security or AI testing firm	Recurring per test cycle, with a larger initial exercise	Number of generative AI use cases, depth of adversarial testing, and whether testing is tied to major model or data changes
Independent review of impact ratings	External reviewer or assessor, engaged by the organization	Recurring, usually tied to a review cycle	Scope of systems reviewed, the risk tier of each system, and whether the reviewer must reperform ratings or only read the evidence
Auditor or assessor sampling time	The independent auditor or assessor, not DeployOpen	Recurring per assessment cycle	Number of systems sampled, quality of the evidence trail, and how many ratings must be explained from documents
Ongoing monitoring and re-rating after changes	Operations, MLOps and risk staff	Recurring	Frequency of model updates, retraining, new data sources and new deployment contexts

Which self-hosted open-source tools can produce MAP 5.1 evidence

Three self-hosted open-source tools can supply parts of the evidence behind MAP 5.1, each covering a different source of impact data: framework and assessment records, model behavior testing, and production traces. None of them rates impacts on its own, so the organization still decides likelihood and magnitude and must review each tool's license before deployment.

CISO Assistant for assessment records

CISO Assistant is an AGPL-3.0-only platform that runs audits and compliance assessments and tracks evidence against a library of more than 200 frameworks. For MAP 5.1, it can hold the register of identified impacts, the scale used to rate them and the assessment history that shows how ratings changed over time. Confirm whether its framework library includes the NIST AI RMF before you rely on it for this purpose, and check the AGPL-3.0-only terms with counsel if you plan to modify or distribute the software.

garak for red-team test records

garak is an Apache-2.0 scanner that probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. Its JSONL report logs every attempt and evaluation, and its hit log lists the attempts that found a vulnerability. These records can support the red-teaming and adversarial testing that NIST AI 600-1 suggests under MAP 5.1 for generative AI, and they show which harms were actually reproduced, which strengthens the likelihood estimates. Test in a non-production environment and confirm that the probes match the harms your impact register lists.

Langfuse for production traces

Langfuse is an MIT-licensed observability platform. Each trace records inputs, outputs, timing, token usage, tool executions and metadata for every operation in a request. For MAP 5.1, these traces let you compare observed behavior in deployment with the likelihood and magnitude ratings you set before launch. Define which fields may contain personal data, decide retention and access controls, and confirm that trace capture does not expose the content that the impact assessment is meant to protect.

Ratings stay with the organization

Each tool produces records, not judgments. Someone accountable must choose the impact scale, such as red-amber-green or a quantitative method, assign likelihood and magnitude to each impact, and sign off on the result. Auditors and internal reviewers will ask who made each rating and what data supported it. Document the rating method, the evidence source for each entry and the reviewer, so that tool output is clearly input to a human decision and not a self-generated score.

License review before deployment

The three tools use different licenses: AGPL-3.0-only for CISO Assistant, Apache-2.0 for garak and MIT for Langfuse. Each carries different obligations when you modify, host for others or redistribute the software. Have legal counsel review the terms against your intended use, including any internal customization, and record the decision in your AI governance documentation. Licensing review is a prerequisite for the evidence trail, because an evidence system that cannot be lawfully deployed produces no usable records.

Who owns the MAP 5.1 ratings and who prepares the evidence

Your organization owns the outcome that MAP 5.1 describes: the likelihood and magnitude of each identified impact, beneficial or harmful, must be identified and documented by people accountable for the system. That means your AI risk owner, system owner and the business leaders who accept residual risk must decide what each rating means for your context, approve the impact scale you adopt, and make the call to deploy, restrict or change a system. These decisions cannot be delegated to a vendor or a consultant, because the accountability sits with the organization that operates the system.

Rating decisions: Assign likelihood and magnitude for each impact in your register, citing the expected use, past uses in similar contexts, incident reports and external feedback behind each rating.
Accountable sign-off: Name the individual who approves the ratings and records the date, version and any dissent.
Deploy or change decisions: Record whether the impact profile supports go live, conditional release or rework, and who made that call.
Scale adoption: Decide whether your impact scale is qualitative, such as red-amber-green, or more quantitative, and confirm it applies across the AI portfolio.
Governance linkage: Connect ratings to the risk tolerance and policies set under GOVERN 1 and GOVERN 2.

Confirm that your internal owners can explain each rating without referring to the partner who helped build the template.

Where MAP 5.1 programs usually fall short and how to close each gap

MAP 5.1 asks for the likelihood and magnitude of each identified impact to be identified and documented, using expected use, past uses, public incident reports, external feedback and other data. The gaps below share a root cause: impacts get listed but the ratings behind them are not repeatable, traceable or current. Each card names one gap and a closing action the organization can take and check.

Impacts listed but never rated

Many inventories name harms such as biased outcomes, wrong advice or data exposure, then stop. MAP 5.1 expects each identified impact, beneficial or harmful, to carry a documented likelihood and magnitude. Close the gap by adding two fields to every impact record, a likelihood rating and a magnitude rating, with a named owner and a review date. Check that no impact in the register is blank on either field before a go or no-go decision is made.

Ratings with no traceable inputs

A rating of "high" with no note on how it was reached cannot be defended to an assessor. Record the inputs for each rating: the expected use, comparable past uses, public incident reports reviewed, user or external feedback and any test results. Ask whether a second reviewer could reach the same rating from those inputs. If not, the rating needs a rationale field before it is accepted.

Teams using incompatible scales

Product, security, legal and data science teams often rate the same system on different scales, so their results cannot be compared or added up. The Playbook suggests a uniform impact assessment scale across the AI portfolio, which can be qualitative, such as red-amber-green, or based on simulation. Agree one scale, define each level in writing with examples, and require every team to use it. Confirm that a rating from one system can be compared with a rating from another.

No re-rating after model or data changes

A rating made at launch can be wrong after a model update, a new training data source, a wider user base or a change in the deployment setting. MAP 5.1 ratings should be revisited when the system's context changes. Tie re-rating to the change process: a model version change, a new data source or a new use case triggers a rating review before release. Keep the prior rating and the new one side by side so the change is visible.

Generative AI provenance harms left unranked

For generative AI, NIST AI 600-1 suggests identifying content provenance harms, such as deepfakes or tampered content, and ranking them by likelihood and impact. Many programs list these harms under general misinformation and never rate them separately. Add provenance harms as their own line items, rate them with the same scale, and test them through red-teaming or adversarial role-play to find failure modes the team did not expect. Record the test results against each rating.

Ratings that exclude affected communities

Ratings written only by the builders tend to miss harms that fall on people outside the team, such as applicants, patients or customers in a particular group. Include the perspective of affected individuals and communities in the rating process, through structured feedback, user research or review by people who represent those groups. Record which perspectives informed each rating. Where feedback disagrees with the internal rating, document how the disagreement was resolved.

Questions buyers ask before they hire help for MAP 5.1

Can we do MAP 5.1 in-house without an outside partner?

Yes, in most organizations the core work can be done internally, because NIST AI 100-1 is voluntary and no one is required to use it. MAP 5.1 asks you to identify and document the likelihood and magnitude of each impact, beneficial or harmful, using expected use, past uses in similar contexts, public incident reports, external feedback and other data. Your AI product, risk and legal staff own those judgments. An outside partner is useful when your team lacks time, has no tested impact scale, or needs independent testing evidence such as red-teaming. Be aware that the NIST AI RMF Playbook is a suggestion set, not a checklist, so the scope of your own program is a decision you must document.

What drives the cost and effort of MAP 5.1 work?

Cost depends on how many AI systems are in scope, how many impacts each one has, and how much testing evidence you need to support the likelihood and magnitude ratings. The main cost components are internal staff time for impact workshops and documentation, any external assessment or testing work, and tooling. Costs recur when systems change, when new use cases appear, and when incident data should be reviewed. To control costs, define a uniform impact assessment scale across your AI portfolio first, then scope systems by risk. When you request a quote, ask for the written list of systems, deliverables, testing activities and the documents each one produces.

How long does MAP 5.1 work take?

No duration is supplied for this work, so treat any timeline you receive as an estimate to be confirmed in writing. The length depends on the number of systems, whether impact scales already exist, how much past incident and external feedback data is available, and whether red-teaming or testing at key lifecycle stages is included. Organizations with an existing risk register and clear system inventories usually move faster than those starting from no documented context. Ask any partner to show the phases, dependencies on other MAP and MEASURE work, and the dates that depend on your staff's availability.

What does the work require from our staff?

The work needs named people from the teams that design, build, deploy and monitor the AI system, plus people who understand the affected users and any legal or regulatory exposure. The Playbook tags MAP 5.1 with several AI actor categories, including AI design, development, deployment, impact assessment, operation and monitoring, and end-users. AI 100-1 Appendix A describes the tasks of each category. Expect staff to provide system documentation, usage data, complaint and incident records, and time for review sessions. Assign one accountable owner in your organization, because an implementation partner can prepare evidence but cannot make your impact decisions.

Is there a certification for MAP 5.1 or the NIST AI RMF?

No. The NIST AI RMF has no certification, accreditation or conformity assessment program, and NIST does not validate or endorse any individual organization's use of it. Neither an implementation partner nor DeployOpen issues reports or certificates. If a customer or contract asks for evidence, the organization owns its documented MAP 5.1 process, and an independent auditor or assessor can review it and issue whatever report that engagement type provides. Confirm with the requesting party exactly what evidence and what kind of independent review they accept before you start.

What should our scope document contain before we engage a partner?

Write down the AI systems in scope, their intended purposes and expected users, the impact categories you already recognize, and the data sources you can share, such as past uses, incident reports and external feedback. Add the impact scale you use or plan to use, the lifecycle stages where testing applies, and whether red-teaming or generative AI actions from NIST AI 600-1 are included. List the deliverables you expect, who signs off, and which parts you will keep in-house. Fixed-scope engagements are agreed in writing before work starts, so a clear scope document is what makes a quote comparable across vendors.

Which tools can support MAP 5.1 evidence, and what do they not do?

Open-source tools can hold and organize evidence, but none of them decide likelihood or magnitude for you. CISO Assistant, licensed under AGPL-3.0-only, supports audits and compliance assessments across many frameworks, including NIST CSF and ISO 27001, with automatic mapping between frameworks. garak, licensed under Apache-2.0, probes language models for issues such as misinformation, toxicity and jailbreaks and writes reports of attempts and vulnerabilities. Langfuse, licensed under MIT, records traces of LLM calls, tool executions and token usage that can support monitoring of impacts. Confirm each tool's fit with your environment before adopting it, and check the license terms that apply to your use.

How DeployOpen prepares MAP 5.1 evidence

DeployOpen works on MAP 5.1 by preparing the systems, test records and documentation that show how an organization estimates the likelihood and magnitude of each identified impact. The organization owns its impact judgments and risk decisions, and DeployOpen implements the self-hosted open-source tooling and evidence trail that support them.

How we scope

The written scope names which AI systems, impact categories and lifecycle stages the MAP 5.1 work covers.
Scope records which impact sources are in use, such as expected use, incident reports, user feedback and monitoring data.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins.

How we build

We deploy self-hosted tooling inside your own infrastructure so impact records stay under your control.
We configure trace logging and evaluation pipelines that capture inputs, outputs and failure cases for each assessed system.
We run red-teaming and adversarial testing against generative AI systems and record which failure modes were found and how severe they were.

How we hand over

You receive documented impact scales, likelihood and magnitude records, and the test logs that support each rating.
Handover includes runbooks explaining how staff update ratings when new incidents or external feedback arrive.
Your assessors and risk owners can trace each documented rating back to the data and testing behind it.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after the initial handover.
Retainer work can refresh impact ratings when systems change, new incident reports appear or the NIST Playbook is revised.
Independent auditors and certification bodies, not DeployOpen, review and issue any assessment reports or certificates.