NIST AI 600-1 confabulation risk: managing hallucinations

NIST AI 600-1 treats confabulation as a natural property of generative models that needs grounding, provenance checks and post-deployment monitoring. This guide explains what the profile asks for and who does the work.

On this page

What NIST AI 600-1 asks for on confabulation and hallucinations

NIST AI 600-1 has no certification and no pass or fail test for confabulation. No assessor will hand out a score for hallucination control, so the practical question is how your organization manages the risk and what evidence it can show that the management works. The profile defines confabulation as the production of confidently stated but erroneous or false content, often called hallucinations or fabrications, by which users may be misled or deceived. NIST describes it as a natural result of generative models approximating the statistical distribution of their training data. That framing matters for planning: you should expect confabulation to occur and design controls around detection, containment and response rather than assume it can be eliminated by a single model choice.

The profile's suggested actions relevant to this risk are MS-2.5-005, which covers verifying the provenance of training and TEVV data and checking that fine-tuning or retrieval-augmented generation data is grounded, and MG-4.1-002, which covers post-deployment monitoring procedures with attention to potential confabulation. Both are voluntary. Your organization decides which actions to adopt, how to adapt them to its own systems, and what risk tolerance applies. Confirm with your compliance lead which AI RMF subcategories your internal policy maps to these actions, because the profile ties each action to a framework subcategory and your evidence should follow the same links.

Responsibilities split cleanly. Your organization owns the controls, including the grounding sources, review rules, monitoring thresholds and escalation paths. An implementation partner can prepare the systems, logging, test sets and evidence records that show those controls operating. Only an independent auditor or assessor can report on whether the controls meet a stated standard. If a customer or regulator asks for proof, ask your partner for a written scope that lists which systems, data sources and monitoring procedures are covered, and ask your assessor to confirm the criteria they will test against before fieldwork begins.

Where confabulation causes harm in consequential decisions

NIST AI 600-1 treats confabulation as most dangerous when generative output feeds a consequential decision. The cards below set out where that happens, the harm it can cause and which trustworthy characteristics each scenario touches. The first card uses the profile's own example; the others are illustrative.

Clinical patient summaries

The Generative AI Profile's own example: a confabulated summary of patient information leads doctors to an incorrect diagnosis. The harm falls on the patient, and the clinician may not notice the error because the summary reads as authoritative. It touches safe and valid and reliable. Check whether summaries are traced back to source records and whether clinicians can see which statements the source supports.

Credit and underwriting notes

Illustrative: a model drafts an underwriting memo that cites income figures or prior loan history that do not exist in the file. A reviewer relying on the memo may approve or decline an applicant on invented grounds. This touches fair with harmful bias managed when errors fall unevenly across applicant groups, and explainable and interpretable because the stated reasons are not the real ones. Check that every cited figure traces to a system of record.

Legal research and citations

Illustrative: a research assistant returns case citations that look authentic but do not exist, or cite real cases for holdings they never made. Counsel who files or relies on them faces professional and sanction risk. The profile notes that confabulated citations can make an answer look justified, which invites trust in the wrong output. It touches valid and reliable and explainable and interpretable. Check that each citation is verified against an authoritative database before use.

Regulatory filings and reports

Illustrative: a drafting tool summarizes a control test or incident in a regulatory submission and states a figure or remediation date the underlying evidence does not support. The organization then files an inaccurate statement with a supervisor. This touches valid and reliable and accountable and transparent when no one can show which sources produced each statement. Check that generated figures are reconciled to system records before submission.

Security incident triage

Illustrative: an assistant summarizing alerts invents a root cause or states that a system was not affected. Responders who accept the summary may close an active incident or direct effort away from the real compromise. This touches safe and valid and reliable. Check whether triage summaries link each claim to log entries and whether analysts must confirm high-severity conclusions against raw telemetry.

How an organization plans its confabulation controls

The sequence moves from mapping where confabulated output could cause harm, through choosing measures and making go or no-go decisions, to monitoring the system after release. Each step closes with a record that an assessor or internal reviewer can inspect.

  1. Map the use case and consequential decisions

    Start under the MAP function by recording what the generative AI system is for, who uses its output, and which decisions it feeds. Note where the output could be relied on in a consequential decision, such as a clinical summary, credit file note or regulatory filing. Domain expertise and open-ended, long-form prompts are the conditions AI 600-1 links to confabulation, so record those explicitly. The output is a context record that names each consequential decision and the people accountable for it. The step is done when a domain owner has signed off that list.

  2. Identify where confabulation could mislead people

    For each mapped use, document the forms confabulation could take in that setting: invented facts, wrong summaries, fabricated citations or logic that appears to justify an answer. Record who would act on the error and how a person would notice it. AI 600-1 notes that confabulated citations can lead humans to trust incorrect output, so the review should test whether a reader could detect a fabricated source without outside checking. The output is a risk register entry per use case, tagged to the trustworthy characteristics AI 600-1 links to confabulation: fair with harmful bias managed, safe, valid and reliable, and explainable and interpretable.

  3. Verify the data that grounds the system

    Under the provenance action MS-2.5-005, confirm where training, evaluation and fine-tuning data came from, and confirm that retrieval or fine-tuning sources are the ones the program approved. Check that retrieval sources are current, owned, and documented for the use case. The output is a data provenance record listing each grounding source, its owner, and its last review. The step is complete when the team can show which source supports a given answer type and when a source was last checked.

  4. Choose confabulation measures and metrics

    Under MEASURE, select the metrics and test methods that match the mapped risks, following the MEASURE 1 approach of prioritizing the most significant risks. Typical methods include evaluating answers against a verified reference set for the domain, checking citations resolve to real and relevant sources, and having subject-matter experts review a sample of outputs. Define the threshold that triggers escalation. The output is a measurement plan that names each metric, its owner, the test data, and the acceptance threshold. Ask the team to confirm that independent reviewers, not only the builders, will review results.

  5. Run and document pre-release evaluations

    Execute the measurement plan on a test set that resembles deployment conditions, including the open-ended and long-form prompts the mapping identified. Record results by use case, including failures, the examples that failed and any cases where confabulated logic or citations appeared plausible. The output is a TEVV evaluation report that states the results against each threshold and lists known limitations. Confirm that the report is reviewed by someone outside the team that built the system.

  6. Make the go or no-go and treatment decision

    Under MANAGE 1, a named decision owner reviews the evaluation report and the risk register and decides whether the system may go live for each use case. Where results fall short, the decision may limit the use case, add human review before output reaches a decision, restrict the system to lower-stakes tasks, or defer release. Record the residual risk accepted and who accepted it. The output is a signed decision record per use case with the conditions attached. The step is complete when every consequential use has an explicit decision and no use has launched without one.

  7. Operate post-deployment monitoring

    Under MG-4.1-002, establish procedures for monitoring the deployed system, with confabulation named as a priority, and assign the people who review samples, user reports and error logs. Set a schedule for repeating evaluations and a route for pausing the system when monitoring shows a rise in errors. The output is a monitoring plan and a log of reviews, findings and actions taken. Ask the vendor or internal platform team to confirm how logs and user reports are captured, because the evidence depends on that data existing.

  8. Feed incidents and changes back into the controls

    When monitoring finds confabulated output that reached a user or decision, record it as an incident, correct the affected use, and update the mapping, measurement plan or grounding sources as needed. Under MANAGE 4, communicate significant errors to the relevant parties. The output is an incident record linked to the risk register, plus a revised version of any control it changed. The step closes when the change has been tested and the decision owner has approved it, which restarts the cycle for that use case.

Grounding, provenance and monitoring actions for confabulation

Action MS-2.5-005 asks organizations to confirm that data used for fine-tuning, or fed into retrieval-augmented generation, is grounded in sources the organization can vouch for. The concern is that a model answering from weak, outdated or unverified material will confabulate with confidence. A retrieval layer can also make an answer look sourced when the underlying document is wrong, missing or superseded. The work usually falls to the teams that curate the knowledge base and the teams that run fine-tuning jobs.

  • Source inventory: List every document, database and fine-tuning dataset the system uses, with an owner and a last-reviewed date.
  • Retrieval coverage tests: Sample questions whose answers exist in the corpus, then check that the retrieved passages actually support the generated answer.
  • Citation fidelity: Confirm that each citation points to a passage containing the claimed content, since the profile notes that confabulated citations can make wrong answers look justified.
  • Stale content handling: Define how withdrawn policies or superseded guidance are removed or flagged in the index.
  • Evidence to keep: Corpus manifests, retrieval test results, citation sampling records and change tickets for index updates.

Questions to ask: Who approves a source before it enters the index? What happens to answers already generated from a source that is later withdrawn? Check whether your team can show, for a sampled answer, which source it came from and whether that source says what the answer claims.

What must exist before an assessment can test confabulation controls

Confabulation controls are only testable when the surrounding records exist: which systems generate content, where their grounding data came from, how errors are measured, and who reviews consequential outputs. The six items below are the internal checks an organization should complete before inviting an independent assessor to examine its controls.

Generative AI system inventory

List every generative AI system in use or in development, including vendor-hosted tools, and mark each consequential use, such as summaries that feed clinical, credit or legal decisions. Check that the inventory is reviewed on a set schedule and that owners can name the outputs their system produces. The inventory shows which confabulation risks apply to which system, and it is the starting point for MAP 1 and MAP 2 documentation. Evidence: a dated register with system owner, purpose, output type and consequential-use flag.

Data provenance for training and retrieval

Document where training data, fine-tuning data and retrieval-augmented generation sources came from, who approved them and how they are kept current. Confirm that retrieval sources are grounded in authoritative records and that stale or unverified documents are removed. NIST suggests verifying the provenance of this data as part of MS-2.5-005. Evidence: a source register with origin, owner, approval date, refresh cadence and removal log.

TEVV records for factual accuracy

Keep test and evaluation records that show how the system was checked for false or unsupported content before release, including the test sets, domain experts involved and the pass or fail decisions. Confirm the tests reflect the domains and prompt types where confabulation is most likely, such as open-ended long-form answers. Evidence: versioned test sets, evaluation reports, reviewer sign-off and records of changes made after failures.

Confabulation metrics tied to mapped risks

Define how confabulation is measured for each system and link each metric to a risk recorded in the MAP outputs. Check that the metric covers the failure that matters, such as a wrong diagnosis or a fabricated citation, not only overall response quality. Confirm thresholds are approved by the accountable owner. Evidence: a metric definition sheet listing the risk, measurement method, threshold, owner and review date.

Logged post-deployment monitoring results

Confirm that production outputs are sampled and reviewed for confabulation, that results are logged and that escalation paths exist when errors are found. NIST suggests procedures for post-deployment monitoring under MG-4.1-002, particularly for confabulation. Check that logs record the input, output, reviewer finding and any corrective action. Evidence: monitoring logs, sampling records, incident tickets and trend reports reviewed by management.

Documented human oversight for consequential outputs

Define which outputs a person must verify before action, who that reviewer is and what authority they have to override the system. Check that reviewers are trained to spot confabulated logic or citations that appear to justify an answer, since NIST notes such content can lead people to trust incorrect output. Evidence: an oversight procedure, reviewer role assignments, training records and override logs.

Choosing between in-house controls and outside help

This table compares who does what for confabulation controls: building them in-house, engaging an implementation partner, or bringing in an independent assessor. Read each row as a division of work, with the organization keeping ownership of its risk decisions in every option. The main takeaway is that an implementation partner can prepare grounding, provenance and monitoring evidence, but only an independent assessor can test and report on it.

AreaBuild and run in-houseEngage an implementation partnerIndependent assessment
Grounding setup for retrieval and fine-tuning dataInternal ML and data engineers select sources, define grounding rules and test answers against themPartner configures retrieval pipelines and data lineage on the organization's own infrastructure, with the organization approving source choicesAssessor samples grounded outputs and checks whether the grounding design matches documented intent
Provenance checks on training and TEVV dataData governance team verifies source provenance and keeps the records itselfPartner builds lineage tooling and checklists that capture provenance records for reviewAssessor tests whether provenance records exist, are complete and are reviewed on a stated schedule
Monitoring design for post-deployment confabulationPlatform team defines metrics, alert thresholds and review queues using existing observability toolsPartner designs monitoring pipelines, sampling methods and dashboards, while the organization sets thresholds and owns responsesAssessor reviews whether monitoring procedures are evaluated for effectiveness and whether findings led to action
Evidence packs for internal and external reviewCompliance staff assemble logs, test results and decision records as they arisePartner structures evidence packs mapped to the organization's own control list and keeps them currentAssessor reads the evidence independently and forms an opinion; no partner evidence pack is an audit result
Ownership of risk decisions and accountabilityNamed executives and risk committee accept residual confabulation risk and approve deploymentPartner advises and documents options, but accountable roles stay inside the organizationNot applicable; assessor reports observations and does not accept or manage risk
Best fitMature AI teams with staff who already run evaluation, data governance and monitoring at production scaleTeams building their first generative AI controls, or teams that need a fixed-scope build with written deliverablesOrganizations whose customers, contracts or regulators ask for independent testing of controls
Main risk of this optionConfabulation tests stay narrow and monitoring is not updated after model or data changesHandover gaps if the organization cannot operate the tooling after the engagement endsAssessment only covers the scope and evidence provided, so unclear scope leads to weak findings

Who owns, prepares and assesses confabulation controls

The organization that deploys or offers the generative AI system owns its confabulation risk. No outside party can accept that risk, decide how much erroneous output a consequential workflow may tolerate, or sign off on a go or no-go decision. NIST AI RMF has no certification, so the organization is also the party that decides which NIST AI RMF subcategories it adopts and how it documents them.

  • Risk decisions: Set the acceptable error rate for each use case, decide where human review is mandatory, and record the residual risk.
  • Accountability: Name the owner of confabulation monitoring and the escalation path for a harmful output, following the roles documented under GOVERN 2.
  • Policy and scope: Define which tasks may use generative output without verification, such as patient summaries, legal citations or financial figures.
  • Vendor oversight: Keep responsibility for third-party models and retrieval sources, including contract terms for model updates.
  • Evidence ownership: Retain logs, review records and monitoring reports in the organization's own systems.

Before engaging any partner or assessor, confirm internally which person can approve a risk acceptance and where the evidence will be stored.

Questions buyers ask about confabulation controls

Can NIST AI 600-1 confabulation controls be certified?

No. NIST AI 600-1 is a profile of the NIST AI Risk Management Framework, and NIST does not run a certification, accreditation or conformity assessment program for the AI RMF or its profiles. NIST also states that it does not validate or endorse any individual organization's use of the framework. What a buyer can ask for is evidence: documented grounding and provenance checks, test results and monitoring records that an internal reviewer or an independent assessor can examine. Any party that claims to certify confabulation controls under NIST AI 600-1 is making a claim NIST does not support, so ask them to name the body that issues the credential and how it is accredited. DeployOpen prepares systems and evidence for assessment but does not issue reports or certificates.

How much effort do confabulation controls take, and what drives it?

Effort depends mostly on how the system is used, not on the profile itself. The main drivers are the consequence of a wrong answer, the number of data sources the model draws on, how often the knowledge base changes, the number of users and the domains covered. A system that summarizes internal documents for drafting carries less effort than one that feeds clinical, credit or legal decisions. Expect one-time work for grounding design, provenance checks on training and retrieval data, and test sets built from real domain questions. Expect recurring work for monitoring, reviewing flagged outputs and refreshing test sets as the data changes. The NIST AI 600-1 profile does not publish a cost or duration, so ask any provider for a written scope that lists the systems, data sources and decision types covered before comparing quotes.

What does the work need from our internal staff?

Internal staff must own the decisions that an outside partner cannot make for you. These include defining which decisions the system may support, setting the tolerance for erroneous output, deciding who reviews flagged outputs, and accepting residual risk. Subject matter experts are needed to judge whether answers are correct in the domain, because confabulation is most relevant where users need contextual or domain expertise. Engineering staff need access to the model stack, retrieval pipeline, logs and deployment environment. Plan for named owners in risk, data and operations, and make sure someone can pause or roll back the system when monitoring shows a problem. A partner can prepare the technical controls and evidence, but accountability for the control environment stays with your organization.

Can a customer contract require NIST AI 600-1 compliance?

A contract can require an organization to follow NIST AI 600-1, or to provide evidence of its confabulation controls, because NIST publishes the profile for voluntary use and parties may agree to any terms they choose. The enforceable part is the wording. Ask for a defined scope, a list of the controls or suggested actions adopted, the evidence to be provided, the frequency of reporting and the right to audit by an independent party. Avoid clauses that require a NIST certificate, since none exists. Where the customer is subject to a law that names the AI RMF, the legal obligation may differ from the contract, so have counsel review both. Use the outline of this page as a checklist for what to put in scope, not as legal language.

How does post-deployment monitoring differ from pre-deployment testing?

Pre-deployment testing checks a system against a defined test set before release, and it shows how the system behaved on those questions. Post-deployment monitoring checks how it behaves on live traffic, where users ask questions the test set did not anticipate and sources change over time. The GAI Profile action MG-4.1-002 suggests establishing, maintaining and evaluating procedures for post-deployment monitoring, particularly for potential confabulation. In practice, monitoring includes logging prompts, retrieved sources and outputs (subject to privacy rules), sampling answers for expert review, tracking citations that do not match sources, and defining thresholds that trigger investigation or rollback. Testing shows the system is ready; monitoring shows it stays ready. Budget for both, and confirm with your provider which logs and review steps they will build.

How do we know an implementation partner is qualified?

Ask for evidence of work that matches your environment, not general claims about AI. A qualified partner should explain how they ground model outputs, verify the provenance of training and retrieval data (MS-2.5-005), and design post-deployment monitoring (MG-4.1-002). They should state clearly that they do not issue audit reports, attestations or certifications, and that independent auditors or certification bodies perform those tasks. Look for a fixed written scope with deliverables and timelines agreed before work starts, named staff with relevant experience, and a plan for handover of logs, documentation and tests. Ask how they will test their own work, including red teaming, and whether they can support the system after launch under a separate written retainer. Be cautious about any partner that promises a result or a certificate.

Common reasons confabulation programs stall

The first reason programs stall is treating a hallucination rate from a public benchmark as proof of how the system will behave in production. A benchmark score tells you how a model performed on a test set built by someone else, under prompts that may not resemble your users' questions. AI 600-1 is explicit that confabulation is especially relevant for open-ended prompts, long-form responses and domains that need contextual or domain expertise, which is where production traffic usually lands. The practical step is to build a confabulation test set from your own intended use, including the long-form answers and domain questions your staff actually receive, and to run it against the exact model version, prompt template and retrieval settings you plan to ship. Confirm with your model provider which version a benchmark result refers to, because a change to the model or its system prompt can invalidate an earlier score.

The second reason is retrieval sources that nobody documented. In a retrieval-augmented system, many confabulations look grounded because the answer cites a document, yet the cited passage does not support the claim or the index contains outdated or unapproved material. AI 600-1 notes that outputs may include confabulated logic or citations that justify an answer, which can lead people to trust incorrect output. Teams stall when they cannot say which corpora feed the index, who approved each source, when it was last refreshed, or how a citation is linked to its passage. Keep a source register that records owner, origin, refresh date and access rights for every corpus, and log retrieved passages alongside each answer so a reviewer can check whether the cited text actually supports the output.

The third reason is that nobody owns post-deployment monitoring. The GAI Profile action MG-4.1-002 asks organizations to establish and evaluate procedures for monitoring generative AI systems after deployment, particularly for potential confabulation. Without a named owner, the monitoring plan exists on paper while sampled outputs go unreviewed, user complaints about wrong answers are routed to a general helpdesk, and no one decides when a rising error rate triggers a rollback or a retraining review. Assign a single accountable role for confabulation monitoring, define the sample size and review cadence in writing, and set escalation thresholds with the business owner who can approve a change. Ask your team which role would be paged at 2 a.m. if a confabulated output reached a clinician or a customer, and if the answer is unclear, the program is not yet operating.

How DeployOpen prepares confabulation controls and evidence

DeployOpen plans and builds the technical and documentary side of confabulation management for generative AI systems that run inside the client's own infrastructure. The organization owns its controls and decisions, and an independent auditor or assessor, not DeployOpen, reviews and reports on them.

How we scope

  • Written scope names the generative AI systems, use cases and decisions where confabulated output could reach people.
  • Scope lists the data sources, retrieval components and fine-tuning datasets whose provenance must be verified.
  • Scope states which monitoring, logging and review activities the client owns and which DeployOpen prepares.

How we build

  • We help configure grounding and retrieval pipelines so outputs can be traced to approved source documents.
  • We set up provenance records for training, evaluation and retrieval data used in the system.
  • We implement logging and monitoring that flags suspected confabulations for human review in production.

How we hand over

  • Deliverables include documented procedures for post-deployment monitoring aligned with the GAI Profile's MG-4.1-002 action.
  • Evidence packs record data provenance checks, test results and monitoring configurations for the client's assessor.
  • Handover includes runbooks so the client's own staff can operate, review and update the controls after delivery.

How we support

  • Ongoing support, upgrades and continuous testing are available as a retainer after handover.
  • Retainers can cover periodic review of confabulation test results and changes to retrieval sources.
  • Ask any partner to confirm in writing who signs the assessment report and who keeps the monitoring records.

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.