GOVERN 1.5 ongoing monitoring: requirements and evidence

What GOVERN 1.5 asks for and why organizations struggle with it

When a customer, contract or regulator asks how AI risk stays under review after launch, the honest answer starts with this: GOVERN 1.5 is a planning outcome that requires ongoing monitoring and periodic review of the AI risk management process, with clearly defined roles, a set review frequency and feedback paths for the people the system affects. NIST AI 100-1 does not certify anyone against it, and no organization can be certified against the AI RMF at all. The framework is voluntary, and NIST states that it does not validate or endorse any organization's approach to using it. What you can show is a documented plan, the records it produces and the decisions it drove.

The risk GOVERN 1.5 addresses is simple to describe and easy to miss: an AI system behaves differently after deployment than it did in testing. Data shifts, users find new ways to use it, models are updated by a vendor and bias or security problems appear with no one assigned to notice. Organizations struggle for two common reasons. First, monitoring is configured for availability, with dashboards for uptime, latency and errors, but not for the trustworthiness problems the Playbook names, including bias and security problems. Second, review is scheduled late or never, so the first formal look at the system happens after an incident. Confirm with your own teams which signals are collected today and who reads them.

Which AI systems, teams and third parties fall under GOVERN 1.5

GOVERN 1.5 asks for monitoring and periodic review that is planned, owned and scheduled, but the amount of work depends on which systems, teams and suppliers sit inside the plan. The cards below cover the scoping decisions that change effort most, and each one should be settled in writing before monitoring is built.

AI systems in the monitoring plan

Start from the AI system inventory described under GOVERN 1.6 and the risk tiers set under GOVERN 1.3, then decide which entries need ongoing monitoring and at what depth. A high-tier system that affects credit, hiring or patient care usually needs performance, bias and security monitoring on a defined schedule. A low-tier internal tool may need only an annual review. Check that every production system appears in the inventory, because a system missing from the inventory will not appear in the monitoring plan either.

Monitoring owners versus incident responders

The Playbook suggests naming the functions responsible for monitoring and for incident response, and these are often different teams. Monitoring usually sits with the operations or model owner team, while incident response may run through security or a risk function. Write down who receives an alert, who decides it is an incident and who signs off on closure. Unclear handoffs between these teams are a common reason monitoring findings are logged but never acted on.

Third-party models and hosted AI services

Decide whether vendor models, API-hosted AI services and pre-trained models you fine-tune are inside the monitoring plan. You may not control their training data or internal testing, so monitoring usually focuses on your inputs, outputs, performance drift and vendor change notices. Confirm with each vendor what logs, change notifications and incident notices they provide, because those set what your own monitoring can see. Contract terms that define these obligations are often the limit of what you can monitor.

Self-hosted models and infrastructure

When models run in your own cloud accounts or data centers, monitoring covers the serving stack as well as model behavior. Include logs from inference endpoints, access records, resource usage and the components that retrieve data or call tools. Tracing tools such as Langfuse can capture the operations in a request, which supports both performance review and incident reconstruction. Confirm which telemetry is retained, for how long and where, because retention decisions affect what a periodic review can examine later.

Generative AI systems and NIST AI 600-1 actions

Generative AI systems in scope should also receive the GOVERN 1.5 suggested actions in NIST AI 600-1, the Generative AI Profile. These cover after-action reviews of generative AI incident response and incident disclosures, plus document retention for testing records and digital content transparency records. Decide early whether your generative AI systems meet this bar, since the extra actions add review steps and record keeping that non-generative systems do not need. Retention periods should be set by legal and records owners, not only by the AI team.

Affected people and feedback channels

GOVERN 1.5 also covers feedback, recourse and opt-out mechanisms for people affected by AI decisions. Decide which systems expose these channels, who reviews complaints, and how decisions are reversed or escalated. Systems that make or shape decisions about individuals need these channels more than systems that only summarize internal documents. Check whether complaints feed back into the monitoring plan, since a recourse process that does not inform monitoring leaves the same problems in place.

How an assessor tests ongoing monitoring and periodic review

This table compares six test areas for GOVERN 1.5, showing the approach, the evidence requested and the findings that usually come up in each. NIST runs no assessment program for the AI RMF, so the assessor here is the organization's internal audit, a customer's auditor or a contracted reviewer testing against the organization's own stated requirements. Use it as a pre-review checklist before any of those reviews begin.

Test area	Test approach and evidence requested	Common findings
Monitoring policy and lifecycle coverage	Read the policy for monitoring of AI performance and trustworthiness, including bias and security problems. Confirm it applies from pre-deployment through retirement, and request a sample of monitoring records for two or three systems.	Policy covers production only, so pre-deployment testing and retirement have no monitoring rules. Bias and security are named in the policy but no metric or alert exists for them.
Named roles for monitoring and incident response	Request a responsibility matrix or RACI and the job descriptions or terms of reference that assign monitoring and response duties. Check that named owners match the people who actually receive alerts.	Roles are listed by team rather than by person, so no one owns the alert queue on nights or weekends. The named owner has left the organization and the matrix was not updated.
Review frequency and on-schedule completion	Confirm the review frequency is written down, then pull the calendar entries, meeting minutes and sign-off records for the last review cycles. Compare dates against the stated schedule.	A frequency is stated but no records show reviews happened. Reviews occurred late, or only after an incident, and the minutes record no decisions or follow-up actions.
AI incident response policy or confirmed coverage	Request the incident response policy and check whether it names AI systems, model failures and AI-specific events such as harmful outputs or data leakage. If the general policy is used, request a written confirmation of coverage and a tabletop exercise record.	The general security incident policy is used but never mentions AI. Model drift, harmful output and prompt injection are not classified as incidents, so they are not escalated.
After-action reviews and incident learning	Sample two or three past AI incidents or near misses. Ask for the after-action review, the gap list and evidence that the gaps led to changes in controls, tests or documentation.	Incidents are closed without a review. Action items are logged but never tracked to completion, and the same failure recurs in a later release.
Feedback, recourse and opt-out mechanisms	Request the published or internal process for affected people to report problems, appeal a decision and opt out where that applies. Check that intake records are kept and that responses and timing are logged.	The feedback channel exists but is not linked to the monitoring queue. Opt-out requests are handled by hand with no record, or the process is not explained to the people affected.

How to plan ongoing monitoring and review, step by step

This sequence turns the GOVERN 1.5 outcome into a working plan: a defined scope, monitoring rules, named owners, a review schedule, an incident process and a way for affected people to be heard. Each step ends with a written artifact the organization can show a reviewer.

Confirm the monitoring scope from the AI inventory
Start from the AI system inventory (the GOVERN 1.6 record) and list every system that needs ongoing monitoring, including third-party models and any generative AI tools in use. Mark which systems are high-impact or customer-facing so later steps can apply deeper monitoring to them. Done when the scope list is signed off by the system owners and each entry names its business owner, deployment status and risk tier. Ask your team to confirm that shadow AI tools reported outside procurement are either added to the list or formally excluded with a reason.
Set monitoring policies for performance and trustworthiness
Write the rules for what is watched across the lifecycle: accuracy and reliability drift, bias and fairness indicators, security problems such as prompt injection or data leakage, and output quality for generative systems. For each signal, record the metric, the data source, the threshold that triggers action and who receives the alert. The Playbook offers suggested actions for GOVERN 1.5, but it is not a checklist, so keep only what fits each system's risk. Done when a monitoring policy exists, approved by the governance body, with one line per monitored signal.
Assign owners for monitoring and incident response
Name the function responsible for each monitoring signal and for each stage of incident response, including who detects, triages, escalates and communicates. Separate the people who build a system from those who judge its monitoring results where your size allows, and document the lines of communication to legal, privacy and business leaders. The NIST AI 100-1 actor categories for governance and oversight and for operation and monitoring are a useful starting vocabulary. Done when a role table shows each responsibility with a named person or team and a backup.
Set the review frequency and review triggers
Decide how often each system's monitoring and risk management process is reviewed, and set the higher-risk systems on a shorter cycle. Also define event-based triggers that force an unscheduled review, such as a material model change, a new deployment context, a significant incident, a new regulatory obligation or a monitoring threshold breach. Ask leadership to confirm the cycle, since no external standard in this framework sets a fixed interval. Done when a review calendar lists each system, its review interval and its trigger conditions, and the owner is recorded for each review.
Confirm or create the AI incident response policy
Check whether the existing incident response policy already covers AI-specific events such as harmful outputs, biased decisions, model compromise and data exposure through prompts. Where gaps exist, add AI incident categories, severity definitions, containment options such as disabling a feature or rolling back a model, and rules for internal and external disclosure. For generative AI, plan after-action reviews that examine both the response and any incident disclosure to find gaps. Done when the policy names AI incidents explicitly, the escalation path is tested in a tabletop exercise, and the lessons record feeds the next review cycle.
Build feedback, recourse and opt-out paths for affected people
Define how people affected by the AI system can report problems, contest an outcome and opt out where the use case allows it. Specify the intake channel, who reviews each case, the response expectations you will commit to in writing, and how a decision can be reversed or escalated to a human. Connect these channels to the monitoring signals and incident process so that complaints feed the review calendar. Done when the feedback and recourse procedure is published internally, the intake channel is tested, and each complaint category maps to an owner and an incident path. Confirm with legal counsel which opt-out and appeal rights apply in each jurisdiction.

Evidence that passes GOVERN 1.5 and how to keep it current

GOVERN 1.5 asks for a plan, named owners and a set review frequency, so the evidence is a set of documents and dated records that show the plan runs. The cards below cover the artifacts a reviewer typically asks for, and each one needs a maintenance habit between reviews so it does not go stale.

Monitoring policy for AI performance

This policy states what is monitored across the AI lifecycle, including performance, bias and security problems, and the thresholds that trigger escalation. It is the document every other artifact points back to. Check that it names the systems in scope, matches the AI inventory, and that the version history shows who approved each change. Review it on a fixed cadence that the organization sets, and record the review even when nothing changes.

Role assignments and RACI

A RACI table shows who is responsible, accountable, consulted and informed for monitoring, review and incident response for each AI system. Reviewers test whether the named people still hold those roles. Keep it current by tying each change to HR and access records, such as role changes or departures, and by reconfirming the table at every periodic review. Gaps often appear where a system was added to the inventory without an owner.

Periodic review record with dated minutes

Dated minutes with decisions, open actions and owners show that the review frequency is followed in practice. Each entry should list which monitoring data was examined, which risks changed and what was decided. Store minutes under version control or a records system with write-once controls, and set a calendar reminder so a missed review is visible. A review that produced no decisions still needs a minute recording what was checked.

AI incident response policy and after-action reviews

The incident response policy should cover AI-specific events, or cross-reference the existing security incident policy where it already does. Past incident records with after-action reviews show that gaps were found and closed. Maintain a case record for each incident with timeline, evidence and corrective actions. Tools such as DFIR-IRIS can keep each case as one record with an audit trail, though the organization should confirm that fit against its own process.

Feedback, recourse and opt-out procedure records

This procedure describes how affected people report problems, appeal decisions and opt out where the policy allows it. Evidence includes the intake channel, the log of requests, the decisions taken and the time to respond. Review the log at each periodic review and check whether repeated complaints led to changes in the system. Confirm with legal counsel which opt-out rights apply in each jurisdiction where the system is used.

Document retention policy for TEVV and transparency records

A retention policy sets how long testing, evaluation, verification and validation records and digital content transparency records are kept, and how they are disposed of. For generative AI, NIST AI 600-1 suggests keeping such a policy under GOVERN 1.5. The retention period should come from the organization's own legal and business requirements, not from a default. Review it when laws change, and confirm that logs and traces needed for review, such as those from Langfuse, are included in its scope.

What meeting GOVERN 1.5 costs in money and effort

This table lists the cost components of running GOVERN 1.5 monitoring and periodic review, who is paid for each, whether it is one-time or recurring, and what moves it up or down. No price or duration is given because none is supplied for this topic, so treat the table as a checklist for building a quote. Any figure you receive should follow from a written scope.

Cost component	One-time or recurring	What drives it and how to keep it down
Monitoring tooling and licenses	Recurring	Commercial monitoring suites charge per seat, system or volume. Open-source options such as Langfuse (MIT) for trace capture and garak (Apache-2.0) for model probing remove license spend, but hosting, storage and upkeep still need budget. Confirm each tool's current license terms before adoption.
Engineering time to instrument monitoring	One-time, then recurring upkeep	Effort rises with the number of AI systems, data sources and pipelines to instrument. Start with the highest-risk systems so the first instrumentation set is small and reusable.
Policy and role-setting time	One-time, with periodic revision	Drafting the monitoring policy, naming responsible functions and setting review frequency needs legal, risk and AI owner input. Reuse existing incident and change management policies where they already cover AI.
Recurring review meetings	Recurring	Cost scales with the number of systems reviewed, the number of attendees and how often reviews run. Tie the cadence to risk tier so low-risk systems do not consume the same meeting time as high-risk ones.
Evidence upkeep	Recurring	Dashboards, review minutes, monitoring logs and retention records must be kept current. Automated exports from the monitoring stack and a single evidence location reduce manual collection time.
Incident response exercises	Recurring	Tabletop or live exercises need facilitator time, participant time and after-action write-ups. Frequency and scope set the effort. Exercise the scenarios most likely to affect people, such as bias or data leakage.
Case management and investigation tooling	Recurring	Platforms such as DFIR-IRIS (LGPL-3.0-or-later) can host AI incident cases with audit trails, which removes license spend for self-hosting. Analysts still spend time on triage, timelines and reporting.
External review or testing	Recurring or per engagement	Independent red teaming, model evaluation or readiness review is priced by scope, system count and depth. Ask providers for a written scope and deliverables before comparing quotes.
Independent auditor sampling	Per assessment cycle	The auditor selects samples of review records, incident files and feedback logs. Costs rise when evidence is scattered or incomplete. Confirm the assessment scope and sample method with the auditor, since NIST AI RMF itself has no certification.

Which self-hosted open-source tools can produce GOVERN 1.5 evidence

GOVERN 1.5 asks for monitoring, review and incident records that show the AI risk process is actually running. The four tools below each produce a different slice of that evidence, from review cycle tracking to incident case files, model test logs and production traces. None of them defines the policies, roles or review frequency; those remain the organization's decisions to document.

CISO Assistant for review cycles

CISO Assistant (AGPL-3.0-only) ships more than 200 frameworks and runs audits and compliance assessments against them, with automatic mapping between frameworks. It can hold the dated review cycles, assessment results and findings that show periodic review took place and that gaps were tracked to closure. Check the AGPL-3.0-only license terms with counsel before modifying or offering the tool to others, and confirm that the framework libraries you need are included in your version. The review frequency and named owners must still come from your own governance policy.

DFIR-IRIS for incident records

DFIR-IRIS (LGPL-3.0-or-later) keeps each case as one record of assets, indicators, tasks and evidence, with a full audit trail. It triages alerts from SIEM, EDR and REST sources into cases and logs evidence with SHA-256 verification and chain of custody. These records let you show that AI incidents were handled, and an after-action review can be attached to the case. Confirm that its alert sources and case fields fit AI-specific incidents, such as model misbehavior, and check the LGPL obligations that apply to your deployment.

garak for recurring model tests

garak (Apache-2.0) probes language models for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks. Each run writes a JSONL report of every attempt and evaluation, plus a hit log of attempts that found a vulnerability. Scheduling runs against each release or on a fixed cadence creates a dated history that shows monitoring for those risks. It tests only the probes it includes, so confirm coverage against the risks in your own risk register and keep the retained reports as evidence.

Langfuse for production monitoring

Langfuse (MIT) records traces of each request, including LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs, token usage and metadata. These traces give reviewers a record of how a deployed system behaved over time, which supports monitoring for performance and trustworthiness problems. Confirm how inputs and outputs are stored, masked and retained, since traces can contain personal data. The monitoring thresholds and the people who review them must be set in your policy.

Who does what: your team, an implementation partner and the auditor

Your organization owns GOVERN 1.5. The outcome requires that monitoring and periodic review are planned, that roles and responsibilities are clearly defined, and that a review frequency is set. Those choices are management decisions, so they cannot be delegated to a vendor or consultant. Your team decides which AI systems are monitored, what counts as a material performance or trustworthiness problem, who can accept residual risk, and whether a review cycle leads to changes in the system, its controls or its use. The NIST AI RMF Playbook suggests naming the functions responsible for monitoring and incident response, and that naming is a decision your leadership signs off.

Policy and roles: Write the monitoring policy, assign named owners for monitoring, incident response and feedback handling, and record the review frequency with its rationale.
Risk acceptance: Decide which findings are fixed, deferred or accepted, and document who approved each acceptance.
Incident response function: Confirm whether existing incident response policies cover AI, staff the on-call and escalation path, and own disclosure decisions.
Affected-person mechanisms: Decide how people can give feedback, seek recourse or opt out, and who reviews those requests.
Review outcomes: Chair periodic reviews, record decisions and track follow-up actions to closure.

Check that every named owner has accepted the role in writing and that the review calendar is on someone's schedule, not just in a document.

Where GOVERN 1.5 programs usually fall short and how to close each gap

GOVERN 1.5 asks for monitoring, review, roles and incident handling to be planned together, so a gap in one usually weakens the others. The cards below cover the failure points we see most often in organizations that already have an AI policy but no working monitoring cycle. Each card names the gap and a concrete step that closes it.

Monitoring built for uptime only

Dashboards that track latency, errors and availability say nothing about whether a model is still biased, leaking data or being manipulated. GOVERN 1.5 expects monitoring of AI performance and trustworthiness across the lifecycle, including bias and security problems. Check whether each production AI system has at least one signal for drift, fairness and security, and whether someone reviews those signals on a set schedule. Tracing tools such as Langfuse can record inputs, outputs and token usage per request, which gives reviewers the raw material for these checks.

Review frequency without a review record

Many policies state that AI risk reviews happen quarterly or annually, but no dated minutes, sign-offs or findings exist to show they happened. An assessor will ask for the most recent review of each in-scope system and the actions that followed. Set a review calendar, assign a named owner to each review, and store the outcome, open actions and the decision to continue, change or retire the system in one controlled location.

Incident response policy that omits AI

The security incident plan covers outages and data breaches but never mentions model failures, harmful outputs or prompt injection. The Playbook suggests either creating AI incident response policies or confirming that existing ones cover AI. Add AI-specific triggers, such as a biased decision reported by a customer or a model that exposed training data, to the existing runbook. After each incident, run an after-action review to find gaps, as NIST AI 600-1 suggests for generative AI incidents.

No path for affected people to be heard

People affected by an AI decision often have no way to ask why it was made, challenge it, correct their data or opt out. GOVERN 1.5 suggests providing feedback, recourse and opt-out mechanisms for these people. Check whether a published contact route exists, whether requests are logged with a response deadline, and whether the outcomes feed back into the monitoring review. Test the route once a year by submitting a request and timing the reply.

Roles named on paper only

The policy assigns monitoring to the model owner, the risk function and the security team, but the people holding those roles may not know they are responsible or what they must do. Clear roles require a written responsibility matrix for monitoring, review and incident response, plus a short briefing for each role holder. Confirm each name against the current org chart, because staff changes often leave the policy pointing at people who have moved on.

Third-party model monitoring missing from the plan

Teams often monitor the models they train but assume that a hosted or pre-trained model from a vendor needs no monitoring. The risk changes when the vendor updates the model, alters its training data or changes its safety filters, and the organization may not be told. Include vendor models in the monitoring plan, define which vendor changes trigger a review, and record which version each system uses. The GOVERN 1.5 Playbook actions apply to these systems just as they apply to internally built ones.

Questions buyers ask about GOVERN 1.5 cost, time and ownership

Can GOVERN 1.5 be certified under the NIST AI RMF?

No. The NIST AI RMF has no certification, accreditation or conformity assessment program, and NIST does not validate or endorse any organization's use of the framework. GOVERN 1.5 is an outcome in a voluntary framework, so the question an assessor or customer can answer is whether your monitoring and review process exists, has defined owners, runs on a set review frequency and produces records. Only an independent auditor or assessor can issue a report or opinion on that work, and what they issue depends on the contract or regulation that asked for it. Ask the requesting party which report or attestation they expect before you start.

What must our own team do for GOVERN 1.5?

The organization owns the outcome, so the core decisions stay in-house. Your team must decide which AI systems are monitored, which trustworthiness characteristics and performance signals matter for each, who is responsible for monitoring and for incident response, how often the risk management process is reviewed, and how affected people give feedback, seek recourse or opt out. These are management decisions. An implementation partner can draft the policies and set up tooling, but the accountable owners need to approve them and show that the reviews really happen. Keep dated review minutes, decisions and open actions, since those records are the evidence an assessor will ask for.

What drives the cost and effort of GOVERN 1.5 up or down?

The main drivers are the number of AI systems in scope, how many of them are generative AI or third-party models, how much monitoring data already exists, and how many teams must agree on roles. Costs fall into one-time work (defining policies, setting roles, wiring logging and dashboards, writing the incident response procedure) and recurring work (running reviews, reviewing alerts, testing the incident process and updating records). Narrowing scope to a documented AI inventory with risk tiers usually reduces both. When requesting a quote, ask for the scope list, the monitoring data sources assumed, the number of review cycles included and which deliverables are documentation versus configured systems.

How long does GOVERN 1.5 take to put in place?

No duration is supplied for this work, and the timeline is set by scope. A single chatbot with existing logs can be covered far faster than a portfolio of high-impact models with several business owners and external vendors. The time depends on how quickly your organization approves policies, whether monitoring data is already collected, and how many review cycles you want to see before an assessment. Ask any partner for a written schedule tied to the agreed deliverables, and confirm which milestones depend on your internal approvals.

Can self-hosted open-source tools replace paid AI monitoring?

They can cover parts of the evidence trail, but they do not replace the governance decisions. Langfuse (MIT) captures traces of LLM calls, retrieval steps, tool executions, timing, inputs, outputs, token usage and metadata, which gives your monitoring function data to review. garak (Apache-2.0) probes language models for failures such as prompt injection and data leakage and writes a JSONL report and a hit log. DFIR-IRIS (LGPL-3.0-or-later) can hold incident cases with an audit trail and generate reports. Confirm each tool's fit with your AI systems, check what it captures against your monitoring policy, and budget for the staff who review the output. Commercial monitoring products may still be the better choice where your organization already depends on them.

When should we bring in an outside implementation partner for GOVERN 1.5?

Bring in a partner when you need working monitoring and incident tooling on your own infrastructure and do not have the staff to build it alongside daily work. DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your cloud accounts, data centers or air-gapped networks, and prepares evidence for compliance assessments. It does not issue audit reports or certifications, and it does not set your risk tolerance or make the accountability decisions. Engagements are normally fixed-scope, with deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer. Ask for a written scope that names who approves each policy.

How DeployOpen scopes and prepares GOVERN 1.5 evidence

DeployOpen helps organizations plan the monitoring, review cadence, incident response and feedback mechanisms that GOVERN 1.5 describes, and builds them inside the client's own infrastructure. The organization owns its AI risk management process and its decisions, and DeployOpen prepares the systems and records that show how the process runs.

How we scope

Engagements are fixed-scope, with the GOVERN 1.5 deliverables and boundaries agreed in writing before work starts.
The written scope names which AI systems, monitoring sources and review bodies are covered, and which are excluded.
Scoping confirms who owns monitoring, incident response and feedback roles, so responsibilities are recorded before tooling is set up.

How we build

We configure self-hosted tracing, such as Langfuse, to record model operations so performance and trustworthiness signals can be reviewed.
We set up garak-based test runs for language models, keeping JSONL reports and hit logs as repeatable evidence of periodic testing.
We configure case management in DFIR-IRIS so AI incidents are recorded with timelines, tasks and a full audit trail.

How we hand over

Handover includes written monitoring and review procedures, the review frequency and the named roles that perform each review.
We document incident response runbooks, including after-action review steps and the document retention rules for test and transparency records.
Clients receive the configured systems, the evidence exports and an operating guide their own staff can run without us.

How we support

Ongoing support, upgrades and continuous testing are available as retainers, and we can align them to the agreed review cycle.
Retainers can cover periodic re-testing and updates when the NIST AI RMF or its Playbook is revised, once the revision is published.
Assessment of the program remains with the organization and its independent assessors, since DeployOpen prepares evidence but does not issue reports or certificates.