A.6.2.6 AI system operation and monitoring: evidence

What A.6.2.6 asks for once an AI system is live

A.6.2.6 asks the organization to define and document what an AI system needs to keep running responsibly once it is live. When an auditor or customer asks how a production model is run, the answer should be a written set of operating needs that covers, at a minimum, system and performance monitoring, repairs, updates and support. The control does not ask for a particular tool. It asks for a defined position on each of these needs, who holds it, and how the organization knows it is being met. Many teams can show dashboards but cannot show that anyone is accountable for acting on them, which is the gap an assessor will probe first.

The difficulty is usually ownership rather than technology. Data science teams build and tune the model, operations teams run the infrastructure, and nobody is clearly responsible for the behavior of the model in between. Monitoring often exists, with latency and error graphs in one tool and output quality samples in a spreadsheet, but no threshold is linked to a decision such as rollback, retraining, human review or suspension. Decommissioning is another common gap, since retired models frequently keep serving traffic or keep their data and credentials. Confirm, for each system, who owns each operating need, what triggers action, and what happens when the trigger fires.

A.6.2.6 is easy to confuse with two neighboring requirements. A.6.2.5 covers the deployment of a system: the plan, release approval and rollback arrangements confirmed before go-live. A.6.2.6 begins after release and governs the ongoing run state. Clause 9.1 of ISO/IEC 42001 is different again, because it concerns monitoring of the AI management system itself, such as whether controls are working and objectives are being met. A.6.2.6 concerns each individual AI system. Your documented operating needs should show that the system-level monitoring feeds the management-level review, so an auditor can trace a drifting model to a governance decision.

Which AI systems and operating dependencies fall under A.6.2.6

A.6.2.6 applies to each AI system the organization builds, buys or runs, so the first job is deciding which systems and dependencies are in scope. The choices below change how much monitoring, repair and decommissioning work the control implies, and each one must be reflected in the statement of applicability under clause 6.1.3.

In-house versus pre-trained models

A model the organization trains or fine-tunes owns its full operating record, from retraining triggers to version history. A pre-trained or third-party model still falls under A.6.2.6 once it runs in the organization's services, but the provider may hold the model weights, update cadence and some monitoring data. Confirm in writing which operating tasks the provider performs, which the organization performs, and how model version changes reach the organization.

Customer-facing versus internal systems

Systems that answer customers, make decisions about individuals or affect payments carry higher operational stakes, so their performance thresholds, alert routing and rollback paths need to be defined more tightly. Internal assistants and analytics tools still need monitoring, but the acceptable error rate and response time can be set differently. Record the classification in the scope so an assessor can see why each system received its monitoring depth.

Retrieval and agent tool use

Systems that retrieve documents or call tools change behavior when their knowledge sources, connectors or permissions change, even if the model itself stays the same. Monitoring must cover retrieval quality, the tools an agent can invoke, and the access those tools hold. Each connected data source and tool should appear in the operating record as a dependency with an owner and a review point.

Teams and vendors that run infrastructure

The people who keep the model serving stack, logging pipeline and data stores running are often split between platform engineering, cloud accounts and outside managed service providers. A.6.2.6 expects the organization to define who handles repairs, updates and support for each component. Map each team and vendor to a component, record the support hours and escalation path, and confirm which contracts cover the work.

Statement of applicability justification

Clause 6.1.3 requires the organization to justify including or excluding each Annex A control. A.6.2.6 is usually included whenever an AI system is in production, so an exclusion needs a documented reason tied to the system's actual status. Before writing the justification, list every AI system in scope, note which are live, and state which Annex A control applies to each one.

How an assessor tests operation and monitoring

This table lists each test step an assessor is likely to take for A.6.2.6, the evidence requested for it and the findings that most often come up. Read it as a checklist: if the evidence for a row does not exist or is not current, the control is unlikely to pass on that point.

Test step	What the assessor checks	Evidence requested	Common finding
Review operating requirements	The documented definition of what the system needs to run in production, covering monitoring, repairs, updates and support	Approved operations runbook or operating requirements record, with version history and owner sign-off	Requirements exist only for the model at launch and were never updated for the current system
Sample production systems	Whether the documented requirements match what runs in production for a selected sample of AI systems	Monitoring configuration exports, dashboards and alert rule lists for each sampled system	Monitoring defined on paper but not operating, with dashboards empty or alerts silenced
Check thresholds and alert ownership	Whether performance and system thresholds exist, trigger alerts and name an accountable owner who responds	Threshold register, alert rule definitions, on-call or escalation roster, sample alert tickets	Alerts fire to a shared mailbox with no named owner or response record
Check drift detection criteria	Whether defined criteria detect changes in input data or output behavior and prompt a review	Documented drift criteria, drift reports, and records of the reviews they triggered	No drift criteria are defined, so no one can show when drift would be noticed
Review change and update records	Whether repairs, model or prompt updates, and version changes follow a recorded process with testing before release	Change tickets, release notes, rollback records and links to the verification results used for each update	Updates applied directly in production with no linked testing or approval record
Review support arrangements	Whether a support path exists for the AI system, including arrangements with any third-party model provider	Support contracts or service terms, escalation contacts, and provider notification records	No support path for a third-party model, so the organization cannot obtain fixes or change notices
Review decommissioning records	Whether retirement of an AI system is planned, approved and carried out with data and access handled	Decommissioning plan, approval record, data disposal evidence and access removal confirmation	Systems are switched off without a record, leaving live endpoints or stored data behind
Confirm findings feed back	Whether monitoring results lead to changes in the AI system, its requirements or its risk assessment	Post-incident reviews, corrective action records and updated documentation linked to monitoring findings	Findings are logged but never reviewed, so the same issues recur across reporting periods

How to build the operating plan for each AI system

This sequence turns A.6.2.6 into a documented operating plan for each production AI system, so that monitoring, repair, updates, drift checks and retirement have named owners and written criteria before an assessor asks for them. Each step closes with a record that can be shown as evidence.

Inventory every production AI system
Start with a complete register of AI systems running in production, including systems built in-house, third-party models called through an API, and AI features embedded in purchased software. Record the business owner, the technical owner, the hosting location, the users affected and the data each system touches. The step is complete when the register is reviewed by the owners and every system has a named accountable person. Without this list, the operating plan has gaps that an assessor will find first.
Define operating needs for each system
For each system in the register, document what it needs to run responsibly: the performance and behavior metrics to track, the logs that must be kept, the thresholds that count as a deviation, and who receives each alert. Tie thresholds to the requirements and risk assessment already written for the system, so the limits trace back to documented intent. The output is one operating needs record per system, approved by the technical and business owners.
Set up monitoring and alerting
Instrument each system so the metrics and logs defined in the previous step are actually collected, then configure alert rules that route deviations to the named owner. Open-source options such as Prometheus for metrics, Grafana Alerting for rules and notifications, and Langfuse for tracing LLM calls can support this, but the organization must confirm that each tool covers the data sources in its own environment. The step closes with a test alert that reaches the owner and a dashboard that shows the agreed metrics.
Define repair, update and support paths
Document how each system is repaired, updated and supported, including the escalation route, the people authorized to change production, and the rollback method. Pay particular attention to third-party models and APIs: record the vendor's change notification channel, the version currently in use, the support terms in the contract, and what the organization does when the vendor changes or withdraws a model. The output is a runbook per system and a list of vendor dependencies with their contacts and notice arrangements.
Set drift checks and review cadence
Define how the organization detects that a system's behavior or input data has moved away from what was validated, such as comparing live outputs with reference results or tracking shifts in input distributions. Set a review cadence for each system and state what triggers an unscheduled review, such as a threshold breach or a material change to the data or model. Confirm the chosen cadence with the risk owner. The step is complete when drift checks have run once against baseline and the review calendar is in the governance tracker.
Define decommissioning criteria
Write the conditions under which a system is retired, such as sustained failure to meet thresholds, loss of vendor support, a legal or regulatory change, or a business decision to stop the use case. Document the steps for retirement: disabling access, revoking credentials, archiving logs and records for the period the organization's own retention rules require, and informing affected users. The output is a decommissioning procedure that is approved in the same way as the operating plan.
Approve, record and keep each plan current
Have the accountable owner sign off each operating plan and store it with the system's technical documentation and event log records. Re-run the register and plans whenever a system is added, materially changed or retired, and feed incidents and drift findings back into the thresholds and runbooks. The step is complete when every production system has a current, approved plan and a dated record of its last review, which is the evidence an assessor will sample.

What to monitor in production and how drift shows up

Monitoring for A.6.2.6 has to cover three things at once: whether the AI system is healthy, whether its outputs are still good, and what those outputs do downstream. System health is the familiar part: service availability, latency, error rates and resource use, which your existing observability stack should already capture. Output quality is where AI systems differ from ordinary software. A model can keep responding quickly and without errors while its answers become less accurate, less consistent or less appropriate for the people using them. For each system, the operating documentation should name which signals you watch in each of these three groups, who receives the alerts, and who decides whether a degraded result triggers a rollback, a retraining request or a restriction on use.

Drift is the main reason output quality declines without any visible fault. Data drift means the inputs arriving in production no longer resemble the data the system was validated against, for example because customer behavior, document formats or upstream data sources changed. Model behavior drift means the way the system responds changes, even when inputs look similar, for example after a vendor updates a hosted model or a prompt template is edited. Both should be watched on inputs and outputs, not only on the final business metric, because the business metric often lags. Monitoring also needs to reach downstream impact, such as complaints, overrides by human reviewers or decisions that were later reversed, since those show whether the system is causing harm that the technical metrics miss.

Before you commit to a monitoring design, confirm three things with your own system owners rather than relying on general guidance. First, the baseline each signal is compared against, which should come from the validation evidence produced under A.6.2.4. Second, the thresholds and review cadence, which must be set by your organization for each system and recorded in its operating documentation. Third, which third-party model or service changes you will be notified of, since a provider update can create drift that your own deployment never touched. The ISO/IEC 42001 text and its implementation guidance describe the expected outcome; the NIST crosswalk, contributed by Microsoft and posted by NIST without endorsement, maps AI RMF subcategories such as MEASURE 2.4 on production monitoring and MANAGE 4.1 on post-deployment monitoring plans to B.6.2.6, which is a useful cross-reference when you draft your monitoring plan.

Which evidence shows A.6.2.6 is operating

An assessor testing A.6.2.6 looks for a documented operating plan for each AI system and proof that the plan runs in practice. The cards below cover the artifacts that connect those two things, from the plan itself through monitoring output, response records and decommissioning. Keeping them current means generating and filing them on a schedule, not assembling them in the weeks before an audit.

Documented operating plan per system

A written operating plan for each AI system that states what monitoring runs, which metrics and thresholds matter, who owns repairs and updates, how support is reached and when the system is reviewed. Assessors start here because every other artifact should trace back to it. Check that each plan names a system owner, carries a version and date, and is reviewed whenever the system's purpose, model or data source changes. Plans for systems that were never formally approved for production are a common gap.

Monitoring dashboards and alert rule exports

Dashboards show live and historical behavior such as latency, error rates, token usage, output quality checks and input volume. Exported alert rules prove which conditions are watched and at what thresholds, independent of what someone remembers configuring. Tools such as Grafana Alerting can build rules from queries across several data sources, so export the rule definitions into version control or the evidence store on a schedule. Confirm each exported rule still points to the production system it describes.

Alert history and response records

The alert history shows which alerts fired, when they cleared and who acknowledged them. Response records show what the on-call person did, whether the issue was contained, and whether the AI system was rolled back, restricted or left running. Auditors often sample a few alerts and ask for the matching tickets, so link each alert to a response record. Silenced or suppressed alerts need a documented reason and an approver, or they look like hidden failures.

Repair and update tickets

Tickets for fixes, model or prompt changes, dependency patches and configuration updates show that operation is maintained rather than only watched. Each ticket should record the trigger (an alert, a review finding or a vendor notice), the change made, the test performed and the approver. Check that changes to a production AI system pass through the same release controls used under A.6.2.5, and that emergency changes are reviewed after the fact.

Support and vendor service terms

Where the organization relies on a cloud provider, model vendor or open-source support contract, keep the signed terms that define response expectations, service availability and the escalation path. Record the support contacts and the date each contract was last reviewed. Only record service levels that appear in the actual contract or in terms the organization has supplied, and confirm with the vendor what is covered. Missing or expired support terms leave repair obligations without a counterparty.

Drift review records and decommissioning records

Drift reviews compare recent inputs, outputs and performance against the baseline the system was approved on, and record whether retraining, recalibration or retirement is needed. Decommissioning records document when a system was withdrawn, how its data and logs were retained or deleted, and which users and downstream systems were told. Schedule drift reviews at a cadence set in the operating plan and keep the dated outcome of each review, including reviews that found no action was needed.

What meeting A.6.2.6 costs in money and effort

This table lists each cost component of running A.6.2.6 monitoring, who is paid for it, whether it is one-time or recurring, and what pushes it up or down. Use it to build a quote request, since no figures are supplied here and the real amounts depend on your systems and environment. The main takeaway is that open-source tooling removes license spend but not the engineering and review effort.

Cost component	One-time or recurring	Who is paid or who does the work	What drives it up or down
Monitoring tooling	Recurring for hosting and storage; one-time for setup	Your cloud or data center costs, plus any implementation partner for setup	Open-source tools such as Prometheus and Grafana carry no license fee, but hosting, retention and staff time still apply
Instrumentation and tuning engineering	One-time per system, then recurring for changes	Your platform engineers, or an implementation partner under a written scope	Number of AI systems, number of data sources, and how many production metrics you define before go-live
Model and output quality monitoring	Recurring	Your ML or product owners, with tooling such as Langfuse traces or garak test runs	Model change frequency, traffic volume and the number of quality signals you track per system
Drift review and alert handling	Recurring	Your operations and model owners, who review alerts and decide on action	Alert noise, the number of systems and how quickly you must respond to a threshold breach
Evidence upkeep	Recurring	Your compliance or governance lead, who keeps monitoring records, change logs and decommissioning records current	Number of systems, release cadence and how much of the record is generated automatically
Repairs, updates and support	Recurring	Your engineering and support staff, or a retainer with an implementation partner	Support hours you commit to, upgrade cadence of the platform and vendor models, and incident volume
External security testing or review	Periodic, often one-time per release cycle	An independent tester or reviewer, engaged under a fixed-scope agreement	Scope of AI systems covered, whether agents with tools are in scope, and how often you retest
Auditor sampling and assessment	Periodic, tied to the assessment cycle	The independent certification body or auditor, who samples evidence	Number of AI systems in scope, quality of records, and how easily evidence can be retrieved

Which open-source tools produce operating evidence

Prometheus collects metrics from instrumented jobs by pulling them over HTTP on a schedule, then stores each value as a time series identified by a metric name and labels. Its storage is local to the server, and PromQL lets engineers query that history for trends, thresholds and comparisons. For A.6.2.6, this is the most direct source of evidence that the operating conditions of an AI system, such as request volume, latency, error rates and resource use, are being observed over time.

Pull-based collection: Each AI service exposes a metrics endpoint, and the server scrapes it. The scrape configuration is a file an assessor can review to see exactly which systems are covered.
Labeled time series: Labels such as model version or deployment environment let you show that monitoring distinguishes between releases.
Local storage: Retention depends on the server configuration and disk capacity, so the retention period is a design decision you must document.
PromQL queries: Saved queries and dashboards show the thresholds the team watches and how it reviews them.
What it does not do alone: Prometheus stores numbers. It does not judge output quality, detect semantic drift in model answers or record who responded to an event.

Confirm the scrape targets, retention period and the list of metrics tied to each AI system before an assessment.

Where operation and monitoring usually falls short

Most A.6.2.6 gaps are not missing tools. They are missing decisions: who acts on a signal, what counts as drift, which outside model changes reach the system, how long evidence is kept and when the system is retired. Each card below names one gap and the step that closes it.

Monitoring without owners or response paths

Dashboards and alerts often exist, but nobody is named to act on them. For A.6.2.6, document for each AI system who receives each alert, the response time expected, the escalation route to the system owner and the condition that pauses or rolls back the system. Check that the on-call rota and the documented procedure name the same people. An assessor will ask who acted on the last alert and what was decided.

No drift criteria

A system can degrade for months without anyone defining what degradation means. Write down the metrics that signal drift in input data, output distribution and accuracy against the thresholds agreed in verification and validation under A.6.2.4. Set the review trigger and the action each threshold causes, such as retraining, human review or withdrawal. Confirm that the thresholds appear in the monitoring configuration, not only in a policy document.

Third-party model updates not tracked

When a model provider changes weights, a hosted endpoint or safety filters, the system behaves differently without any change on your side. Record each upstream model version in use, subscribe to the provider's change notices, and re-run the relevant tests after each change before accepting it into production. Ask suppliers which changes they notify in advance and how long older versions remain available.

Logs not retained for investigation

Operational logs are often rotated away within days, long before an incident is reported or a complaint is investigated. Define the fields each AI system must log, such as inputs, outputs, model version and user actions, where they are stored and for how long. Test that you can reconstruct one past decision from the retained records. Coordinate retention with the event log control under A.6.2.8 and with privacy obligations for personal data in those logs.

Repairs and updates without a support model

Patches, model refreshes and bug fixes are applied ad hoc, so nobody can say which version ran at a given time. Name the team that supports each system, the change procedure for updates, and the support hours and contacts for the business owner. Keep a dated record of every repair and update linked to its test results. This record is the evidence an assessor uses to confirm operation is controlled.

No decommissioning criteria

Retired AI systems often keep running in the background, still processing data or serving stale outputs. Set criteria for retirement before launch, such as sustained drift past threshold, loss of vendor support, a changed intended purpose or a regulatory change. Document the steps to stop the service, remove access, archive or delete data according to retention rules, and notify affected users. Confirm the criteria were reviewed at least once in operation.

Questions about A.6.2.6 cost, effort and outside help

How much effort does AI system monitoring take to run?

The effort depends mostly on how many AI systems are in production, how many data sources they produce signals through, and how much of the monitoring is automated. No researched figure is available for typical hours, so ask for an estimate tied to your own inventory. Ongoing work usually includes reviewing alerts, triaging performance and drift signals, approving changes to the monitoring setup, and keeping the decommissioning and support records current. Each recurring task should have a named owner and a frequency written into the operating procedure, because an assessor will test whether the reviews happened, not whether the dashboards exist.

Is open-source monitoring tooling enough for A.6.2.6?

Open-source tools can produce much of the monitoring evidence, but they do not decide what must be monitored. Prometheus collects metrics and Grafana Alerting builds alert rules across data sources, and Langfuse traces LLM calls and tool executions. garak can probe language models for prompt injection and data leakage. The control also requires you to define repairs, updates, support and decommissioning, and those are governance and process gaps that no tool closes. Confirm with each vendor that a given capability fits your architecture before you rely on it.

How long does it take to set up AI system monitoring?

No researched duration is available, so treat any timeline as something to request in writing. Setup time depends on how many systems are in scope, whether instrumentation already exists, how many data sources must be connected, and how many owners must approve alert thresholds and escalation paths. A fixed-scope engagement is normally agreed on a written scope with deliverables and a timeline before work starts. Ask for the timeline to be tied to specific systems, environments and deliverables so you can check progress against it.

What does our team need to provide for monitoring work?

Your team owns the controls, so it must supply decisions and access. Expect to provide an inventory of AI systems, named business and technical owners, access to production telemetry and cloud accounts, and approval authority for monitoring thresholds, rollback criteria and retirement decisions. Security and privacy staff should agree on what logs may contain personal data before collection begins. Partners can build and configure the pipelines, but the thresholds and sign-offs must come from people accountable for each system.

Can an outside partner set up A.6.2.6 monitoring for us?

Yes, an implementation partner can plan, configure and document the monitoring, alerting, drift checks and operating procedures inside your own infrastructure. DeployOpen implements self-hosted open-source platforms in client cloud accounts, data centers or air-gapped networks, and most engagements are fixed-scope with the scope agreed in writing. The partner prepares systems and evidence. What stays with your organization is the control itself: deciding which systems are in scope, approving the criteria, and running the reviews after handover. Ask any partner to show which decisions remain with your staff.

Can the same firm that builds our monitoring also audit it?

No. An implementation partner that designs and configures your monitoring cannot also issue the assessment or certification. Only an independent certification body, accredited where accreditation applies, issues an ISO/IEC 42001 certificate, and assessors must remain independent of the work they test. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports, attestations or certifications. Check the certification body's independence rules before you sign any engagement with it.

What recurring costs should we plan for after monitoring is live?

Recurring costs usually fall into four groups: infrastructure to run and store the telemetry, staff time for alert review and tuning, updates to the monitoring stack, and periodic testing or retesting of AI systems. One-time costs cover design, initial configuration, documentation and handover. Ask any quote to separate these items and state whether support or upgrades are included or offered as a retainer. Ongoing support, upgrades and continuous testing are available as retainers, so ask for those terms in writing.

What should a written quote for this work include?

A useful quote states the scope in terms of named AI systems, environments and data sources, along with the deliverables such as monitoring configurations, alert rules, drift checks, runbooks and decommissioning procedures. It should list what the client must provide, what is excluded, and how changes to scope are priced. Ask whether testing, training and handover are included, what support follows go-live, and how the engagement ends. DeployOpen does not publish a price list, so a quote follows from the written scope.

What DeployOpen prepares and what stays with you

DeployOpen plans and implements the self-hosted monitoring stack that supports A.6.2.6 inside your own cloud accounts, data centers or air-gapped network, and prepares the operating evidence under a fixed written scope agreed before work starts. In practice that means configuring metric collection, alert rules, trace capture, drift checks and the runbooks for repairs, updates and decommissioning, then documenting how each piece was set up and tested. The organization still owns the control itself. Your team decides which monitoring thresholds matter for each AI system, who receives alerts, what triggers a rollback or retraining, and when a system is retired. DeployOpen does not make those decisions on your behalf and does not issue audit reports, attestations or certificates. An independent auditor or certification body does that, and it is the only party whose conclusions count for certification.

Before scoping, confirm with your internal owners which decisions must be recorded as your own, and ask any implementation partner to state in writing which artifacts it will deliver and which it will not. A reasonable scope covers the monitoring architecture, the configuration of each tool, the test evidence showing that alerts fire as designed, and the documented operating procedures. It should exclude judgments about whether an AI system is acceptable for use, which belong to the accountable owner. Ask for named deliverables, acceptance criteria and a handover date so that the evidence trail remains usable after the engagement ends. Quotes follow from that written scope, so the clearer the scope, the more precise the quote.

Keep ongoing responsibility in view as well. Monitoring produces evidence only if someone reviews it, so a retainer for upgrades, tuning and continuous testing may be useful, but it does not transfer accountability for A.6.2.6 to the provider. Plan for internal ownership of the alert queue, periodic review of drift and performance reports, and an annual check that the documented operating definition still matches the production system. If you are unsure which parts of your environment a self-hosted tool can reach or how it integrates with existing logging, confirm that against the vendor documentation and your own architecture before committing.

How DeployOpen scopes and evidences A.6.2.6

DeployOpen works on A.6.2.6 by setting up the operating side of an AI system inside the client's own infrastructure: monitoring, alerting, drift checks and decommissioning records. We prepare the systems and evidence, while the organization keeps ownership of its controls and the independent auditor issues any report or certificate.

How we scope

Written scope lists each AI system in production, its owners, and the operating needs it must meet.
Scope defines monitoring coverage for system health, model performance and data inputs before any build starts.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins.

How we build

We deploy self-hosted monitoring, such as Prometheus metrics collection and Grafana alert rules, in the client's environment.
Trace and event capture is configured so inputs, outputs and tool calls can be reviewed during an investigation.
Security testing of AI systems, including garak probes, is run and its JSONL reports are kept as operating evidence.

How we hand over

Runbooks document repairs, updates, support contacts and escalation paths for each monitored AI system.
Decommissioning and drift review procedures are written so the organization can run them without DeployOpen.
Evidence packs map each monitoring control to its configuration, alert history and review records for the auditor.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can cover alert tuning, monitoring reviews and re-testing when a model or its data changes.
Ask for the written support scope and response terms in your quote, since these are agreed per engagement.