The largest category in the AI RMF: TEVV documentation, human subject evaluations, deployment-like testing, production monitoring and evaluation of each trustworthiness characteristic.
Documenting the test sets, metrics and details of the tools used during test, evaluation, verification and validation (TEVV), using established documentation practices such as datasheets and model cards.
Making sure evaluations involving human subjects meet applicable requirements, including human subject protection, and are representative of the relevant population.
Measuring AI system performance or assurance criteria qualitatively or quantitatively and demonstrating them under conditions similar to the deployment setting, with the measures documented.
Monitoring the functionality and behavior of the AI system and its components in production, as identified in MAP: distribution drift, anomalies, alerts and human review of unreliable outputs.
Demonstrating that the AI system to be deployed is valid and reliable, and documenting the limits of its generalizability beyond the conditions under which it was developed.
Regularly evaluating the safety risks identified in MAP and demonstrating that the system is safe, stays within risk tolerance and can fail safely, with safety metrics on reliability, robustness, real-time monitoring and response times.
Evaluating and documenting AI system security and resilience as identified in MAP: security metrics, red-team exercises, countermeasures, third-party security audits and sharing of attack information.
Examining and documenting risks associated with transparency and accountability: audit logs and histories, oversight statistics, overrides, complaints, escalations and go or no-go decisions by accountable parties.
Explaining, validating and documenting the AI model and interpreting its output in context to inform responsible use and governance: explanation methods, model documentation and tests of explanation quality.
Examining and documenting the privacy risk of the AI system: handling of personally sensitive information, re-identification metrics, access controls for training and production data, and privacy-enhancing techniques.
Evaluating the fairness and bias issues identified in MAP and documenting results: harm types, affected groups, fairness metrics, sources of bias in training and TEVV data, and proxies for protected attributes.
Assessing and documenting the environmental impact and sustainability of AI model training and management: energy, water and emissions indicators, baselines and acceptable levels. Climate references are a target of the pending AI RMF revision.
Evaluating and documenting the effectiveness of the TEVV metrics and processes used in the MEASURE function: whether metrics sustain improvement, remain useful and acceptable, and actually identify risk.
Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.