What MAP 3.4 asks for in operator proficiency and why it is hard to evidence
MAP 3.4 asks an organization to define, assess and document the processes that make operators and practitioners proficient with an AI system's performance and trustworthiness, including the relevant technical standards and certifications. If an auditor, customer questionnaire or internal review asks how the people running your AI system are qualified, this outcome is the answer you need to produce. NIST AI 100-1 is voluntary, and there is no AI RMF certification, so no external body will issue a badge for MAP 3.4. The evidence has to come from your own documented processes, and an independent assessor or customer will judge whether those processes are real. The risk behind this subcategory is ordinary but serious: a person who misreads a model's output, trusts a score beyond its stated limits, or uses the system for a purpose it was never tested for. Teams struggle to evidence this because proficiency is spread across three places that rarely connect: training courses, operating documentation and hands-on practice on the job. Most of it is never written down. A reviewer will ask for the skills matrix, the known-limitations document, the training records and the certification criteria for each context of use, and many teams can produce only one of them.
The NIST AI RMF Playbook suggests actions for MAP 3.4, including identifying skill requirements, writing operational documentation on known risks and limitations, developing training materials, defining certification procedures for each context of use, and including operators in prototyping and testing. It is neither a checklist nor a set of ordered steps, so your organization decides which suggestions fit its system and risk tolerance and must be able to explain those choices. Start by confirming which operator roles exist, what each one is permitted to do, and which of your records would show that a named person met the requirement before taking that role. Those records are the evidence the rest of this page depends on.
Which operators, practitioners and systems fall in scope for proficiency
MAP 3.4 applies wherever people operate, build, oversee or rely on an AI system. The decisions below determine who needs defined proficiency, how deep that proficiency must go and how much training and testing the organization must document. Each card names one scoping decision and the work it adds or removes.
Roles tagged to MAP 3.4
The Playbook tags MAP 3.4 to six AI actor categories: AI Design, AI Development, Human Factors, End-Users, Domain Experts, and Operation and Monitoring. Start by mapping named people or teams to each category for your system, because a role left off the list usually gets no training record. Check that each role has written skill requirements and that the assessment covers the tasks AI 100-1 Appendix A assigns to that category. A narrow system may need only two or three roles; a system with many user groups multiplies the training and testing work.
Contexts of use that need proficiency
A proficiency requirement attaches to a context of use, not only to a system. An operator who reviews low-stakes drafts needs different knowledge from one who acts on model output in a credit, clinical or safety decision. List each context where a person's judgment changes the outcome, then define the minimum proficiency for each. Expect the testing burden to grow with the consequence of an error, and document which contexts are excluded from the scope of the system.
Features that change operator knowledge
Some features require more operator knowledge than others: automated actions, confidence thresholds, retrieval from internal data, model updates and fallback modes. When a feature is added, changed or switched off, the knowledge an operator needs also changes. Keep a feature-level list that states what each feature does, its known limitations and which operator actions it affects. Retraining is usually needed when a feature changes behavior, not only when the vendor issues a new version.
Vendor and third-party tool operators
Staff who use a vendor's AI tool still need proficiency for your context, even when the vendor supplies its own documentation. Identify which vendor tools touch your data or decisions, then confirm which training the vendor provides and which you must add. Ask the vendor for the known limitations and configuration options that affect your operators. Record gaps as open items, because the organization owns its MAP 3.4 process even when a third party builds the model.
Generative AI actions from AI 600-1
Generative AI deployments add MAP 3.4 actions from NIST AI 600-1, the Generative AI Profile. These include MP-3.4-001 to MP-3.4-006, which cover adapting training to digital content transparency, developing certification programs that test proficiency in managing generative AI risks, and keeping human proficiency tests separate from tests of AI capabilities. The profile ties these actions to four risks: Dangerous, Violent, or Hateful Content; Harmful Bias and Homogenization; Human-AI Configuration; and Information Integrity. If you run generative AI, plan a separate proficiency test set and budget time for the additional content and certification work.
Why proficiency needs a documented process, not a one-time course
MAP 3.4 asks for a process, not a training event. The outcome is that processes for operator and practitioner proficiency with AI system performance and trustworthiness are defined, assessed and documented. In practice that means four linked steps: defining the skills and proficiency each role needs, delivering training against those requirements, testing whether people actually meet them, and recording the results so they are revisited when the system, its data or its context of use changes. A course that a team completed once in a previous year does not show any of these steps, and an assessor reviewing MAP 3.4 will ask where the definitions, test records and update triggers live.
The NIST AI RMF Playbook lists suggested actions that fit together as one cycle. Identifying skills and proficiency requirements sets the target. Writing operational documentation on known risks and limitations gives operators the content they must understand. Developing training materials turns that content into teaching. Defining certification procedures for operating systems within defined contexts of use sets the pass criteria, and including operators in prototyping and testing feeds real operator experience back into the requirements. Action owners usually span AI design, development, human factors, end-users, domain experts and operations, so the document should name who approves each requirement and who signs off a test result.
These certification procedures are organizational definitions, not external credentials. The organization decides what a qualified operator looks like for a given system and context, and records how that is tested. An external certificate can be part of the program if the organization chooses one, but the Playbook does not require it, and NIST does not certify or accredit AI RMF practice. Before writing the procedure, confirm with your own risk and HR owners which roles need a documented pass decision, how often retesting happens, and which events (a model update, a new user group, a change in decision authority) force a retest and a record update.
How an assessor tests MAP 3.4 and what they will ask to see
This table compares what an assessor typically samples, asks and reviews for each part of MAP 3.4, along with the evidence they will request and the gaps they most often record. Read it as a set of sampling and interview steps rather than a pass or fail certificate, because NIST's AI RMF has no certification and only an independent assessor or the organization's own reviewers test these outcomes. Each row is a question you can prepare for before anyone asks it.
| Area | Test approach and evidence requested | Common findings |
|---|---|---|
| Proficiency requirements per role | Interview the role owners, then sample the documented skill and proficiency requirements for operators, reviewers and practitioners. Ask for the role list and the link between each role and the AI systems it touches. | Requirements exist for job titles but not for AI-specific tasks, or they were written once and never reviewed after the system changed. |
| Operational documentation of known risks and limitations | Pick two or three AI systems and check whether operators have written guidance on known failure modes, input limits and situations where output should not be used. Ask who approved the document and when it was last updated. | Documentation describes intended use only, with no limitations section, or operators say they have never seen it. |
| Training materials and completion records | Review the course content against the proficiency requirements, then sample completion records for current operators. Confirm that records cover people who joined after the initial rollout. | Materials exist but do not match the system's current version, or completion is recorded for only part of the operator group. |
| Internal certification procedure per context of use | Ask for the written procedure that decides when an operator is cleared for a given context, who signs off and how clearance is withdrawn. Test one case end to end with the approver. | Clearance is informal, such as manager confirmation in email, with no defined context of use or revocation step. |
| Operator participation in prototyping and testing | Ask for test plans, meeting records or sign-off sheets that show operators or end users took part before deployment. Confirm their findings were logged and acted on. | Operators were invited late, after design decisions were made, or their feedback is not traceable to a change. |
| Separation of human proficiency tests from AI capability tests | For generative AI systems, review how human proficiency assessments are designed and stored apart from model evaluations. Confirm the two result sets are not mixed or used to score each other. | Human tests and model benchmarks share one scorecard, or the team has not considered the distinction at all. |
| Ownership and review cadence | Ask who owns MAP 3.4 across the program, how often the requirements, materials and procedures are reviewed and what triggers an early review after a system change. | No named owner, or reviews happen only at initial launch and never after model or workflow updates. |
How to implement MAP 3.4 in six steps and what each step produces
This sequence moves from knowing who operates the system to proving those operators are qualified, documented, trained and re-checked as the system changes. Each step leaves an artifact an assessor can inspect.
Map operator roles and contexts of use
Start from the context established under MAP 1 and the targeted application scope set under MAP 3.3. List every operator and practitioner role that touches the system, including builders, reviewers, support staff and domain experts, and tie each role to the contexts in which it works. The owning team is usually the product owner with input from risk and HR. The step is done when a role register names each role, its tasks and the contexts of use it covers. Ask the business to confirm that no operating context is missing.
Define skill and proficiency requirements per role
For each role in the register, write the knowledge and skills needed to operate, interpret and challenge the system's outputs, and state what proficiency looks like in practice. Separate general AI literacy from task-specific competence. Have the accountable manager and a subject matter expert approve the requirements. The output is a role-based proficiency matrix that an assessor can compare against training records and assessment results.
Write operational documentation of known risks and limitations
Document what the system does poorly, where its outputs are unreliable, which inputs it should not receive and which failure modes operators must watch for. Build this from the MAP 2 categorization, testing results and any known limitations raised by vendors. Owners of the system and the model or platform team write it, and operators review it. The step closes when each document is version-controlled, dated and linked to the role it serves, so an assessor can see what operators were told and when.
Build training materials tied to each role
Create training that teaches the requirements in the matrix and the limitations in the documentation. Cover how to recognize errors, when to escalate and how to handle content that may be harmful or misleading. Learning platforms such as Moodle can record completion, and wikis such as XWiki can keep the materials under version control. Delivery should be recorded per learner so that each training record maps to a role and a version of the material.
Define the internal certification procedure and criteria
Set out how an operator demonstrates proficiency before gaining access to a given context of use. Define the assessment method, the pass criteria, who signs off, how often access is reviewed and how a lapsed or revoked certification is handled. The NIST Playbook suggests defining certification procedures for operating systems within defined contexts of use, and the Generative AI Profile adds that proficiency tests should be kept separate from tests of AI capability. The output is a written procedure with a signed-off criteria table and a register of certified operators per context.
Involve operators in prototyping and testing
Bring operators into prototype reviews and test runs before deployment so that their observations shape the requirements, documentation and training. Record their findings, the changes made in response and any issues they raise about usability or trustworthiness. Keep the test records and the feedback log together. An assessor should be able to trace an operator observation to a change in the system, the documentation or the training.
Set re-assessment triggers for system and context changes
Define the events that require operators to be re-assessed or re-trained, such as a model update, a new data source, a change in context of use or a significant incident. Assign an owner to monitor these triggers and record each decision to re-assess or not. Confirm the triggers with your own change management process. The evidence is a trigger list, a change log and dated re-assessment records showing that proficiency was checked after the system changed.
Which evidence passes MAP 3.4 and how to keep it current between audits
MAP 3.4 is tested by looking at whether the documents describe the same people, risks and procedures as the training and testing records. The cards below cover six artifacts an assessor usually asks for, how to keep each one dated and owned, and where documents and records tend to drift apart.
Proficiency requirements matrix by role
A matrix that lists each operator and practitioner role, the skills and proficiency level it needs, and the AI system contexts it covers. It is the anchor for the other artifacts, because the NIST AI RMF Playbook suggests identifying these requirements first. Assessors check that every role named in the operational documentation and training plan appears here, and that the required levels match what the training is meant to produce. Give each row an owner and a review date, and revisit it when a role changes or a new system enters use.
Operational documentation with dated versions
Operating guides that describe known risks, limitations and correct use of each AI system, with a version number and an effective date on every revision. Assessors compare the stated limitations with the risks recorded in MAP 2 and MAP 5 and with the failure modes shown in test results. Keep a change log that states what changed and why, and name a document owner who must approve each release. A guide that describes a model version no longer in production is a common finding.
Training materials with change history
Course content, slide decks, job aids and labs used to build operator skill. Under MAP 3.4 these should cover the known risks and limitations listed in the operational documentation, and for generative AI, the content-transparency topics that NIST AI 600-1 suggests. Store each release in version control so an assessor can see what a trainee was taught on a given date. Review the materials whenever the operational documentation or the system's risk profile changes, and record the review even when no edits are needed.
Learner completion and assessment records
Records that show who completed which module, when, and with what result. Completion alone does not prove proficiency, so pair attendance with graded assessments that test the role's required skills. Moodle can mark activities complete on viewing, on grading or manually, so the completion condition for each course should be set to match the proficiency claim you make. Assessors check that the people operating a system appear in these records at the required level, and that access to the system was withdrawn or retested when a learner's role or proficiency lapsed.
Internal certification procedure and decisions per context of use
The procedure that defines how an operator earns permission to run a system in a given context, who approves it, and how permission is withdrawn. The NIST AI RMF Playbook suggests defining certification procedures for operating systems within defined contexts of use, and AI 600-1 suggests testing proficiency in managing generative AI risks. Record each approval decision with the context, the evidence reviewed, the approver and the expiry or re-certification trigger. This is an internal certification. It is not an external certificate, and the procedure should say so to avoid confusion with third-party credentials.
Records of operator participation in prototyping and testing
Evidence that the people who run a system took part in prototyping and testing before deployment, as the Playbook suggests. Keep test plans that name operator participants, session notes showing what they found, and the defects or usability issues they raised and how each was resolved. Assessors check that the same operators appear in the proficiency matrix and training records, and that the testing feedback changed the operational documentation where it should have. Keep proficiency tests separate from tests of the AI system's capabilities, as AI 600-1 recommends for generative AI.
What meeting MAP 3.4 costs in money and effort
This table lists each cost component of building and keeping up MAP 3.4 proficiency processes, who is paid for it, whether it is one-time or recurring, and what drives it. There is no AI RMF certification fee, so every cost comes from the work of defining, documenting and testing proficiency, and the main takeaway is that most of the spend is recurring labor that grows with the number of operators and system changes.
| Cost component | One-time or recurring | What drives it and how to keep it down |
|---|---|---|
| Proficiency requirement analysis and role mapping | One-time, then reviewed | Number of operator roles and system contexts. Keep it down by mapping roles to the system's documented contexts of use first. |
| Operational documentation on known risks and limitations | One-time, then updated | Number of known failure modes and how often the system changes. Keep it down by writing it from existing test and incident records. |
| Training material development | One-time, then updated | Number of roles, languages and delivery formats. Keep it down by reusing one course across roles and hosting it on an existing learning platform. |
| Learning platform and tooling | Recurring licence or hosting | Platform choice and user count. Self-hosted open-source options can remove licence spend, but hosting, patching and administration remain recurring staff effort. |
| Internal assessment time | Recurring | How many proficiency tests and certification procedures the organization defines. Keep it down by automating completion records and grade tracking. |
| Operator time in prototyping and testing | One-time per release, recurring with change | Number of operators involved and the frequency of releases. Keep it down by limiting participation to the roles that need it. |
| Re-training after system or policy changes | Recurring | Rate of model, data, interface or policy changes. Keep it down by tying retraining triggers to the change management process. |
| External review of the proficiency design (if purchased) | One-time or periodic, if bought | Scope of the review and the reviewer's fees, which are set by the written scope. Ask whether the reviewer is independent of the implementation work. |
How Moodle, XWiki and BookStack can hold the MAP 3.4 evidence
MAP 3.4 asks for defined, assessed and documented proficiency processes. Three self-hosted open-source tools can each hold a different part of that record: proficiency testing, versioned procedures and controlled approval. None of them certifies anyone, so the organization still decides what counts as proficient and who signs off.
Proficiency testing in Moodle
Moodle's activity completion marks an activity complete for each learner when they view it, when they receive a grade from it, or when they tick it manually, depending on the conditions set for the activity. For MAP 3.4, the completion condition must match the proficiency test. A view-based rule would record that an operator opened a training module, not that they can operate a system within its context of use. Confirm the condition, the grade threshold and how completion records are exported before relying on them. Moodle is licensed under GPL-3.0-or-later, so confirm that this license fits your environment and distribution model.
Versioned procedures in XWiki
XWiki keeps every change to a page under version control, recording what changed, when and by whom. Reviewers can compare any two versions and roll back to an earlier one. This suits the operational documentation that MAP 3.4 expects, including known risks and limitations, proficiency requirements and certification procedures for operating a system in a defined context. Auditors and internal reviewers can see who edited a procedure and what it said at the time an operator was assessed. Confirm the retention behavior and backup approach in your own deployment. XWiki is licensed under LGPL-2.1-or-later, so confirm that this license fits your environment.
Controlled approval in BookStack
BookStack assigns user abilities through roles, and permissions can be overridden per role at the shelf, book, chapter or page level. This lets you separate who can read training materials, who can draft operational documentation and who can approve it for use. For MAP 3.4, that separation shows that the people who write proficiency criteria are not the same people who certify operators against them. Map the roles to your approval chain and test the overrides before you rely on them. BookStack is licensed under MIT, so confirm that this license fits your environment.
Who does what: your team, an implementation partner and the auditor
Your organization owns the proficiency process. The NIST AI RMF is voluntary and has no certification, so any certification for operators and practitioners is one your organization defines, runs and accepts internally. Your team decides which skills each operator role needs, what counts as proficient for each context of use, and who may approve an operator to run a system. Those decisions carry accountability that cannot be handed to a vendor or consultant.
- Proficiency requirements: Define the skills, knowledge and experience required for each operator and practitioner role, tied to the contexts in which the system is used.
- Internal certification decisions: Approve the procedures that grant, renew and withdraw operating authority, including the tests and the people who score them.
- Operational documentation: Own the known risks and limitations documents that operators rely on, and confirm they match how the system actually behaves in production.
- Sign-off and records: Sign the documentation, retain training and approval records, and decide when proficiency must be reassessed after a change to the system.
Before engaging any partner, confirm that your internal owners can name every requirement they have approved and produce the signed record behind each one.
An implementation partner prepares the systems and evidence that your team's process depends on. DeployOpen plans, configures and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, so the training records and documentation remain under your control. DeployOpen does not set your proficiency thresholds, approve operators or certify anyone. Its role is to make the process buildable, repeatable and traceable.
- Training design: Draft course structures and training materials for your approved requirements, for review by your domain and human factors owners.
- Documentation templates: Prepare templates for operational documentation and certification procedures, so each version and approval is recorded in a consistent way.
- Platform setup and evidence collection: Configure learning, wiki and access tools on your infrastructure. Moodle, for example, marks an activity complete based on conditions set for that activity, so a completion mark is only meaningful if the conditions match your proficiency criteria. XWiki keeps version history and BookStack ties permissions to roles, which helps show who changed a document and who could approve it. Confirm each capability in your own environment before relying on it.
- Prototyping sessions: If scoped in writing, run sessions that bring operators into prototyping and testing, and record what they observed.
Request a written scope that names each deliverable, the platforms involved and who on your side receives the evidence.
An independent auditor, assessor or certification body tests whether your proficiency process is defined, assessed and documented, and issues any report or attestation. DeployOpen does not issue reports, attestations or certificates, and an auditor cannot be the same party that designed the process. Keeping this line clear matters because an assessor's conclusion is only credible if the people who built the evidence did not also grade it.
- Record sampling: Select a sample of operators and check their training completions, approvals and reassessment dates against your documented requirements.
- Operator interviews: Ask operators to explain the known risks and limitations of the system and how they would respond to a failure, then compare their answers with the documentation.
- Process testing: Confirm the procedure is defined, that it was carried out as written, and that exceptions were recorded and reviewed.
- Scope confirmation: If your attestation names another framework, such as ISO/IEC 42001 or a customer questionnaire, confirm with the auditor which controls and evidence they will assess.
Agree the auditor's scope, sampling approach and evidence list with your team before fieldwork begins, and confirm in writing which party will issue any report.
Questions buyers ask about MAP 3.4 cost, effort and outside help
Is there a certification for MAP 3.4?
No. NIST does not certify, accredit or validate organizations against the AI RMF, and it describes no conformity assessment program for MAP 3.4 or any other subcategory. The framework is voluntary. Proficiency is shown through your own documented process: defined skill requirements, assessed operators, training records and testing evidence. If a customer or contract asks for a certificate, ask them which standard or program they mean, since the AI RMF itself offers none. An independent assessor can review your evidence, but that review is not an AI RMF certification.
How much effort does MAP 3.4 take?
Effort depends mostly on the scope you write down, not on the subcategory itself. The main drivers are the number of operator and practitioner roles, the number of contexts of use, and whether generative AI is in scope. Each role needs a skills and proficiency definition, operational documentation of known risks and limitations, and training material. Each context of use may need its own certification procedure. Generative AI adds the actions in NIST AI 600-1 for MAP 3.4, including keeping human proficiency tests separate from tests of AI capabilities. Ask for the effort split by role and context in any quote so you can see what drives it.
Can our team do MAP 3.4 without outside help?
Yes, if your team already has the skills. The work is mostly writing and testing: defining what proficiency means for each role, deciding how to assess it, documenting the results and keeping the records current. Someone on the team needs to write requirements that are specific enough to test, and to design assessments that measure operator behavior rather than attendance. Your AI, security and human factors staff should review the output. If those skills are thin, a partner can help draft the process, but your organization still owns the definitions and the decisions about who is qualified to operate the system.
What does MAP 3.4 require from our operators in time?
Operators need time for three activities. First, training on the system, its known risks and its limitations. Second, assessments that test whether they can operate and oversee it correctly. Third, participation in prototyping and testing, which the Playbook suggests so that operators can shape the process before deployment. Plan for this time as a recurring cost, not a one-off. Operators also need time for refresher training when the system, its contexts of use or its risks change. Confirm with your managers how these hours fit into existing workloads before the program starts.
When should we bring in an implementation partner for MAP 3.4?
Consider a partner when the number of roles and contexts of use is large, when several teams run the same system differently, or when the platform is self-hosted and needs setup before proficiency can be tested. Operators cannot practice on a system that is not yet running in a test environment. DeployOpen plans, implements and documents self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks, and prepares the systems and evidence that proficiency testing uses. It does not define your proficiency requirements for you or issue any certificate. Those decisions and any independent review remain with your organization and its assessor.
How long does a MAP 3.4 engagement take?
Duration depends on the written scope: the number of roles, the contexts of use, whether generative AI is included and the state of your platform and training materials. DeployOpen's engagements are fixed-scope, so scope, deliverables and timelines are agreed in writing before work starts. Ask any provider for a schedule tied to named deliverables, such as role definitions, assessment designs, training materials and test records, and ask which of those depend on your team's review or on platform setup. Note that the Playbook was last updated in August 2023 and that NIST plans to revise the AI RMF, so build in time to check the guidance you use.
How DeployOpen works on this
DeployOpen helps organizations prepare the systems, operator processes and evidence behind MAP 3.4 Operator and practitioner proficiency. The organization owns its proficiency requirements and decisions, and an independent assessor reaches any conclusion about them.
How we scope
- Agree in writing which AI systems, operator roles and proficiency records fall inside the MAP 3.4 scope.
- Confirm which AI actor categories from the Playbook apply, such as operators, end users and domain experts.
- Engagements are fixed-scope, so deliverables and timelines are settled before any work starts.
How we build
- Configure open-source learning and documentation platforms inside your own infrastructure to record training and proficiency checks.
- Set up versioned operational documentation of known risks and limitations for each operator role.
- Test with operators during prototyping and record results separately from tests of AI system capability.
How we hand over
- Deliver the proficiency requirements, training materials and certification procedure as documented artifacts your team owns.
- Provide an evidence map linking each record to the MAP 3.4 suggested actions you chose to adopt.
- Walk your operators and compliance leads through the records so they can maintain them after handover.
How we support
- Offer ongoing support and updates as a retainer when your proficiency requirements or AI systems change.
- Review records with your assessor's questions in mind, but DeployOpen does not issue audit reports or certifications.
- Ask your assessor and counsel to confirm which proficiency evidence they will accept before the build begins.
