A.4.4 Tooling resources: requirements and evidence

A.4.4 tooling resources: what the requirement asks for and why it is hard

A.4.4 asks the organization to document information about the tooling resources used for its AI system, and that record has to stay current as those tools change. When an audit or a customer questionnaire asks which frameworks, pre-trained models and evaluation tools sit behind a system, the answer should come from a maintained record, not from memory or a slide prepared for the last review. Without that record, an organization cannot show what it depends on, which versions are running in production, or which components carry license or known vulnerability exposure. Pre-trained models are a particular gap: they often enter a project through a notebook or a package install, outside any approved procurement path, and nobody outside the data science team knows they are there.

The work is hard for two reasons. First, tooling changes faster than documentation: a framework is upgraded, a model checkpoint is swapped, or an evaluation library is added during a sprint, and the record stays at the previous state unless someone is responsible for updating it at the moment of change. Second, ownership is split. Data science teams choose the models and libraries, platform engineering runs the build and deployment pipelines, and security teams look for vulnerabilities and license risk, yet none of them owns the consolidated tooling record by default. Before scoping the work, confirm who will own that record, which systems will feed it, and how changes are caught. The NIST AI Resource Center crosswalk maps Annex B guidance for this control to AI RMF inventory and monitoring actions, but confirm that mapping against your own control wording and your statement of applicability before relying on it.

Which tools, models and teams fall inside A.4.4

A.4.4 covers every tool that shapes how an AI system is built, evaluated, deployed or run. The scope question is less about whether a tool is “AI” and more about where it sits in the life cycle, who supplies it and whether it touches the production system. Use these five categories to decide what goes on the tooling record.

Model frameworks and libraries

Machine learning frameworks, inference libraries and the open-source packages a team imports into training or application code. These are the most common entries on a tooling record and the easiest to miss, because they arrive through package managers. Check the exact versions pinned in each repository and whether a framework is used only for experiments or also ships inside the deployed service. Record the owning team and the upgrade path, since version changes can alter model behavior.

Pre-trained models used for development

Base models, embeddings and fine-tuned checkpoints downloaded from a model hub or licensed from a vendor and then adapted for the system. Record the source, license, version or checkpoint identifier and whether the organization has changed its weights. The NIST crosswalk links pre-trained models used for development to ongoing monitoring during maintenance, so the record should show who watches for updates or deprecations. Confirm the license terms with counsel before you rely on them.

Evaluation and testing tools

Benchmark harnesses, red-teaming utilities, bias test suites and output-scoring tools used to measure whether a system meets its requirements. These tools often run only before release, which makes them easy to leave off the production inventory. Record the evaluation version and the datasets it runs against, because an assessor will ask whether the results you cite came from the tool you document. Third-party evaluation services should appear here with their contract reference.

MLOps and deployment tooling

Pipelines, experiment trackers, model registries, container build systems and infrastructure-as-code used to move a model from development into operation. These tools change how models are versioned and released, so they carry governance weight beyond convenience. Check whether the pipeline can alter a deployed model without a documented approval step, and whether the tool's own access and logs are retained. Treat the CI/CD runner and its credentials as part of the record.

Runtime tools in production

Guardrail filters, retrieval components, monitoring agents, vector stores and any library that executes while the AI system answers a request. These are the tools whose failure reaches users, so they need the most complete entries, including the version running in each environment. Dependency inventories such as an SBOM help show which components are present in the running service. Confirm whether each runtime component is supplied by a third party and how the organization receives its updates.

How an assessor tests the tooling record

This table sets out the checks an assessor typically runs against the A.4.4 tooling record, the evidence each check asks for, and the gaps that most often come up. Read each row as a test the organization should run on itself before the assessor arrives.

Test	Evidence requested	Common findings
Trace the tooling list to a sample of AI systems	For a sample of AI systems, the tools each one uses, matched against the tooling record	Frameworks or evaluation tools used in a pipeline but absent from the record
Check versions against deployed artifacts	Container images, lock files, dependency manifests or SBOM output from the running environment	Version numbers in the record that no longer match what is deployed
Check pre-trained models for recorded sources and update monitoring	Model source (provider, repository, licence), the version used and the record of who watches for updates	Pre-trained models listed by name only, with no source, version or update owner
Check that the record changed when tools were replaced	Change tickets or pull requests for tool swaps, with the matching record update and dates	A framework replaced in code while the record still names the old one
Check ownership of each entry	A named owner per tool or model, with a stated responsibility for upgrades and retirement	An embedded model or vendor SDK with no owner, so nobody reviews it when it changes
Check coverage of tools inside third-party services	Records of tools the organization does not host but relies on, such as hosted evaluation or model APIs	Externally hosted tools omitted because they are not installed on internal systems

How to build a tooling record that survives tool changes

This sequence turns A.4.4 into a living record: it defines which AI systems are in scope, lists the tools each one depends on, and records enough detail to check the register against what is actually running. The output is a per-system tooling register plus a change log that an auditor can trace over time.

Define the AI systems in scope
Start from the AI system inventory maintained under A.4.2 and confirm which systems the tooling record must cover, including systems in development, in production and retired but still holding artifacts. The accountable owner, usually the AI governance lead with the platform team, signs off the list. The output is a scoped system list with a named owner for each system, which becomes the index for every later step.
Inventory tools from build and runtime sources
For each system, collect the tooling from the places where it actually enters the build: source repositories, dependency lockfiles, container images and model registries. Include model frameworks, pre-trained models, evaluation harnesses and data processing libraries. Platform engineers can generate much of this with software bill of materials tools such as Trivy, which can identify OS packages and software dependencies in a target and produce them as an SBOM. The output is a raw tool list per system, with the source each entry came from.
Record purpose, version, license, source and owner
Turn each raw entry into a register row that states what the tool does for the system, its exact version or model identifier, its license, where it was obtained, and the team responsible for it. Pre-trained models need the same treatment as libraries, including the provider and the version pulled. Confirm license identifiers against the upstream project rather than the package name. The output is a per-system tooling register that a reviewer can read without asking the engineer who built it.
Set review triggers for tool changes
Write down the events that require the register to be updated before the change reaches production. Typical triggers include a dependency update that changes a major version, a model swap or fine-tuned checkpoint replacement, a new evaluation tool, a license change, and a vulnerability advisory that affects a listed component. The control owner approves the trigger list and the pipeline owner wires the checks into change management. The output is a documented trigger list and an approval record for it.
Log each change with its approver
When a trigger fires, the engineer updates the register row and adds an entry to a change log that states what changed, why, the previous and new versions, the risk review outcome, and who approved it. Dependency tracking platforms such as OWASP Dependency-Track can consume CycloneDX SBOMs, flag outdated or modified components and track license use by component, which helps surface changes that were never logged. The output is a dated change log tied to register entries.
Reconcile the register against deployed systems
On a schedule the organization sets and documents, compare the register with what is running. Compare SBOMs and image digests from the deployed environment against the register, and check model identifiers in serving endpoints against approved entries. Any mismatch becomes a finding that either updates the register with an approved change or triggers a rollback. Confirm the reconciliation interval with the control owner and the auditor before relying on it. The output is a reconciliation record showing discrepancies and their closure, which is the evidence that the record survives change.

Evidence that passes an A.4.4 review

An assessor testing A.4.4 Tooling resources looks for a tooling record that matches what actually runs, not a document written once for the audit. The cards below cover the artifacts that connect the record to real systems, and each one carries a trigger that keeps it current between audits.

Tooling register per AI system

A register lists every tool used to build or run each AI system, with its version, license, source repository or vendor, and a named owner. Assessors compare it against the system's deployment and development environments to see whether anything is missing. Keep it current by making a register entry a required field in the release checklist, so a new framework, library or evaluation utility cannot reach production without a row. Confirm that owners are people who can answer for the tool, not only teams that installed it.

Pre-trained model records

For each pre-trained model used in development or inference, record its provenance, the source it was obtained from, the version or checkpoint, the license terms, and monitoring notes. The NIST crosswalk links this kind of record to the monitoring of pre-trained models as part of regular maintenance, which is what an assessor will ask about. Update the record when a model is swapped, fine-tuned or retired, and review monitoring notes at each model release rather than once a year.

Dependency and SBOM exports

A software bill of materials exported for each AI system shows the libraries and packages actually present in the built artifact, which can be compared with the register to find gaps. Dependency-Track can ingest CycloneDX SBOMs and flag components with known vulnerabilities, outdated versions or license risk. Trivy can generate SBOMs from a container or filesystem. Regenerate the export on every build in the pipeline, and store each export with the build identifier so the record matches a specific release.

Tool change records

Change records show when a tool was added, upgraded or removed, who approved the change, and which AI systems it affected. An assessor uses them to check whether the register reflects history, not only the current state. Tie each change to the ticket or merge request that introduced it, and require a change record whenever a dependency manifest or model reference changes. This makes the trigger part of the normal change process rather than a periodic clean-up.

Evaluation tool version records

Evaluation tools and test harnesses shape the results used to judge an AI system, so the version used for each test run must be recorded alongside the result. If a benchmark suite or scoring utility changes between runs, earlier results may not be comparable. Store the tool name, version and configuration with each evaluation report, and check that the record is written by the evaluation job itself rather than typed in afterward. Revisit the record whenever the evaluation pipeline is updated.

Reconciliation review minutes

Periodic review minutes show that someone compared the tooling register with the SBOMs, model records and change log, and resolved the differences. Assessors look for named participants, the date, the discrepancies found and the actions assigned. Hold the reconciliation at a fixed trigger, such as each production release of an AI system or each significant tooling change, and record the outcome in the same register so the review trail stays linked to the artifacts it checked.

What meeting A.4.4 costs in money and effort

This table breaks the cost of keeping an A.4.4 tooling record into its components, showing who is paid, whether each cost is one-time or recurring, and what moves it up or down. Use it to compare line items in any written quote, since no supplied figures exist for these costs and every estimate should come from a scope you can inspect. The main takeaway is that open-source platforms usually remove license spend, but staff time for the register and its upkeep stays.

Cost component	One-time or recurring	What drives it and how to keep it down
Tooling or licenses for the inventory platform	Recurring if commercial, near zero license cost if open source	Commercial seats and modules scale with asset count. Self-hosted open-source inventory removes license fees, but hosting, storage and patching of the platform remain. Confirm each tool's license terms with its vendor.
Engineering time to build the first register	One-time	Driven by how many models, pre-trained components, evaluation tools and libraries are in use, and whether they are already in source control or package manifests. Ask a partner for effort estimates by tool category.
Engineering time for ongoing change tracking	Recurring	Driven by how often models, frameworks and evaluation tools are upgraded or swapped. Automating discovery from build pipelines and SBOM generation reduces manual updates, but someone must review the output.
Evidence upkeep and review meetings	Recurring	Driven by the review cycle set in your management system and the number of owners who must confirm entries. Fewer tools and clear ownership per entry keep meetings short.
External review of the record before audit	One-time per cycle, if used	Priced by the reviewer's time against the record's size and completeness. An internal dry run first reduces the findings an outside reviewer must chase.
Auditor sampling time	Recurring per audit cycle	Auditors choose samples and ask for version history and ownership for selected tools. A record that already shows dates, versions and owners reduces follow-up time. Confirm the audit plan and sampling approach with your certification body.

Open-source tools that can produce tooling evidence

No single tool produces a complete tooling record for an AI system. The tools below each cover a different layer, from hardware and endpoint software to the software components inside an AI application. Combine them, then confirm which parts of your AI stack, including model registries, each one actually reaches.

GLPI Assets for asset records

GLPI's Assets module tracks computers, software, network equipment and other managed items, entered by hand or pushed automatically by the GLPI Agent through native inventory built into GLPI since version 10. For A.4.4, it can hold the owner, location and assigned purpose of the hosts that run AI tooling, which gives the record an accountable entry point. It does not describe the frameworks, pre-trained models or evaluation tools inside an application, so confirm how far your inventory fields reach into those workloads.

Wazuh Syscollector for endpoint software

Wazuh's Syscollector module collects installed software, running processes, services and other system details from each endpoint and stores them in the Wazuh indexer for dashboard reports. This gives evidence of which packages and runtimes are actually present on machines that host AI tooling, not only what was planned. It only sees software on monitored endpoints, so AI workloads in managed cloud services or containers you do not run on those agents will need separate records.

Dependency-Track for SBOM tracking

OWASP Dependency-Track consumes and produces CycloneDX SBOMs and VEX, flags components with known vulnerabilities, outdated or modified versions, and license risk, and tracks license use by component with SPDX IDs. For A.4.4, it turns a list of libraries used to build an AI system into a maintained, reviewable record that changes as dependencies are upgraded. Model weights and datasets are not software packages in the usual sense, so check whether your model registry entries can be represented in the SBOM or must be recorded separately.

Trivy for dependency and SBOM output

Trivy identifies OS packages and software dependencies in a target, can generate them as a software bill of materials, and can scan existing SBOMs for known issues. In a build pipeline it can produce a fresh SBOM for each AI service image, which keeps the tooling record tied to the artifact actually deployed. It does not inventory pre-trained model files or evaluation tools that are not packaged as dependencies, so confirm coverage before treating its output as the complete record.

Who owns the tooling record and what can be outsourced

Your organization owns the tooling record and every decision that changes it. The record is an organizational control under ISO/IEC 42001, so an auditor will ask who maintains it, who approves changes and how the organization knows the list is complete. These answers must come from staff who run the AI systems, not from a vendor. Ownership usually sits with a named role in AI governance, supported by security and platform engineering for the technical inputs.

Register ownership: Name one accountable owner for the tooling register and a deputy. Record their responsibilities in the AI roles documentation so the ownership is traceable.
Change sign-off: Require an approval step before a new model framework, pre-trained model, evaluation tool or major version enters a production AI system, and record who approved it and why.
Statement of applicability decisions: Decide whether A.4.4 is included, and justify any tooling category you exclude. The justification belongs to your organization, not to the partner.
Periodic review: Set a review cadence and confirm that tools which were retired or replaced are removed from the active record.

Before any outside work starts, confirm that these owners and approval steps exist and are written down, because no partner or auditor can supply them after the fact.

Where A.4.4 records usually fall short

A.4.4 records fail less often because tools are missing and more often because the record is incomplete, stale or limited to the places that are easiest to inventory. The cards below describe the gaps an assessor is most likely to notice, what each one looks like in practice, and the specific change that closes it.

Pre-trained models without versions or sources

Models are often listed by name only, such as a model family recorded in a project wiki. An assessor will ask which exact version was used, where it was obtained, and under what license. For each pre-trained model, record the version or checkpoint identifier, the source repository or provider, the license, the date it was adopted, and the system it supports. Confirm that the record matches the artifact actually deployed.

Tooling captured once and never updated

Many teams document frameworks and libraries at project start and leave the record untouched through later releases. Auditors look for evidence that the record changed when tools changed. Tie the tooling record to the change process so that a framework upgrade, a new model or a swapped evaluation library triggers a documented update with a date and an owner.

Evaluation tools that run only in CI

Evaluation harnesses, benchmark suites and red-teaming scripts often run inside pipelines and never appear in the resource record. Yet they shape the claims made about the AI system. List each evaluation tool with its version, the metrics it produces, and the pipeline stage where it runs. Include the tool in the record even when no one interacts with it directly.

Third-party components with no named owner

A product that ships an AI component often embeds open-source libraries and model runtimes that someone must track for vulnerabilities and license changes. When no person or team is named for them, the record decays. Assign an owner to each third-party component, state how that owner receives upgrade and advisory notices, and check that the owner is still in the role at each review.

Inventories that miss notebooks and developer environments

Server inventories capture production hosts but often skip the notebooks, local development environments and experiment workspaces where models are actually built and tuned. These environments hold tooling and pre-trained weights that reach production. Extend discovery to developer endpoints and shared notebook platforms, and record them as tooling resources when they touch the AI system.

How much work A.4.4 takes and whether to do it in-house

How much effort does the first tooling register take?

Effort for the first register depends mainly on how many tools, repositories and runtime environments the AI system touches, and on whether those tools are already inventoried elsewhere. A single model served from one cluster with a few pinned frameworks is a small job. A platform with several pipelines, dozens of pre-trained models and evaluation tools spread across teams is a much larger one, because each entry needs a version, a source, an owner and a purpose. No effort figure is supplied for this work, so ask any implementer for a written estimate that lists the systems in scope and the number of tool entries expected before you accept a price or timeline.

What sets the timeline for documenting AI tooling?

The timeline is set less by writing than by discovery and sign-off. Discovery means finding every framework, library, pre-trained model and evaluation tool that the AI system uses, which often requires reading build files, container images and notebooks that nobody has reviewed recently. Sign-off means the system owner and security reviewer agree the record is accurate. Ask the implementer to state in writing which systems are in scope, who approves each stage, and how many review cycles are included, because these are the variables that move the schedule most.

What does the work need from our own team?

Your team needs named owners, read access to source repositories, package and model registries, and container registries, and time for review. Assign one owner for each AI system who can confirm its purpose and approved tools, and one security or governance reviewer who signs off the record. Without repository and registry access, discovery becomes guesswork. Agree access requests, the people who grant them and the review hours needed before the engagement starts, since these dependencies usually delay work more than the documentation itself.

Can a small team document tooling resources on its own?

A small team with one AI system and a handful of tools can often do the first register internally, provided someone understands the build pipeline and has authority to approve the record. The work becomes hard to finish alone when tools are spread across many repositories, when the team lacks someone who can explain why each model or evaluation tool is used, or when no one has time to keep the register current. Compare the internal effort, including discovery and review, against the fixed-scope proposal from an implementation partner. Partners prepare the record and evidence, but the organization must own its controls and approve the final content.

Will an auditor accept open-source inventory output as evidence?

An auditor will accept inventory output as evidence only if it is complete, traceable and reviewed, and the decision belongs to the auditor, not the tool vendor. Tools such as GLPI, Wazuh, OWASP Dependency-Track and Trivy can produce software inventories, SBOMs and vulnerability views, which can support the record. The auditor will still ask who approved the list, how it links to the AI system and how gaps are handled. Confirm the expected evidence format with your certification body before you build the process, and keep the raw exports with dated approvals.

How do we keep the tooling record current after the audit?

Keep the record current by tying updates to events that already happen in your delivery process, such as a new framework version, a new pre-trained model or a change to an evaluation tool. Automated sources like Dependency-Track and Trivy scans can flag component changes, and Wazuh or GLPI inventory can show new software on hosts. A named owner should review changes on a fixed cadence and record who approved each one. Ask any implementer for a written plan that covers update triggers, the owner for each trigger and any retainer support for ongoing maintenance.

How DeployOpen scopes and prepares the A.4.4 record

DeployOpen works on A.4.4 Tooling resources as a fixed-scope engagement inside your own infrastructure. We agree in writing which tools, environments and records are covered, then build the inventory and evidence trail your organization owns and your auditor can test.

How we scope

Agree in writing which AI development and runtime environments fall inside the A.4.4 record
List the model frameworks, pre-trained models and evaluation tools your teams actually use today
Confirm which systems of record, such as GLPI or Dependency-Track, will hold each tooling entry

How we build

Configure asset and software inventory collection in your own environment, using the open-source tools agreed in scope
Generate software bills of materials for AI application components with Trivy and load them into Dependency-Track
Define the fields each tooling record holds, including version, owner, license and the date of last review

How we hand over

Deliver the tooling register, its update procedure and the configuration documentation for each collection tool
Provide a written walkthrough of how a tool change is recorded, reviewed and linked to the affected AI system
Transfer administrative access and documentation so your staff own the record after handover

How we support

Offer ongoing support and upgrades as a retainer so the tooling record keeps pace with new releases
Review the register with your AI governance owner before any internal or external audit begins
Confirm with your certification body which evidence format and sampling approach it expects for A.4.4