NIST AI 600-1 data privacy risk for generative AI

What NIST AI 600-1 data privacy risk asks organizations to show

NIST AI 600-1 is a voluntary profile of the AI RMF, and it has no certification, so no fee or certificate exists for privacy risk itself. The cost of meeting it comes from internal work (inventorying data, setting access controls, running privacy testing and keeping records) and from any outside help you choose to buy. The Generative AI Profile defines the data privacy risk as impacts from leakage and unauthorized use, disclosure or de-anonymization of biometric, health, location or other personally identifiable information or sensitive data. Your organization owns its privacy controls. An implementation partner can prepare systems, logs and evidence, but only an independent auditor or assessor issues any report, and NIST does not validate or endorse any organization's use of the AI RMF.

The work you must show depends on how your organization touches generative AI. If you train a model from scratch, you carry the most exposure: you must document where personal data in training sets came from, whether individuals had notice or could consent, and why the data was used for that purpose. AI 600-1 notes that most model developers do not disclose their training data sources, so a buyer cannot rely on vendor silence as evidence. If you fine-tune a model or connect one to your own records through retrieval, the data you feed it becomes your responsibility even when the base model belongs to someone else. If you only call a third-party generative AI service, your evidence centers on what you send in prompts, what the vendor retains, and what your own applications store from outputs.

Across all three cases, confirm with your privacy and legal teams which personal data categories are in scope, who approves each data flow, and where the decisions are recorded. Ask your vendors in writing whether they train on your inputs and how long they retain them. Then decide who runs the privacy measurement and red teaming work described in the sibling sections, and who signs off on the results. If an outside firm will prepare the evidence, the scope, deliverables and timelines should be agreed in writing before work starts, and any formal attestation about your privacy controls should come from an independent assessor you select separately.

Which privacy exposures the Generative AI Profile names

NIST AI 600-1 treats privacy as a risk that runs through how a generative model is trained, how it behaves when queried and what data it can reach at runtime. The four exposures below cover those paths, and each one maps to a different set of evidence an assessor will ask to see.

Personal data in training sets

Training on personal data raises risks to transparency, individual participation (including consent) and purpose specification. NIST notes that most model developers do not disclose the specific data sources used for training, so a buyer or deployer often cannot confirm what personal data the model absorbed. Check whether the organization can document where its training and fine-tuning data came from, which personal fields it contains, and what legal basis or consent covered their use. If the model comes from a third party, ask the vendor for this documentation in writing.

Memorization and leakage

Data memorization occurs when a model reproduces sensitive information from its training data in its outputs. The risk is not limited to public models, since fine-tuned models trained on internal records can repeat names, identifiers or health details verbatim. Evidence that this exposure is managed includes red teaming records showing attempts to extract training data samples, deduplication checks on training data (particularly synthetic data), and output filters that are tested against known personal records. Ask whether these tests ran before release and again after each model update.

Inference of sensitive data

A model may correctly infer personal or sensitive data that was never in its training set and was never provided by the user, by combining information from disparate sources. Location, health, biometric or other sensitive attributes can be deduced from ordinary prompts. The control question is whether the organization has identified which inferences its system could make, restricted sensitive attribute inference in the deployment design, and monitors outputs for periodic exposure of personal or sensitive data, as suggested by action MP-4.1-001.

RAG knowledge base compromise

AI 100-2e2025 classifies privacy compromise as an attacker objective, including unauthorized access to sensitive data a model reaches at runtime, such as the knowledge base of a retrieval-augmented generation application. Because the retrieval layer can expose documents to users who should not see them, access control must follow the source document, not just the chat interface. Evidence includes document-level permissions enforced at retrieval, logs of which records were returned for each query, and red team results showing whether restricted content can be surfaced through prompts.

Which suggested actions address the privacy risk

The Generative AI Profile's privacy actions form a loop: find what personal data is in play, test whether the model exposes it, measure re-identification exposure, control access, and keep watching production. Each step leaves an artifact an assessor can inspect.

Document sensitive personal data flows
The Playbook suggests documenting how personally sensitive information in datasets is collected, used and disclosed. The data owner and the AI team map each dataset that holds biometric, health, location or other personal data, including training sets, fine-tuning data and retrieval sources. The output is a data flow record stating sources, lawful basis or consent position where applicable, purpose, recipients and retention. Treat it as complete when every dataset feeding the system has an entry that a privacy reviewer has signed off. Most model developers do not disclose their training sources, so this record is often the first place an organization can show its own provenance position.
Verify training data deduplication
Under MEASURE 2.10, the Profile asks teams to verify deduplication of training data, with particular attention to synthetic data (MS-2.10-003). Duplicated records make memorization more likely, so the data engineering team checks for exact and near-duplicate records and documents the method and thresholds used. The output is a deduplication report showing input and output record counts, the matching method and any residual duplicates with a rationale. The step is complete when the report is reviewed and the open exceptions are accepted or remediated by a named owner.
Quantify re-identification risk
The Playbook suggests measuring re-identification risk with metrics such as k-anonymity, l-diversity and t-closeness on datasets that contain personal attributes. The data science team computes these metrics on the quasi-identifiers that matter for the dataset and records the parameters chosen. Teams should confirm with a privacy specialist which metric fits each dataset, because no single score settles whether a record is identifiable. The output is a metrics table per dataset with the thresholds accepted and the decision taken for any dataset that falls below them.
Red team for memorization and inference
MEASURE 2.10 actions call for red teaming on output of training data samples, model extraction and membership inference (MS-2.10-001). A red team prompts the model to reproduce training content, attempts to determine whether specific records were in the training set, and tests whether the model infers sensitive attributes it was never given. Results are logged with prompts, outputs, the tester and the severity assigned. The step closes when findings are triaged and any reproduced personal data has a remediation or acceptance decision recorded.
Apply access controls to sensitive data
The Playbook suggests setting access controls for sensitive training and production data. Platform and security engineers restrict read and export rights to named roles, separate training data stores from general analytics, and cover retrieval knowledge bases as well as model artifacts, since AI 100-2e2025 counts unauthorized access to those sources as a privacy compromise. The output is an access matrix, the identity configuration exported from the platform, and recent access review records. Confirm with your platform team that retrieval indexes and logs inherit the same restrictions as the source data.
Monitor queries that isolate personal records
The Playbook suggests monitoring queries that isolate personal records, meaning prompts or sequences designed to pull out one individual's data. Security operations configure logging of prompts and responses, define detection rules for repeated or narrowing queries about a single person, and route alerts to a reviewer. The output is the rule set, sample alert records, and a log of investigations with outcomes. Confirm retention periods and privacy limits on the logs themselves with counsel, since prompt logs can contain personal data.
Apply differential privacy when sharing dataset information
Where dataset information is shared, such as aggregate statistics, external research releases or model evaluation summaries, the Playbook suggests using differential privacy. The team decides whether sharing is needed, selects a mechanism and privacy budget with a specialist, and records those parameters alongside the released output. The evidence is the release memo showing the mechanism, parameters and reviewer approval. Where differential privacy is not used, the file should explain the alternative control applied.
Monitor generated content for privacy exposure
Action MP-4.1-001 suggests periodic monitoring of AI-generated content for privacy risks and addressing any possible exposure of personal or sensitive data. The operations team samples production outputs on a schedule it sets, scans for personal identifiers, and sends confirmed exposures to incident handling. The output is the monitoring schedule, sampling records, detection results and remediation tickets. Because the Profile frames this as periodic, the schedule and its review date should be written into the program documentation and checked against what the team actually runs.

How the program changes between building models and buying them

An organization that trains or fine-tunes a generative model on personal data owns the privacy risk from the first dataset onward. The Generative AI Profile's data privacy risk covers leakage and unauthorized use, disclosure or de-anonymization of biometric, health, location or other personally identifiable information, and a builder has to answer for every one of those pathways inside its own pipeline. That means documenting where each personal data source came from, what purpose it was collected for, whether the people in it agreed to this use, and how records are de-identified before training. It also means controlling memorization, which NIST AI 600-1 describes as models leaking sensitive information from their training data. Builders should plan to run the red teaming and privacy measurement actions under MEASURE 2.10 against their own model, and they carry the cost of doing that work themselves or through a partner.

An organization that calls a third-party model API or runs retrieval over internal records has a different and often narrower job. Its main privacy exposure shifts to what it sends in prompts, what its retrieval index exposes, and who can query both. Most model developers do not disclose the specific training data sources, so the buyer cannot verify the training provenance directly. The practical response is to rely on contract terms and on its own documentation: confirm in writing whether prompts and outputs are retained, whether they are used for training, where they are processed, and how deletion requests are handled. Ask the vendor which of these terms are available in the agreement itself, and record what the buyer has checked. Retrieval systems need their own controls, including access checks that mirror the source records and monitoring for queries that isolate individual people.

Consider an illustrative case in which a bank fine-tunes a model on customer correspondence. The bank would own the lawful basis review, the de-identification method and the memorization testing for that model. A second, illustrative bank that only calls a vendor model over an API would instead focus on prompt filtering, retention settings in the contract and access controls on its retrieval index. Both organizations should confirm with their vendor or legal team which obligations stay with them, because a buyer cannot fully outsource accountability for personal data it chooses to send.

What a privacy program costs and who is paid for each part

This table breaks a generative AI privacy program into its cost components, showing who is paid for each, whether the cost is one-time or recurring, and what pushes it up. Use it to build a scope request, then ask each vendor or assessor to price the components separately so you can see where your money goes. No public figures for these components are supplied here, so treat every cost as something to confirm in a written quote.

Cost component	Who is paid or who carries it	One-time or recurring	What drives it up
Internal privacy and security staff time	Your own staff (salaried time, not a vendor invoice)	Both: setup work is one-time, monitoring and review recur	Number of models in production, volume of personal data, and how many teams build or buy generative AI
Data mapping and documentation of personal data in training and prompt flows	Your staff, or an implementation partner preparing the records	Mostly one-time, with updates on each new dataset or model	Number of data sources, undisclosed or third-party training data, and gaps in existing records of processing
Privacy testing and AI red teaming (including output of training samples, membership inference and model extraction)	A security testing firm or in-house red team	One-time for pre-release testing, recurring for each major model change	Model type, number of attack paths, access to model internals, and whether retrieval sources hold personal data
Tooling for monitoring outputs, access control and query monitoring	Software licenses or open-source tooling, plus staff or partner time to configure it	Recurring for licenses, hosting and operations; one-time for integration	Log volume, number of users and data stores, and retention requirements
Re-identification and privacy measurement (k-anonymity, l-diversity, t-closeness, differential privacy)	Internal data scientists or a specialist consultant	One-time per dataset, recurring when data is refreshed	Size and sensitivity of datasets, and how often data is shared outside the organization
Independent assessor, only if a customer or regulator requires one	An independent auditor or assessor, not DeployOpen	One-time per assessment cycle, recurring where the requirement repeats	Scope of the assessment, number of systems covered, and the assurance report type requested
Ongoing monitoring, periodic review and re-testing	Internal team or a retainer with an implementation partner	Recurring	Release frequency, incident history, and changes in data sources or regulation

What must be in place before a privacy review

A privacy review of a generative AI system tests whether the organization can say where personal data sits, who can reach it and what the model emits. The six items below are the prerequisites an assessor or internal reviewer will look for before they can judge the privacy risks NIST AI 600-1 describes.

Inventory of generative AI systems and datasets

Keep a register of every generative AI system in use or in build, including vendor-hosted models, fine-tuned models, retrieval knowledge bases and the datasets behind each. Each entry should name the owner, the training and grounding sources, and whether personal, biometric, health or location data is present. To check it, pick one production system and ask whether its register entry lists every data store the model can read. Gaps usually sit in knowledge bases that teams added after launch.

Documented personal data flows

Map how personal data moves from collection to training, grounding, prompts, outputs, logs and deletion. NIST AI 600-1 notes that most model developers do not disclose their training data sources, so the deployer's own flow map is often the only record of what went in. To check it, trace one sample record end to end and confirm the map shows each copy, each processor and each retention point.

Access controls on training and production data

Restrict who and what can read training sets, fine-tuning corpora, model artifacts and retrieval stores that hold personal or sensitive data. Separate service accounts from human access and review grants on a set schedule. To check it, export the access list for one training dataset and one knowledge base, then confirm each entry has a current business reason and that no shared credentials remain.

Logging of queries and outputs

Record prompts, retrieved documents and generated outputs with enough detail to detect extraction attempts and reconstruct exposures, while limiting what the logs themselves store. The GAI Profile's monitoring action MP-4.1-001 calls for periodic checks of AI-generated content for personal data exposure, which depends on this record existing. To check it, confirm a test prompt that requests a personal record leaves a log entry that a reviewer can search and that log access is restricted.

Red teaming plan for extraction and membership inference

Write a test plan that covers output of training data samples, model extraction and membership inference, which are the attacks MEASURE 2.10 actions in the GAI Profile name. Define the target systems, the attacker assumptions, the pass and fail criteria and who receives the findings. To check it, confirm the plan includes at least one test for each attack and that a dated record shows results were reviewed by someone outside the team that built the system.

Named owner for privacy measurement

Assign one accountable person for privacy measurement, with authority to pause a system and a documented reporting line to the privacy officer. The Playbook's MEASURE 2.10 suggestions, such as re-identification metrics like k-anonymity and differential privacy for dataset sharing, only produce results if someone is responsible for running them. To check it, ask who signed the last privacy measurement report and whether that person can stop a release.

Who owns, who prepares and who assesses privacy controls

The organization that develops, deploys or buys the generative AI system owns its privacy controls, the decisions about which personal and sensitive data may enter training, retrieval or prompts, and the accountability for those decisions. NIST AI 600-1 frames privacy risk as impacts from leakage and unauthorized use, disclosure or de-anonymization of personal or sensitive data, and the organization is the party that answers for those impacts to individuals and regulators.

Data decisions: Approving which datasets, knowledge bases and user inputs contain personal data, and recording the lawful basis, purpose and retention for each.
Risk tolerance: Setting the privacy risk tolerance that MAP and MANAGE actions are measured against.
Access rules: Defining who may query sensitive training and production data and approving exceptions.
Incident response: Owning the decision to notify affected individuals or regulators after a leakage event.
Documentation: Keeping the dataset records and privacy decisions that an auditor will ask to see.

Check that each of these decisions has a named owner and a signed record, because a partner can prepare the evidence but cannot make these decisions for you.

Questions buyers ask about NIST AI 600-1 privacy work

Does NIST certify privacy compliance for generative AI under AI 600-1?

No. NIST AI 600-1 is a voluntary profile of the AI Risk Management Framework, and neither NIST nor the AI RMF has a certification, accreditation or conformity assessment program. NIST does not validate or endorse any individual organization's use of the framework. If a customer or regulator asks for proof, the organization can show documented privacy actions, test results and monitoring records, and an independent auditor or assessor can report on them under a different standard or engagement. No firm can issue a NIST privacy certificate, and DeployOpen does not issue audit reports or certifications.

What drives the cost of addressing generative AI privacy risk?

Cost depends mostly on how much personal or sensitive data the system touches and how the model was built. The main components are internal staff time for data mapping and policy work, technical work on access controls, logging and monitoring, red teaming against memorization and inference attacks, and any external specialist or assessor fees. Costs rise with the number of training datasets, the use of third-party models whose data sources are undisclosed, the number of systems in scope, and the depth of measurement such as re-identification testing. Recurring costs come from monitoring, periodic retesting and reviews after model changes. Ask any provider for a written scope listing systems, data sources, deliverables and the testing method before comparing quotes.

How long does a generative AI privacy program take to put in place?

The duration depends on the scope and on the current state of the organization's data inventory, access controls and logging. Work usually runs longer when training data provenance is unclear, when many models or retrieval sources hold personal data, or when legal and privacy teams must approve data uses before testing begins. Monitoring and red teaming are ongoing rather than one-time, so a program has a setup phase and a recurring phase. No fixed duration applies to every organization. Ask for a timeline tied to named milestones in the written scope, and confirm which steps depend on your internal approvals.

What does the work need from the internal team?

The work needs people who know the data and the business purpose. Expect to involve the privacy officer or data protection lead, the owners of training and retrieval datasets, security engineers who manage access and logs, and the product owner who decides how outputs are used. Legal input is needed on purpose specification, consent and lawful basis for personal data in training sets. Someone must also own decisions, such as whether a dataset can be used or a model can be deployed. Before starting, confirm who can grant access to data and systems and how quickly those approvals can be given.

Can our organization address generative AI privacy risk without outside help?

Yes, many organizations can, provided they have staff with privacy, security and machine learning skills and the authority to change data handling. The NIST actions, including documenting how sensitive data is collected and used, setting access controls and monitoring queries that isolate personal records, are ordinary governance and engineering work. Outside help is most often used for red teaming against memorization and membership inference, re-identification measurement, and building repeatable monitoring. DeployOpen plans, implements and documents self-hosted open-source platforms inside a client's own environment and runs security testing, including AI systems, but the organization keeps ownership of its controls and decisions.

What do customers usually ask to see as evidence of privacy controls?

Customers typically ask for a description of the training data sources and how personal data is handled, the access control design and logs for sensitive training and production data, and records of privacy testing. Common items include red teaming results for output of training data samples and membership inference, evidence of deduplication checks on training data, and monitoring records for personal or sensitive data appearing in outputs. They may also ask for re-identification risk measurements and the documented decisions taken on the results. Confirm with each customer which items they need, because contracts and regulators may ask for different formats.

How DeployOpen prepares privacy evidence for a review

DeployOpen plans, implements and documents self-hosted open-source platforms inside the client's own infrastructure, and prepares the privacy-related systems and evidence that a review of generative AI data privacy risk may request. The organization owns its controls and its risk decisions, and only an independent auditor, assessor or certification body issues reports or certificates.

How we scope

Scope names the generative AI systems, training datasets and production data stores that fall under the privacy review.
The written scope lists which personal data categories, such as health, biometric or location data, are in and out of testing.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.

How we build

Access controls are configured for sensitive training data, fine-tuning data and retrieval knowledge bases in the client's environment.
Query and output logging is set up so teams can monitor responses for exposure of personal or sensitive data.
Privacy measurement and red teaming for memorization, membership inference and model extraction are run against the agreed scope.

How we hand over

Documentation covers how personal data in datasets is collected, used and disclosed, with the sources recorded for each dataset.
Test results, re-identification risk measurements and remediation records are packaged as evidence the client's assessor can sample.
Handover includes a register of open findings, owners and the decisions the organization still needs to make.

How we support

Ongoing support, upgrades and continuous testing are available as a retainer after handover.
Retainers can keep privacy monitoring, periodic red teaming and evidence refreshes running as models and datasets change.
Ask any provider to confirm in writing which privacy tasks it performs and which the organization's own teams must own.