Self-hosted AI gateway for NIST AI RMF third-party risk

Auditors typically ask for a current list of the AI systems in use, the model providers each one calls, and the owner responsible for each. NIST's GOVERN 1.6 outcome expects mechanisms to inventory AI systems, and the Playbook suggests recording attributes such as documentation, source code links, incident response plans and AI actor contacts. Because every call passes through the gateway, the gateway's list of virtual keys, teams and configured model endpoints can serve as the starting point for the provider inventory. The organization's risk owners decide whether the inventory is complete enough. DeployOpen can prepare the export and the inventory structure, but it does not issue assessments or attest to the result.

The policy is a written document, but auditors also want to see it enforced. The gateway supports this through model restrictions on keys and routing configuration that only lists approved endpoints. Evidence usually includes the policy text, the approved provider list with a named owner, the configuration file or export that shows the allowlist, and the change history for that list. The Playbook suggests policies covering transparency, testing and usage instructions for third-party AI, so the policy should reference those points. Your risk owners and auditors determine whether the policy is sufficient. Confirm with your gateway vendor exactly which allowlist controls exist in your chosen version and edition.

Auditors generally ask for exports showing which keys exist, which team or application owns each one, which models each key may call, and what budget and rate limits apply. In LiteLLM, virtual keys are stored in PostgreSQL and carry per-key, per-user and per-team budgets, TPM and RPM limits and model restrictions, so these settings can be exported and compared against the approved list. Keep dated snapshots, because a single current export does not show how access changed over time. Confirm the export format and retention with your platform team. Sufficiency is judged by your auditors and risk owners, not by the implementation partner.

The gateway produces a record of each model call, which can include the calling key, team, model, timestamp, token counts and cost. The retention period is set by your policy and by any applicable law. For high-risk systems in the EU, Article 12(1) of the AI Act requires automatic recording of events over the system's lifetime, and Article 19(1) requires providers to keep logs for at least six months unless other law provides otherwise. Evidence should show what is logged, where logs are stored, who can read them and how long they are kept. Decide with counsel whether prompt content is logged, since that changes the data protection picture. Auditors and your own risk owners decide sufficiency.

Auditors look for a record of how routing, keys, guardrails and providers changed, who approved each change, and how a model or provider was removed when it exceeded risk tolerance. NIST's MANAGE 3.1 outcome expects third-party AI risks to be monitored regularly with controls applied and documented, and the Playbook suggests decommissioning third-party systems that exceed risk tolerances. Store gateway configuration in version control, record each change in a ticket, and keep a runbook for removing a provider from routing. Retain these records alongside the inventory. The organization's change board sets the standard, and auditors judge whether the records are adequate.

Evidence here consists of the provider's security notices, incident reports and vulnerability disclosures, plus your internal record of how each one was reviewed and acted on. The Playbook suggests letting third parties report vulnerabilities and biases, so your contract and intake process should name a contact and a route for those reports. Gateway logs help show which applications were exposed to a provider during an incident window. Keep the provider's material, your assessment and any routing change together in one case file. Confirm with each provider what notifications its contract provides, since the gateway cannot create them. Your risk owners decide whether the response was sufficient.

The biggest cost drivers are the number of model providers and self-hosted models to connect, the number of applications and teams that will call the gateway, and whether you need features that sit behind an enterprise license. LiteLLM's SSO is free only up to 5 users from v1.76.0, and SCIM, JWT authentication, audit logs and team-level guardrails are part of its Enterprise license, so confirm those needs early. Logging retention, high availability on PostgreSQL and Redis, and the integration work with your identity provider and secret manager also add effort. Ask for a quote that lists each component, which costs are one-time (design, integration, testing) and which recur (licenses, hosting, upgrades, support), and the assumptions behind each line. DeployOpen does not publish a price list, and a quote follows from the written scope.

The timeline depends mostly on how complete your inventory of AI systems and model providers is at the start. Before any gateway is configured, you need to know which models, applications, owners and data types are in use, including shadow usage by teams that call provider APIs directly. An incomplete inventory adds discovery work, changes routing and budget design, and extends testing. Ask any partner to split the timeline into discovery, design, build, integration, testing and handover, and to state which milestones depend on your decisions, such as risk tolerance and approved provider lists. The duration should appear in the written scope, not in a verbal estimate.

A gateway project needs three groups from your side. Risk owners must define approved providers, risk tolerances, the decommissioning criteria for third-party systems and who maintains the AI inventory. Identity staff must provide the SSO configuration for the admin UI and the team or group mapping used for keys and budgets. Platform engineers must provide the Kubernetes or virtual machine environment, PostgreSQL and Redis hosting, secret management and network rules. Ask the partner to name these roles in the written scope with the time each one is expected to commit, so the project does not stall waiting for decisions.

Yes, a capable platform team can deploy LiteLLM or Apache APISIX without outside help, and both are open source under licenses that permit internal use. The harder part is the governance work: defining policies, keeping the inventory current, setting budgets and guardrails, retaining logs, testing failover and documenting evidence for assessments. Ask yourself whether your team has time to maintain upgrades (LiteLLM supports the four most recent stable minor versions), security patches and on-call coverage. A partner can take on the implementation, hardening, documentation and handover, or a retainer can cover ongoing support, while your team keeps ownership of policy decisions.

No single product satisfies the NIST AI RMF, and no certification exists for it. NIST describes the framework as voluntary and does not validate or endorse any organization's use of it. A gateway supports several outcomes, including the GOVERN 1.6 inventory of AI systems, the GOVERN 6.1 policies for third-party AI, and the MANAGE 3.1 monitoring and documentation of third-party AI risks. The policies, risk decisions, testing and review process still belong to your organization. DeployOpen prepares systems, controls and evidence for assessments, but does not issue audit reports, attestations or certifications. Confirm with your assessor which evidence they expect.

Most engagements are fixed-scope, so the request should describe the outcome and the boundaries. List the model providers and self-hosted models in scope, the applications and teams that will call the gateway, the identity provider, the logging and retention rules, the high availability target, and the evidence your assessor needs. Add the deliverables you expect, such as configuration, runbooks, integration tests and a documented exit path, and state what is excluded, such as policy drafting or the AI inventory itself if you will own it. Ask each partner to restate the scope, deliverables, dependencies and timeline in writing before work starts, and compare proposals against that same list.