Why self-hosted LLM inference for CUI keeps prompts in scope of your own controls
A model endpoint inside the CUI enclave keeps prompts and outputs under your own NIST SP 800-171 controls, while a hosted API adds an external system to your scope. Engineers, analysts and program staff want AI assistants that summarize specifications, draft proposal sections and explain test results, and many of those prompts and attached files will contain controlled unclassified information. Once a prompt that carries CUI leaves your boundary, the provider's systems, staff, logs and subprocessors become part of the story you must tell an assessor, and you may need contractual flow-down terms and evidence about that provider that you do not control. Running an open-weight model on hardware you administer keeps the prompt, the retrieved context, the model output and the inference logs inside systems that already fall under your access control, audit logging, incident response and media protection requirements. CMMC Level 2 is built on the 110 requirements of NIST SP 800-171 Rev. 2, and 32 CFR 170.14(c)(3) makes the Level 2 requirements identical to that revision. The current position is more nuanced than the original schedule suggests. The Department of War suspended CMMC Phase II on 13 July 2026, and DoW Class Deviation 2026-O0025, Revision 3 directs contracting officers to work with requiring activities on the CMMC requirements in solicitations and contracts. Phase I self-assessments, SPRS postings, annual affirmations and DFARS 252.204-7012 obligations remain in place. Confirm with each contracting officer which obligations apply to your contracts, and treat the inference servers as in scope for CUI until that confirmation is written down.
The practical decision is where the endpoint sits, not whether you can build one. Place the inference servers, the vector store and the logging pipeline in the same enclave as the CUI repositories that feed them, and confirm that your system security plan names them as components with owners, boundaries and data flows. Ask every party that touches the stack, including any integrator, whether it will have administrative access to prompts or logs, and document the answer. A contractor that cannot show this mapping during a self-assessment will struggle to claim that a requirement is MET, because the assessment requires final evidence for each applicable objective.
Which NIST SP 800-171 requirements the inference servers must meet
This table maps the NIST SP 800-171 Rev. 2 requirement families that CMMC Level 2 uses to what the inference service must do, and to the vLLM settings that support each one. Read it as a starting point for your assessor conversation: the engine supports some controls, while the enclave, identity system and your written policies carry the rest.
| Requirement family (CMMC domain) | What the inference service must do | Where it usually fails or needs confirmation |
|---|---|---|
| Access control (AC.L2-3.1.1, AC.L2-3.1.2) | Allow only authorized users, services and roles to reach the endpoint. Put a reverse proxy in front of vLLM that exposes only the /v1 routes users need, and tie access to the enclave identity system. | vLLM's --api-key option does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the docs say not to rely on it alone. Confirm the proxy blocks every other path. |
| Audit and accountability (AU.L2-3.3.1, AU.L2-3.3.2) | Record who called the model, when, and from where, at the proxy or gateway layer. Keep these records in the enclave with retention set by your own policy. | vLLM's /metrics output reports counts and throughput, not user identity. Logs of prompts and outputs need a deliberate design that respects CUI handling rules. |
| Configuration management (CM.L2-3.4.1, CM.L2-3.4.2) | Document the baseline for each inference server: engine version, model weights, container image digests and launch flags. Track changes to vllm serve arguments as controlled changes. | vLLM ships new minor releases about every two weeks (v0.31.0 on October 5, 2026 is the latest listed). Confirm how often you will test and apply updates within your change process. |
| Identification and authentication (IA.L2-3.5.1, IA.L2-3.5.2) | Give each user and service account a unique identity. Do not let a shared API key stand in for individual accounts on the model endpoint. | A single --api-key value shared across teams cannot identify individuals. Confirm where authentication happens: the proxy, the gateway or the enclave directory. |
| System and communications protection (SC.L2-3.13.1, SC.L2-3.13.8) | Encrypt traffic between clients, the proxy and the engine. For multi-node runs, place GPU nodes on an isolated network, because vLLM's inter-node communication is insecure by default without authorization or encryption. | Multi-node tensor or pipeline parallelism over an open network is a common gap. Keep /metrics and the engine port off user networks. |
| Media protection (MP.L2-3.8.1, MP.L2-3.8.3) | Store model weights, caches, logs and any retained prompts on enclave storage with controlled media handling. Sanitize or destroy disks used for CUI before reuse or disposal. | Hugging Face or other model caches written to default locations outside the enclave are frequently missed. Map every cache path to an approved storage location. |
| System and information integrity (SI.L2-3.14.1, SI.L2-3.14.6) | Flag and fix flaws in the engine, container base images and drivers on a documented schedule. Monitor the enclave for abnormal request patterns and unauthorized endpoint access. | Engine releases arrive often, so an assessor may ask for evidence of patch decisions, not just patches. Confirm where /metrics data is scraped and who reviews alerts. |
| Risk assessment and security assessment (RA.L2-3.11.1, CA.L2-3.12.4) | Record the inference service in the system security plan, with its boundary, data flows and inherited controls. Keep a plan of action for any requirement not yet met. | Teams often document the model but not the proxy, the metrics endpoint or the GPU nodes. Confirm the SSP diagram lists every component that touches CUI. |
Open-source platforms that fit a CUI inference enclave
A CUI inference enclave needs an engine that serves models, a gateway that controls who calls it, and a front end that staff use. The options below cover each layer, along with the license limits that decide whether a component can run in your environment at all.
vLLM for multi-GPU serving
vLLM is the engine to evaluate when a model must span several GPUs or serve many concurrent users. It is licensed under Apache-2.0 and has no paid edition, so the license cost is zero and your spend goes to accelerators, the surrounding infrastructure and the people who operate it. The security risk is the default configuration: the server exposes endpoints beyond the API key's reach, so a reverse proxy must allowlist only user-facing routes. Confirm the GPU vendor and driver stack you need is supported before you buy hardware.
Ollama for smaller deployments
Ollama is MIT licensed and suits single-node or small-team deployments where simple model management matters more than peak throughput. It is a practical choice for a secure workstation or a lab enclave serving a limited group. Check that its model registry and update pulls are disabled or mirrored inside the enclave, since a pull from outside the boundary would breach your data flow controls.
llama.cpp for CPU and edge nodes
llama.cpp runs quantized models on CPUs and modest hardware, which is useful for isolated offices or secondary nodes that cannot host GPUs. It adds operational work at scale because each node is managed separately. Before approving it, confirm the license of the specific build and model weights you plan to use, since the engine and the model carry separate terms.
LiteLLM as the access gateway
LiteLLM sits between applications and the engine, issuing keys, recording requests and enforcing budgets in an OpenAI-compatible format. It is MIT licensed except for its enterprise directory, so confirm that every feature you need, such as SSO or advanced audit controls, sits outside that directory. Logs from the gateway are the main evidence source for access control and audit reviews, so decide where they are stored and how long they are kept before go-live.
Open WebUI or LibreChat front ends
Open WebUI and LibreChat give staff a chat interface on top of the same endpoint. LibreChat is MIT licensed. Open WebUI is under a BSD-style license with a branding clause: removing or altering its branding is restricted for deployments above 50 end users in any rolling 30-day period unless you obtain written permission or meet another listed exception. Read the clause against your user count before rebranding the interface.
Licenses that restrict hosted use
Two components in common RAG and observability stacks carry terms that matter for a shared enclave. Dify's modified Apache License 2.0 requires a commercial license to operate a multi-tenant service from the source code. Arize Phoenix is under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. Treat both as design decisions to confirm with counsel before you place them in the enclave.
Evidence assessors ask for and which component produces it
This table maps the evidence a CMMC Level 2 assessor typically reviews for an inference service to the component that produces it. Read each row as a question to answer before the assessment, and confirm with your assessor which artifacts they expect in final form.
| Evidence item | What it shows | Component that produces it |
|---|---|---|
| System security plan and network diagrams | Boundary of the CUI enclave, where the inference servers, gateway and identity provider sit, and how traffic flows between them | Configuration management repository and architecture documentation maintained by the organization |
| Access control lists and endpoint authentication configuration | Who can reach the model endpoint and which vLLM routes are exposed to users | Identity provider, reverse proxy allowlist and vLLM server configuration (the --api-key option does not cover every route) |
| Audit logs of inference requests | Who called the model, when, from where and which route was used, without relying on the model server alone | Gateway access logs and the reverse proxy, forwarded to the organization's log store; vLLM request logs supplement them |
| Metrics history | Request volume, throughput, latency and KV cache use over time, supporting capacity and availability claims | vLLM Prometheus metrics at /metrics, scraped and retained by the monitoring stack |
| Container and model artifact hash records | That the engine image and model weights in production match what was approved and transferred into the enclave | Build pipeline, artifact registry and the configuration management repository |
| Patch and engine upgrade records | When the vLLM version changed, what was tested and who approved it, given that vLLM ships minor releases about every two weeks | Change tickets, configuration management repository and deployment pipeline records |
| Backup and recovery records for model configuration | That serving configuration and artifacts can be restored to a known state | Backup system and configuration management repository |
| Final-form assessment evidence | Objective-by-objective results against NIST SP 800-171A, with evidence in final form; drafts do not count toward MET | Assessment team and the organization's policy and procedure library, reviewed and signed before submission |
What the engine cannot satisfy on its own
The inference engine will not meet NIST SP 800-171 Rev. 2 on its own, and the vLLM security documentation makes that gap concrete. The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The docs warn against relying on the key alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Node-to-node traffic in a multi-node deployment is insecure by default, with no authorization or encryption, so the nodes must sit on an isolated network. The contractor owns the proxy configuration, the network segmentation, the enclave boundary and the test that proves unlisted paths return nothing to a user. Confirm each of these in the actual deployment, not in the design document.
The rest of the control set sits with the contractor as well: identity for each person who can reach the endpoint, log retention for prompts and access events, personnel screening and training for the staff who operate the servers, and an incident response plan that covers a model server or its logs being exposed. Rev. 3 adds a further obligation. Its organization-defined parameters (ODPs) are values the nonfederal organization must set when a federal agency does not specify them, and NIST does not assign them. Ask the contracting officer whether ODP values have been published for your contract, and record your own assigned values for any that have not. Because the FAR overhaul proposed rule would tie CUI contractors to Rev. 3 with DoW-published ODP values, track that rule before you finalize them.
Treat this as a decision list for the implementation partner: who owns the reverse proxy policy, who signs off on the ODP values, and who keeps the evidence that the endpoint allowlist, network isolation and access logs were tested in final form. Those artifacts are what an assessor will request, and they need to exist before the assessment, not be reconstructed afterward.
Gaps tooling does not close
A model server can meet technical controls and still fail an assessment if the people, documents and records around it are weak. The cards below cover the decisions and records that no inference engine, gateway or dashboard makes for you. Each one names the item the contractor must decide or own before the service carries CUI.
Policy and System Security Plan ownership
The System Security Plan must describe the inference service as part of the CUI boundary, including who runs the GPU hosts, the endpoint, the gateway and the logs. Engines do not write this document for you. Decide who signs each control statement, where the plan lives, and how it changes when the engine is upgraded. Check that the plan names the vLLM server, any reverse proxy and the metrics endpoint as components in scope.
Model artifact provenance and approved lists
Open-weight models arrive from public repositories with their own licenses, versions and checksums. The contractor must decide which model families, quantizations and container images are approved, who may add one, and how each artifact is traced back to its source. Record the hash of every weight file and image in the configuration baseline. Confirm that the approved list also covers the model licenses, since a license can restrict use in some contracts.
CUI marking and data handling in prompts
Prompts, uploaded files and RAG indexes can carry CUI even when no one labels them. Decide how users mark CUI, which fields the gateway must block or log, and whether vector stores inherit the marking of their source documents. Retention of prompt logs is also a decision, because logs can themselves contain CUI. Confirm with your contracting officer how these records are handled under your contract's DFARS 252.204-7012 obligations.
Engine patch cadence and change control
vLLM ships new minor releases about every two weeks, so a fixed upgrade cadence is a control decision, not an operations detail. Decide how each release is tested in a non-CUI staging environment, who approves it, and how the change is recorded in the plan. Decide also how emergency security fixes skip the normal window. Track the version in use and the date of each change so that the assessor can see that the baseline stayed current.
Identity and access reviews
The vLLM API key protects only some endpoints, so user access depends on a reverse proxy, your identity provider and group mappings that you maintain. Decide who may call the endpoint, who may administer the servers, and how service accounts are issued and retired. Schedule periodic access reviews and keep the approvals as evidence. Check that administrative paths such as the metrics and health endpoints are not reachable from user networks.
Assessment readiness and evidence
Level 2 requires a self-assessment or a certification assessment by a C3PAO, and a requirement counts as met only when the evidence is final. Decide which assessment type applies to your contracts, given the current phase status. Assign an owner to collect configuration exports, access reviews, log samples and change records for each inference control. Ask your counsel or compliance lead to confirm the current requirements before you plan the evidence set.
Questions contractors ask before building an in-house CUI model service
Does a self-hosted model keep CUI prompts in scope of our existing controls?
Yes, a model endpoint inside your CUI enclave generally stays within the same controls as any other system that processes CUI, because prompts, retrieved documents and outputs are CUI-bearing data that the enclave already protects. The benefit is that you decide who can reach the endpoint, where logs are stored and how the environment is assessed, rather than inheriting a third party's data handling terms. Whether a given design is in scope depends on your system security plan and boundary, so confirm the data flows with your assessor. A hosted API outside the enclave would usually add an external system to that boundary.
Which NIST SP 800-171 revision applies to CMMC Level 2 today?
For CMMC Level 2 purposes, the CMMC rule still incorporates NIST SP 800-171 Revision 2, which has 110 requirements, even though NIST withdrew Revision 2 on 14 May 2024 when it published Revision 3. The Department of War said on 13 July 2026 that it will enforce Revision 2 in the interim. A FAR overhaul proposed rule published 23 June 2026 would move contractors to Revision 3, but it was not final as of October 2026. Ask your contracting officer which revision the solicitation or contract names, because DFARS 252.204-7012 text in recent class deviations refers to Revision 2 explicitly.
What does an in-house CUI model service cost, and what drives it?
The cost has four parts: GPU or accelerator capacity, the surrounding infrastructure, the people who build and operate it, and the assessment work to show the controls work. GPU capacity and power are largely recurring, while hardware purchase, initial configuration, boundary changes and the first documented assessment preparation are mostly one-time. Engine software such as vLLM has no paid edition, so license cost is not the main driver. Ask any partner for a written quote that separates hardware or rental, implementation, operations hours per month, and assessment support, with the assumed number of concurrent users and models stated.
How long does a CUI inference deployment take, and what depends on the contracting office?
Duration depends mainly on hardware lead times, the readiness of your enclave and the contract requirements, and no fixed timeline should be assumed before scoping. The CMMC schedule is in flux: on 13 July 2026 the Department of War suspended CMMC Phase II, which had been set for 10 November 2026, and a 60-day review by a CMMC Reform Task Force began. Phase 1 self-assessments, SPRS postings and DFARS 252.204-7012 obligations remain in place. Contracting officers can specify Level 1 (Self) or Level 2 (Self) assessments in new and existing solicitations, so confirm with your contracting officer which assessment level applies before you set a delivery date.
Can our own team build and run a CUI-scoped model service?
A capable platform team can build it if it has GPU operations experience, Linux and container administration, network segmentation skills and someone who owns the security documentation. The harder part is ongoing: vLLM ships new minor releases about every two weeks, so someone must test and apply updates within your change control. Plan for on-call coverage, capacity monitoring and evidence collection, not only installation. If those skills or hours are not available, a partner can take fixed-scope implementation or a retainer for support, while you keep ownership of the boundary, the data and the decisions.
Can we use a SaaS model on CUI if the provider signs a contract?
Whether a SaaS model can handle CUI depends on the provider's own assessment status and how your contract and DFARS 252.204-7012 obligations apply to any external service that stores, processes or transmits CUI. Do not assume that a commercial terms page or a general certification satisfies those obligations. Ask the provider for its written statement of how it meets the required safeguards for CUI, where the data is processed and retained, and whether your assessor will accept that evidence. Many teams keep CUI prompts on in-house endpoints for this reason, and use hosted services only for non-CUI work.
What should we ask a partner before signing a CUI inference engagement?
Ask how they will place the inference servers inside your enclave boundary, which NIST SP 800-171 Revision 2 requirements their work supports, and how they will evidence each one. Ask how they secure the endpoint, since vLLM's --api-key option does not protect every route, so a reverse proxy that allowlists only user-facing endpoints is usually needed. Ask how they isolate multi-node traffic, which vLLM documents as insecure by default, and how they monitor metrics, handle engine upgrades and hand over operations. Request a written scope with deliverables, responsibilities and an itemized quote before work starts.
Does DeployOpen certify our CUI environment for CMMC?
No. DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms in your own infrastructure, and it prepares systems, controls and evidence for compliance assessments. It does not issue audit reports, attestations or certifications. Level 2 certification assessments are performed by C3PAOs authorized or accredited by the CMMC Accreditation Body, currently The Cyber AB, and Level 3 assessments are performed by DCMA DIBCAC. Scope and deliverables for any DeployOpen work are agreed in writing before work starts.
Building and scoping the enclave step by step
Start by mapping the CUI data flow before any hardware is ordered. List every point where a prompt, retrieved document, embedding, model output or log record is created, stored or transmitted, including the application that calls the endpoint, any retrieval store, the gateway and the observability tools. Any component that touches CUI is in scope for the requirements, so the decision to keep a component outside the boundary has to be documented and defended. Then draw the boundary around the minimum set of components: the inference nodes, the model artifact store, the reverse proxy, the identity provider and the log store. Confirm with your assessor or the CMMC Accreditation Body's guidance how your organization's external service providers and cloud accounts are treated under your contracts before you decide where the line falls.
Deploy the inference nodes on an isolated network segment with no default route to the internet, and place the vLLM server behind a reverse proxy that allowlists only the paths your users need, such as /v1/chat/completions and /v1/embeddings. The vLLM security documentation says the --api-key option does not protect every route, including /metrics, /health and /pause, so the proxy allowlist is the control that does the work, not the key alone. For multi-node runs, treat inter-node traffic as insecure by default and keep those nodes on a dedicated isolated network. Record each setting, such as the exposed routes, bind addresses and node-to-node network rules, in configuration under version control so the baseline can be shown to an assessor.
Enable logging and metrics as evidence sources, not just operational tools. Scrape the Prometheus-format metrics at /metrics from inside the enclave only, and send proxy access logs, authentication events and administrative actions to a log store that the CUI team controls. Decide now who can read prompts and outputs, how long records are kept and whether prompt content is logged at all, because logging content creates new CUI records to protect. Next, test each requirement against the assessment objectives in NIST SP 800-171A (June 2018), using final-form evidence such as configuration exports, access reviews and test records, since drafts do not count. Finally, write the system security plan describing the boundary, data flows, inherited controls and the owner of each requirement, and confirm the current assessment path with your contracting officer, because the phase schedule has changed in 2026.
How DeployOpen scopes, deploys and documents a CUI inference enclave
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own infrastructure, which is where a CUI inference enclave has to live. The work starts with a written scope that ties each inference server to the NIST SP 800-171 Rev. 2 requirements the contract demands, then builds, tests and documents the enclave so the organization can show its own assessor the evidence.
How we scope
- Confirm which contracts, DFARS clauses and CMMC level apply before any build starts
- Map every inference server, gateway, log store and admin path that touches CUI
- Agree in writing which requirements the client owns and which DeployOpen configures and documents
How we build
- Deploy vLLM or Ollama inside the CUI enclave with an allowlisting reverse proxy in front of the engine
- Configure access control, encryption, audit logging and isolated networking for model nodes and metrics endpoints
- Run security testing against the endpoints the engine exposes, including those the API key does not protect
How we hand over
- Deliver system documentation, control mappings and evidence collected in the form an assessor will review
- Train the client's operators on upgrades, model artifact verification and incident handling
- Document the engine release process so patching and re-assessment follow a defined procedure
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Re-test the enclave when the engine, model or boundary changes and update the evidence set
- Keep the assessment decision with the client, the C3PAO or the assessing body, not DeployOpen
