Self-hosted LLM inference for the public sector: why the boundary decides the design
A self-hosted model endpoint on agency hardware moves the control ownership question in-house: the agency, not a hosted provider, becomes responsible for how prompts, files and logs are protected inside its authorization boundary. The first design decision is therefore not which model to run but where the service sits and who operates it. For serving, vLLM is the engine most teams evaluate for shared multi-user service, because it exposes an OpenAI-compatible server through vllm serve. Ollama fits small or single-host pilots, and a gateway and chat front end sit on top of either engine to handle users, keys and logging.
Scope depends on the service model. FedRAMP's scope covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency, and the M-24-15 scope guidance says systems used only for a single agency's operations on cloud infrastructure, and not offered as a shared service, are out of scope. An inference service that an agency runs on its own hardware in its own data center is not a cloud offering in that sense, but a service shared across departments, or one hosted by a contractor, may be. Confirm with your authorizing official and FedRAMP program office how the service is classified before you choose hardware or a hosting partner, and map each control to the party that operates it. The FedRAMP scope guidance is the starting point for that conversation.
Agencies should also check which regimes attach to the data, since the boundary depends on the data type as much as on the hosting model. Criminal justice information falls under the FBI CJIS Security Policy, and version 6.1 took effect on 26 June 2026 and requires FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys for CJI outside physically secure locations. Controlled unclassified information on non-federal systems points to NIST SP 800-171. Write these requirements into the design before procurement, because they change GPU procurement scope, encryption at rest for model caches and the logging you must keep.
Which public sector requirements shape the inference build
This table maps each public sector requirement that applies to an in-house model endpoint to what it changes in the endpoint, logging, key management and chat front end. Read each row as a question to settle with your authorizing official, compliance office and security team before choosing an architecture, because which rows apply depends on the data, the agency and the contract. The main takeaway is that the model server itself rarely decides compliance; the controls around it do.
| Requirement | What it means for the build | What to confirm |
|---|---|---|
| NIST SP 800-53 Rev. 5 baselines | Low, moderate and high baselines select 149, 287 and 370 controls and enhancements respectively under Release 5.2.0. The inference server, reverse proxy, GPU hosts and logging stack each need a documented owner for the controls they inherit or implement. | Which baseline the system authorization targets and whether your agency uses the current OSCAL baseline profiles |
| FISMA and OMB Circular A-130 | Agency programs must assess and document system security, and Inspectors General independently assess it and report to OMB. Model endpoints need system security plans, risk assessments and POA&M entries like any other federal information system. | Whether the endpoint is added to an existing system boundary or needs its own authorization package |
| FedRAMP scope under M-24-15 | A model service run only for one agency on agency-owned hardware is generally not a cloud shared service, so FedRAMP authorization is usually not the path. A service offered to several departments as a shared service, or hosted by a provider, moves toward FedRAMP scope. | Whether a hosting provider or colocation partner is involved, and whether departments outside the owner will consume the service |
| CJIS Security Policy v6.1 encryption and scanning | For criminal justice information, encryption in transit and at rest outside physically secure locations must use FIPS 140-3 validated modules or FIPS 197 AES with keys of at least 256 bits. Vulnerability scanning must run at least monthly across the inference hosts, proxy and front end. | Whether prompts, retrieved documents or outputs contain CJI, and whether the vendor modules are FIPS validated in your configuration |
| 32 CFR 2002.14(h)(2) and NIST SP 800-171 for CUI | When CUI sits on non-federal systems, agencies apply NIST SP 800-171 requirements unless the CUI category sets other safeguards. Inference hosts, prompt stores and chat history therefore need 800-171 controls even when they run in a contractor or partner facility. | Which CUI categories are in prompts or documents, and which safeguards the governing authority prescribes |
| ADA Title II and WCAG 2.1 AA for front ends | Web and mobile chat front ends delivered through contracts or licenses must meet WCAG 2.1 Level A and AA. The dates are 26 April 2027 for entities serving 50,000 or more people and 26 April 2028 for smaller entities and special districts (28 CFR 35.200). Open-source chat interfaces need accessibility testing and fixes you own. | Your population size category, and whether the chat interface is provided by a vendor whose accessibility conformance you can document |
| NIS2 scope for public administration in the EU | NIS2 covers central government entities regardless of size and lists public administration as a high-criticality sector. Member States may extend it to local administration, and it excludes entities in national security, public security, defense or law enforcement. Inference logs, access control and incident reporting then become NIS2 obligations. | Whether your entity is in scope under national transposition and whether its activities fall under the exclusions |
Who owns which NIST SP 800-53 controls when models run on agency hardware
When the agency operates the GPU nodes, the serving engine, the gateway and the front end, the agency owns most of the NIST SP 800-53 controls that matter for this service. Access control, audit and accountability, configuration management, and system and communications protection sit with the team that installs and runs the stack. A hosting provider or colocation facility may inherit physical and environmental controls, and some infrastructure controls, but only where that provider holds an authorization that covers the relevant control and the agency can cite it. A provider's general security reputation does not transfer control ownership. Before drafting the system security plan, ask the authorizing official which controls are inherited, at what level, and under which provider authorization, then confirm that list against the control baseline in the OSCAL profiles published with SP 800-53B Release 5.2.0.
The most common gap in this design is the endpoint boundary. The vLLM --api-key option protects routes under /v1, /v2, /inference and /cohere, but it does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security documentation warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints meant for users. That proxy allowlist is therefore a control the agency must design, configure, test and document as its own, with the list of permitted routes, the authentication applied to each, and the method used to block the rest. Metrics can also reveal request volumes and user patterns, so decide explicitly who may scrape /metrics and from where.
Which open-source platforms fit an agency inference service
An agency inference service usually needs three layers: an engine that runs the model, a gateway that controls who calls it, and a record of what was asked and answered. The cards below cover the components that commonly fill each layer, with the licensing and exposure checks an agency should make before any of them sit inside its authorization boundary.
vLLM as the shared serving engine
vLLM is an Apache-2.0 serving engine whose server exposes OpenAI-compatible endpoints, so existing applications can point at an agency endpoint without rewriting. It splits large models across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes, and it publishes Prometheus metrics at /metrics for capacity monitoring. Check that the --api-key option does not protect every route. The engine's own security guidance recommends a reverse proxy that allowlists only the endpoints users should reach, and that multi-node traffic runs on an isolated network.
Ollama for smaller deployments
Ollama is an MIT-licensed runtime built on llama.cpp that suits a single server, a pilot team or a branch site with modest concurrency. Its API binds to 127.0.0.1 on port 11434 by default and has no built-in authentication locally, so any change to the OLLAMA_HOST setting must be paired with a proxy that enforces agency authentication. Concurrency is controlled by OLLAMA_NUM_PARALLEL, which defaults to one request per model, and by OLLAMA_MAX_QUEUE. Set OLLAMA_NO_CLOUD=1 so no request leaves the boundary by accident.
LiteLLM for department access
LiteLLM is an AI gateway that presents many model backends in OpenAI format and adds virtual keys, spend tracking and request logging. It lets a shared service issue separate keys to each department, cap spend per key and record which team called which model, which is the basis for chargeback between agencies. Its MIT license covers the core but excludes the enterprise directory, so confirm which features the agency needs are in the licensed part before relying on them. Run the gateway inside the boundary and keep its logging database under the same controls as the model service.
Open WebUI and its branding clause
Open WebUI gives staff a browser chat interface on top of the inference endpoint, but it is not MIT or Apache licensed. Its license prohibits altering or removing Open WebUI branding except for deployments with no more than 50 end users in any rolling 30-day period, with written permission, or under other listed exceptions. An agency serving thousands of staff needs that permission in writing, or it must keep the branding unchanged and confirm the license terms with counsel. Read the full clause before the pilot grows.
LibreChat as an MIT chat option
LibreChat is an MIT-licensed chat interface that can front multiple model endpoints, which makes it a candidate where the branding clause is a concern. Confirm that its authentication options, such as SSO with the agency identity provider, meet the access control requirements the authorization package will cite. Check that conversation storage, file uploads and search indexes sit in agency-controlled databases. Review the release notes on each update, because the chat layer changes faster than the engine beneath it.
Langfuse for interaction records
Langfuse records traces, prompts, outputs and cost for LLM applications in storage the agency controls. Its MIT license covers the core, but the ee directories fall under separate terms, so an agency should confirm which modules it deploys. Record content may itself be sensitive, so decide what is captured, who can read it and how long it is kept before enabling tracing. For high-risk AI systems in the EU, Article 12 and Article 19 of the AI Act set record-keeping and a six-month minimum retention, which a trace store can help evidence. Phoenix, by contrast, is licensed under the Elastic License 2.0, which prohibits offering the software to third parties as a hosted or managed service, so an agency that would run it for other bodies should take that restriction into account.
How the rollout moves from a pilot to a shared department service
The sequence takes an inference service from an agreed authorization boundary and sizing decision to a hardened, monitored endpoint that several departments use through one gateway, with a handover that keeps it patched. Each phase closes with a document or check the authorizing official and the operations team can verify.
Agree the boundary and scope with the authorizing official
Before any hardware is ordered, the agency's security officer, the system owner and the authorizing official decide which models, users, data types and departments fall inside the system boundary. DeployOpen maps the planned components (inference engine, gateway, observability, storage and administrative access) to the control baseline the agency already operates under, and confirms which controls the agency owns and which a cloud or hosting provider would inherit. The phase closes with a signed scope statement and a boundary diagram, and it is complete only when the authorizing official agrees that the proposed shared service matches the agency's authorization path. Confirm early whether any FedRAMP scope question applies to a hosted component.
Size GPUs against agreed concurrency and latency targets
The team turns the agreed user counts, peak concurrency, context lengths and response-time targets into a capacity model for candidate models and quantization levels. Procurement follows the model, not the other way round, and the sizing worksheet records assumptions about peak periods, growth and failover capacity. Quotes from hardware vendors should state accelerator support for the engine chosen, including whether the vendor's platform is among those vLLM documents, and they should itemize power, cooling, rack space and support terms. The phase closes with an approved sizing document and purchase specifications that the agency can test against the engine's own benchmarks before delivery.
Deploy the engine behind a hardened, allowlisted perimeter
DeployOpen installs the inference engine on agency hardware or in the agency's hosting account, with multi-node traffic kept on an isolated network because vLLM's security documentation states that inter-node communication is insecure by default. A reverse proxy exposes only the endpoints users need, since the engine's own API key option does not cover every route, and metrics and health endpoints stay on internal addresses. Model weights are stored in controlled locations with recorded checksums, and configuration is kept under version control. The phase closes with a test report showing that unlisted endpoints refuse outside traffic and that the OpenAI-compatible routes authenticate correctly.
Onboard departments through the gateway with quotas
Each participating department receives its own keys, rate limits and token budgets through a gateway placed in front of the engine, so no department reaches the model servers directly. Department owners agree on who approves access, which data classes each group may send, and how over-quota requests behave. Test accounts validate that one department's usage cannot starve another, and that logs record the calling application and department without storing more prompt content than policy allows. The phase closes with an onboarding record per department and a quota report that the service owner signs off.
Stand up monitoring and log retention
The team connects the engine's Prometheus-format metrics to the agency's monitoring stack, covering request counts, token throughput, latency and KV cache use, and sets alert thresholds agreed with the service owner. Access logs, gateway logs and administrative audit records go to storage with a retention period the agency's records and legal rules set, and retention is tested by restoring a sample. Where the system may fall under the record-keeping duties of the EU AI Act, teams confirm the log content and retention they require. The phase closes with a dashboard, an alert test and a retention configuration that the agency's compliance lead has reviewed.
Hand over runbooks and an upgrade cadence
vLLM publishes new minor releases about every two weeks, so the handover defines who reviews release notes, who tests a candidate version in a staging copy, and who approves the change window. DeployOpen delivers runbooks for scaling, failover, key rotation, model onboarding, incident response and restoring model artifacts and configuration from backup. Agency operators run a rehearsal of the upgrade procedure and a restore test on staging before production responsibility transfers. The phase closes with signed runbooks, a documented upgrade cadence and a recorded handover meeting, and it is complete when the agency's operators have executed both the upgrade and restore drills without outside help.
Buying GPUs for a shared agency service: what drives the bill
The main cost of a shared inference service is accelerator capacity and the people who run it, and vLLM itself has no license fee. vLLM is Apache-2.0 licensed and has no paid edition, so the bill is made up of GPU hardware or rental, power, space and cooling, the surrounding servers and network, implementation, and the operations staff who patch and monitor the stack. Hardware is largely one-time spending, while power, hosting, support contracts and staff time recur every year. Before a budget is set, confirm which accelerator family the agency will buy. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, and the choice narrows the procurement list, the driver stack and the hardware validation work. Ask the integrator to state which GPU families the engine version under evaluation supports, and to record the driver and container image versions it was tested on.
Procurement rules shape the bill as much as the hardware does. Agencies usually buy under a contract vehicle or a fixed set of approved products, so confirm early whether accelerators, servers and any rented capacity must come through that route, and whether the cloud or data center hosting the GPUs is already within the agency's authorization boundary. Chargeback between departments needs the same discipline. Decide the unit that will be billed, such as GPU hours, reserved capacity or request volume, and how the shared fleet's idle time is treated. Tie quotas to the per-user and per-key cost budgets in your gateway, since those limits are what make chargeback enforceable. Remember that recurring patching of the engine, which releases minor versions about every two weeks, and of the host operating system, drives staff hours each year.
When requesting a quote, ask for each line separately: accelerators with part numbers and warranty terms, servers, networking, storage for model weights, power and cooling assumptions, rack space, the software support model, and the named roles and hours covered for operations and on-call. Ask how the price changes if the agency adds nodes for multi-node serving, since vLLM communication between nodes is insecure by default and must sit on an isolated network, which adds switching and segmentation work. Ask for the vendor's and integrator's assumptions about utilization, because low utilization is the most common reason a shared service costs more than the hosted alternative. Confirm which items are one-time, which recur, and which are optional. Ask for the GPU and accelerator support matrix in writing and for a tested upgrade path, so the first engine upgrade is not a surprise cost.
Evidence assessors and inspectors ask for, and which component produces it
This table maps the evidence an assessor or inspector is likely to request for an in-house model service to the component that generates it. Read each row as a question to confirm with your assessor before the evidence set is agreed. The main takeaway is that the engine itself produces little of the required evidence, so the proxy, gateway, configuration management and scanning tools must be designed to capture it from the start.
| Evidence item | Producing component | What the agency should confirm with its assessor |
|---|---|---|
| Access logs for model endpoints | Reverse proxy or API gateway in front of vLLM or Ollama, with identity from the agency directory | Which identity fields, retention period and log destination the assessor expects, and whether logs must be forwarded to an agency SIEM |
| Model and engine version inventory | Configuration repository plus a deployment record for each vLLM or Ollama release and each model weight file with its checksum | How often the inventory must be refreshed and whether model weights are treated as software components in the assessment |
| Configuration baselines | Infrastructure-as-code and server hardening scripts for the inference nodes, proxy and GPU hosts | Which baseline the agency uses for the authorization boundary and who approves deviations, such as exposing /metrics |
| Vulnerability scan results | Scanners on hosts, containers and the engine image, with results stored in the agency ticketing system | Whether CJIS RA-5 monthly scanning applies to the system, and how scans of the inference engine are scoped against newer engine releases |
| Encryption evidence for CJI and CUI paths | TLS termination at the proxy, storage encryption on model and log volumes, and key management records | Whether the modules used are FIPS 140-3 validated, whether AES keys meet the 256-bit requirement for CJI, and which paths the assessor treats as in scope |
| EU AI Act Article 12 event logs and Article 19 retention | Gateway or application logging of events over the system lifetime, stored in a retention store that holds records for at least six months | Whether the system is high-risk under the Act at all, which other law sets longer retention, and who owns the log store |
| Capacity metrics from vLLM /metrics | Prometheus scraping of the vLLM metrics endpoint for request counts, token throughput, latency and KV cache use | Which capacity and availability thresholds the agency uses for reporting, and whether /metrics must be restricted to monitoring networks |
What agencies ask before hiring a partner for an inference service
What drives the cost of a self-hosted inference service for an agency?
The largest cost is accelerator capacity, and utilization decides how much of it is wasted. vLLM has no paid edition, so license fees are not the driver: the cost comes from GPUs or other accelerators, the surrounding infrastructure, implementation, and the people who operate the service. Some costs are one-time, such as hardware purchase, network build-out and the authorization package. Others recur, such as power, space or hosting, support, patching and monitoring. Ask any partner to split a quote into hardware, implementation, operations and compliance evidence so you can see who is paid for what and which lines recur each year.
How long does it take to get a pilot endpoint running?
The timeline depends on the agreed scope, the hardware already available and how fast the agency can approve access, network changes and the authorization path. Engagements are typically fixed-scope, so DeployOpen agrees scope, deliverables and timelines in writing before work starts. A pilot usually needs a named model, one engine deployment, an authenticated endpoint behind a reverse proxy, metrics and a rollback plan. Ask for a dated plan that lists the approvals on the critical path, not a generic duration.
What must the agency provide before the project starts?
Four roles need to be named before work begins: an authorization owner who accepts risk for the system, a network owner who controls segmentation and routing, a hardware owner who provides or approves GPUs and their hosting, and a data owner who decides which datasets and user groups may use the service. The agency also needs to confirm the model list, expected concurrency and latency targets, and which identity provider users will sign in with. Missing any of these usually delays the pilot more than the technical build does.
Can our own staff run the inference service after handover?
Yes, if the agency funds the skills and on-call coverage to operate it. Operating the service means patching the engine, which ships new minor releases about every two weeks (for example, v0.30.0 on September 22, 2026 and v0.31.0 on October 5, 2026), validating new model weights, watching GPU memory and KV cache use, and reviewing access. Ask the partner for a runbook, a documented upgrade procedure and a supported handover, and decide in advance whether a retainer will cover upgrades and continuous testing.
Which FedRAMP scope question applies to an agency-hosted model?
Start with whether the service is a cloud product offered to more than one agency. FedRAMP's scope covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency, and systems used only for a single agency's operations on cloud infrastructure that are not offered as a shared service are out of scope. A model running on agency hardware in its own data center is not a cloud service in that sense, so the agency's authorization path is usually its own ATO under NIST SP 800-53. A shared service across departments may change that answer, so confirm the classification with the agency's authorizing official and its FedRAMP program contacts before design.
How should sharing one inference service across departments be governed?
Define the service owner, the tenant model and the chargeback rules before the first department onboards. Practical controls include per-department API keys or identities, request and token quotas, model allowlists, separate log retention and a catalog of who uses which model. Use a gateway such as LiteLLM or Apache APISIX where budgets and keys are enforced, and make sure the engine itself is not exposed: vLLM's --api-key option does not protect every endpoint, so an allowlisting reverse proxy is needed. Write these rules into a shared service agreement that names the owner and the review cadence.
Does the inference engine need extra hardening for agency use?
Yes. vLLM's security documentation states that communication between nodes in a multi-node deployment is insecure by default, without authorization or encryption, so nodes must sit on an isolated network. Its --api-key option protects the /v1 family of endpoints but not /metrics, /health or /invocations, so a reverse proxy should allowlist only the endpoints users need. Ollama binds to 127.0.0.1 on port 11434 by default and its local API has no authentication, so any exposure must go through a proxy. Ask the partner to show the endpoint inventory and the controls mapped to the agency's SP 800-53 baseline selections.
Questions to put to an implementation partner for agency inference
Start with the questions that decide whether the service can handle load and stay closed. Ask how the partner sizes GPUs for peak concurrency rather than average use, and ask them to show the sizing for each model and context length you plan to serve. Ask which parallelism layout they expect: tensor parallelism inside a node, pipeline parallelism across nodes, and what they will do about the multi-node traffic that vLLM's security documentation says is insecure by default and must sit on an isolated network. Ask for the queue and concurrency settings they will apply, and how they will show that the target latency holds under a full load test. Finally, ask whether they will monitor the Prometheus metrics that vLLM exposes at /metrics and what alert thresholds they will use for KV cache pressure.
Next, ask how they will close the endpoints the engine leaves open. The --api-key option does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the vLLM security documentation advises a reverse proxy that allowlists only the endpoints meant for users. Ask for the proxy configuration, the list of routes they will expose to each department, and how the admin and metrics paths are restricted. Ask how they map controls to the agency's baseline (low, moderate or high under NIST SP 800-53B), which controls the platform inherits from the agency's hosting environment and which the agency must still own, and who writes the control implementation statements. Ask for the written list of inherited controls before any build starts, and confirm that DeployOpen prepares controls and evidence but does not issue authorizations or audit reports.
Last, ask how they will handle engine and model updates under change control. vLLM has shipped new minor releases about every two weeks in 2026, so ask who tracks upstream advisories, how each release is tested in a staging enclave, and how a rollback works if a new version changes behavior. Ask what they will hand over at the end of the engagement: infrastructure as code, container images with recorded digests, model weight checksums, runbooks, the reverse proxy configuration and the evidence pack that assessors will request. Ask for a sample of that evidence package and a named plan for the agency's staff to run the service without the partner.
How DeployOpen scopes and supports an agency inference service
This sequence takes an agency from a loose request for a shared model endpoint to a written scope, a deployed and hardened service, documented control evidence and an ongoing support arrangement. Each step ends with an output the agency can check.
Define the authorization boundary and use cases
The agency's security office, system owners and DeployOpen agree which departments, user groups and data types the service will handle, and which system boundary it sits inside. DeployOpen maps where the inference servers, gateway, storage and logging will run, and confirms with the agency's authorizing official and privacy office whether the service is an agency-operated system or a shared service that may fall under FedRAMP scope. The output is a written boundary diagram and a list of in-scope data categories.
Confirm the control baseline and owners
Working from the applicable NIST SP 800-53 Rev. 5 baseline, the team decides whether the agency's system is assessed at low, moderate or high impact and which control families the agency inherits, shares or owns. DeployOpen drafts a responsibility matrix showing who runs each control on the GPU hosts, the engine, the reverse proxy and the identity layer. The output is a signed control ownership matrix that the agency's security team reviews before any hardware is ordered.
Size hardware and procurement against written targets
Model choices, context lengths, concurrency and latency targets are written into the scope, then mapped to GPU or accelerator counts, memory and networking. Agencies should ask for the sizing assumptions in writing, including how many departments share capacity and how quotas are applied. Procurement may buy hardware directly or through a vendor, and DeployOpen confirms which GPU and accelerator types the chosen engine supports before purchase. The output is a bill of materials and a sizing note tied to the agreed targets.
Deploy the engine behind a controlled endpoint
DeployOpen installs the inference engine, such as vLLM, Ollama or llama.cpp depending on scope, in the agency's environment and places it behind a reverse proxy that allowlists only the endpoints meant for users. Engine ports are not exposed directly, and the team checks that the API key option alone is not relied on for unprotected paths such as metrics or health endpoints. For multi-node setups, inter-node traffic is placed on an isolated network. The output is a deployment record and a list of reachable endpoints checked against the allowlist.
Harden, test and document the service
DeployOpen applies the agreed configuration baseline, integrates identity and logging, and runs security testing on the endpoints, the gateway and the AI interaction paths, including prompt injection and data leakage checks where in scope. Findings are triaged with the agency, and fixes are retested. Runbooks cover model loading, updates, backups of model artifacts and configuration, and incident steps. The output is a test report, a remediation log and a runbook set.
Prepare evidence for the assessment
DeployOpen assembles the artifacts an assessor will ask for, such as configuration exports, access control lists, log samples, metrics dashboards, patch records and the control ownership matrix, mapped to the controls in scope. DeployOpen prepares this evidence but does not issue audit reports, attestations or authorizations; the agency's assessor and authorizing official make those determinations. The output is an evidence package indexed by control ID.
Hand over operations and agree support
Operators from the agency and DeployOpen walk through monitoring, capacity alerts, upgrade procedures and the process for tracking engine releases. The support arrangement is written as either a fixed-scope handover or a retainer covering upgrades and continuous testing. Ask the partner to state the response expectations and what they cover in writing. The output is a signed handover record and an agreed support scope.
