Private LLM chat for the public sector

Why a private LLM chat for the public sector starts with data classification

An agency can deploy a self-hosted Open WebUI or LibreChat front end with Ollama or vLLM models on its own infrastructure, but the security office must first decide what staff may paste, upload or ask. The classification scheme sets the limits: which categories of information may enter a prompt, which may be stored in chat history, and which must never reach a model. Before any configuration work, the agency should write those rules as a short table keyed to its existing classification labels, then confirm with its records and privacy officers which categories fall under statutes that prescribe their own safeguards. Drafting and summarizing use cases tend to fail here, because staff paste whatever sits in front of them unless the platform and the policy both say otherwise.

Sovereign hosting follows from that classification. If the prompts and files in scope are controlled or criminal justice information, the agency must confirm which hosting and cloud authorization rules apply before choosing where the models run, since a hosted service may bring the tool into a federal authorization scope. Running Ollama or vLLM on agency-controlled servers keeps prompts inside the agency's own boundary, and the agency should confirm with its security assessor how the chosen hosting model maps to its control baseline. Procurement also shapes the choice. Open WebUI is distributed under the Open WebUI License, and its branding clause applies to deployments above 50 users in any rolling 30-day period unless the operator holds an enterprise license or written permission. A statewide or department-wide rollout will exceed that threshold, so the procurement file should budget for the license question early, while LibreChat (MIT) is an alternative front end to evaluate on the same criteria.

The procurement team should ask an implementation partner for three written deliverables before work starts: the classification-to-prompt rule set, the hosting and authorization analysis for each data class, and a license position for the chosen front end at the expected user count. Accessibility belongs in the same review. State and local entities that serve 50,000 or more people must meet WCAG 2.1 Level A and AA for web content and mobile apps from 26 April 2027, and smaller entities from 26 April 2028, so the interface selected for staff must be tested against those criteria before rollout.

Which public sector rules shape the assistant build

This table maps each public sector requirement to its source and to the build decision it forces on a staff chat assistant. Read it as a checklist for the architecture review: each row names what the rule asks for and what the implementation team must decide or evidence. Confirm applicability with your legal, records and security owners before scoping, because several rules depend on the data classes and agency type involved.

Requirement and source	What it asks for	Build consequence
CJIS Security Policy v6.1 encryption	FIPS 140-3 validated modules or AES with at least 256-bit keys for criminal justice information in transit and at rest outside physically secure locations	Select TLS and disk or database encryption that meet the validation requirement, and confirm the exact module certificates with your CJIS Systems Officer
CJIS vulnerability scanning	Vulnerability scanning at least monthly (RA-5), per the California DOJ bulletin 2026-ISRS-001	Schedule scans for the chat host, Open WebUI, model server and vector store, and keep dated scan reports as evidence
NIST SP 800-53B baselines	Low, moderate and high baselines select 149, 287 and 370 controls and enhancements respectively in Release 5.2.0	Choose the baseline with the agency's authorizing official, then map each selected control to a platform setting, owner or log source
FedRAMP scope under M-24-15	Cloud services that handle federal information on behalf of an agency fall in scope; single-agency systems not offered as a shared service may be out of scope	If the assistant runs on a commercial cloud account used by several agencies, confirm FedRAMP status of the hosting service with the agency before design freeze
32 CFR 2002.14(h)(2) for CUI	Agencies apply NIST SP 800-171 to protect controlled unclassified information on non-federal systems, unless the category's authority prescribes other safeguards	Where contractors or partner agencies paste CUI, the chat platform and its logs must meet the 800-171 requirements the agency sets
WCAG 2.1 AA under 28 CFR 35.200	State and local government web content and mobile apps, including those provided through contracts or licenses, must meet WCAG 2.1 Level A and AA from 26 April 2027 for entities serving 50,000 or more people, and from 26 April 2028 for smaller entities and special districts	Test Open WebUI or LibreChat customizations against WCAG 2.1 AA, and require the implementation partner to fix accessibility defects in its own changes
Open WebUI License branding limit	Branding may not be altered or removed in deployments above 50 users in any rolling 30-day period, unless an enterprise license or written permission applies	Budget for an enterprise license or keep original branding for pilots above 50 users; confirm terms with Open WebUI enterprise sales before a rollout wave
EU AI Act Articles 12 and 19 logging	High-risk systems must allow automatic event logging over their lifetime, and providers keep those logs for at least six months unless other law provides otherwise	Applies only where the assistant is a high-risk system in your context. Where it applies, design log capture, retention and access from the start
NIS2 scope exclusion	NIS2 covers central government entities and public administration, but excludes entities in national security, public security, defense or law enforcement	Confirm with counsel whether the agency falls in NIS2 scope, since agencies in security or law enforcement roles are excluded

Which open-source platforms fit a government staff assistant

A government staff assistant usually combines an interface, a model runtime and a gateway that controls keys and logs. The cards below cover the components that fit that stack and the licensing and access checks to complete before agency use.

Open WebUI for the staff interface

Open WebUI supports OIDC and LDAP sign-in, SCIM 2.0 user and group provisioning, role-based access control with per-resource permissions, and authenticator-app MFA, which map to agency identity groups. Its retrieval features support 13 vector databases. The licensing check matters most: the Open WebUI License bars altering or removing branding in deployments above 50 users in any rolling 30-day period unless the agency holds an enterprise license or written permission. Confirm the seat count and the agency's branding requirements before rollout.

LibreChat as an MIT-licensed alternative

LibreChat is licensed under MIT, which avoids the branding clause that applies to Open WebUI. It is a fit when an agency wants a permissive license for the interface and can staff the work of configuring authentication, file handling and multi-model access. Ask for the identity provider integration, the file upload policy and the audit output in writing, and confirm each one against the version the agency deploys, since capabilities differ between releases.

Ollama for local model serving

Ollama runs models on agency hardware and its local API has no authentication by default, so the service must sit behind a reverse proxy that enforces identity and TLS. It binds to 127.0.0.1 on port 11434 unless OLLAMA_HOST changes that, and OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE control concurrency. Set OLLAMA_NO_CLOUD=1 where cloud features must be off. Ask the partner to show the proxy configuration and the concurrency test results before any wave of users is added.

vLLM for shared GPU serving

vLLM is an Apache-2.0 inference engine that serves open-weight models behind an OpenAI-compatible endpoint, which suits a shared pool of GPUs across several agency departments. It is the option to evaluate when concurrent users exceed what a single-workstation runtime handles well. Agencies should confirm GPU sizing against expected peak concurrency, the authentication placed in front of the endpoint, and how model weights are obtained, verified and stored inside the agency boundary.

LiteLLM as the key and logging gateway

LiteLLM sits between the chat interface and model endpoints to issue keys, track cost, apply guardrails and log requests in one place. It is MIT licensed except its enterprise directory, so the agency must confirm which features it needs fall outside the MIT scope. For public sector use, the gateway is where per-department budgets, retention of request logs and separation of citizen and staff data are enforced. Ask where the logs are stored and who can read them.

Dify and Phoenix need legal review

Dify is licensed under a modified Apache License 2.0 that requires a commercial license to run a multi-tenant service from the source code, and Arize Phoenix is under the Elastic License 2.0, which prohibits providing it to third parties as a hosted or managed service. Neither is a default choice for an agency. Route both to counsel and procurement before any pilot, and confirm whether an internal, single-agency deployment triggers the restriction in each case.

Integration points and deployment pattern for agency networks

An agency assistant touches identity, databases, storage, document retrieval and hosting, and each connection is a place where data can leave the boundary or access can drift. The cards below cover the components that connect the chat platform to agency systems and the deployment choices that decide where federal or criminal justice information may sit.

Identity through OIDC or LDAP

Open WebUI documents single sign-on through OIDC and LDAP, so staff sign in with the agency directory rather than local accounts. Confirm which protocol your identity provider supports, how session lifetimes are set, and whether MFA is enforced upstream, because the assistant inherits whatever the directory allows. Test that a disabled account loses access to existing sessions and API keys.

SCIM 2.0 group provisioning

SCIM 2.0 user and group provisioning lets the directory create, update and remove accounts and groups in the chat platform automatically. Groups drive role-based access, so a mapping error can give a caseworker access to a policy team's knowledge base. Ask for the group-to-role map in writing, a review of who may create groups, and a test showing that removing someone from a directory group removes their access.

PostgreSQL instead of SQLite

The Open WebUI scaling guide requires PostgreSQL for multi-instance deployments because SQLite does not handle concurrent writes from multiple processes well. Chat history, users and settings all land in this database, so it needs the same backup, encryption and access controls as other agency records. Confirm where the database runs, who holds administrative rights, and how restores are tested.

Redis and shared storage for replicas

Multiple replicas need Redis as a shared state store for sessions, websocket connections and application state, plus shared file storage so uploads and knowledge stay reachable from every replica. Each shared service is another component inside the accreditation boundary, with its own encryption and logging. Ask where uploaded files are written, how they are encrypted at rest, and how they are deleted when a user leaves.

Vector database for internal documents

Open WebUI's retrieval features support 13 vector databases, including ChromaDB, PGVector, Qdrant, Milvus and Elasticsearch, with hybrid BM25 plus vector search and cross-encoder reranking. Choose one that your agency already operates and can patch, and keep it on agency infrastructure so embeddings derived from case files or policy documents never leave it. Confirm which databases your security team has approved before selecting one.

Model endpoint on loopback

Ollama binds to 127.0.0.1 on port 11434 by default, and its local API does not require authentication, so exposing it must be a deliberate step behind a controlled proxy. Set OLLAMA_NO_CLOUD=1 so cloud features cannot route prompts to an outside service. Verify with a network scan from an approved host that port 11434 is reachable only from the chat platform's service network.

Deployment in agency or accredited hosting

Place the platform in agency data centers, or in a cloud arrangement that meets the authorization the data requires. FedRAMP applies to cloud products that handle federal information on behalf of an agency, while a system used only for one agency's operations and not offered as a shared service is out of scope. Ask counsel and the authorizing official to classify the deployment before you pick hosting, and confirm that the CJIS Security Policy requirements for encryption and scanning apply where criminal justice information is involved.

How a public sector rollout moves from pilot to agency-wide use

The sequence turns a staff chat assistant into an agency service in controlled stages, with each stage producing a documented decision or piece of evidence before the next one starts. It covers data rules, control mapping, accessibility, licensing and user onboarding, so the assistant can grow without outrunning its governance.

Set data classification and acceptable use rules
The agency's information security owner, records management lead and business sponsor agree which data classes may be pasted into the assistant, which may be uploaded, and which must never leave the existing case or records systems. They write the acceptable use policy that staff will see at first login and define who approves exceptions. The step is done when the classification table and policy are signed off, and the assistant's configuration can enforce them, for example through disabled upload types or restricted knowledge collections. Ask the partner to show how each rule maps to a setting, not only to a policy paragraph.
Map controls to the chosen baseline
Before any pilot user logs in, the team selects the control baseline that applies to the agency's systems, such as a NIST SP 800-53B low, moderate or high baseline, and maps each selected control to the chat platform, the model server, the vector store and the hosting environment. Where a control is inherited from the hosting platform or an existing identity service, the mapping records that inheritance. The output is a control mapping workbook with named owners and open gaps. Confirm with the agency's authorizing official or compliance office which baseline and which external requirements, such as CJIS Security Policy, apply to the data in scope.
Deploy the pilot with logging and retention set
The partner deploys the interface, model server and supporting services in the agency's approved environment, with SSO enabled for a small pilot group drawn from willing teams. Logging is switched on for authentication, administrative actions and model requests, and the retention period for chats, uploads and logs is set in writing and tested by deleting a sample record. Model endpoints are reachable only from the application tier. The step is done when the pilot group can use the assistant, the audit trail shows the expected events, and the retention job is verified. Ask how log integrity and access to logs are protected, since those records may be requested during review.
Test the interface against WCAG 2.1 AA
An accessibility specialist tests the chat interface, login screens, administration pages and any agency-branded pages against WCAG 2.1 Level A and AA, using both automated scans and manual checks with keyboard navigation and a screen reader. Findings are logged with severity, owner and fix date, and retests confirm each fix. The ADA Title II rule applies WCAG 2.1 Level A and AA to state and local government web content and mobile apps, including content provided through contracts or licenses, so the agency should confirm which compliance date applies to its entity. The step closes when open defects are either fixed or documented with an agreed remediation plan.
Decide branding and license before passing 50 users
The agency reviews the Open WebUI License and decides how it will handle branding before the user count can exceed 50 end users in any rolling 30-day period. The license clause forbids altering or removing Open WebUI branding for larger deployments unless the operator holds an enterprise license or written permission, so the options are a documented enterprise license with white-labeling, written permission from the rights holder, or a branded interface that keeps the original notices. The decision, the quote scope and the monitoring method for user counts are recorded before the next wave. Procurement should request the license terms in writing and confirm them with legal counsel.
Roll out SSO group waves with monitoring
Access expands in waves defined by SSO groups in the agency directory, with each wave limited to a named set of teams and a cap that the platform team can monitor. Before each wave, the team reviews login errors, model queue depth, response times, retention job results, log volume, access review exceptions and user-reported issues against thresholds agreed in advance. A wave proceeds only when the review is signed off by the service owner and security lead. Ask the partner to show the group mapping and the dashboard used for each check, and to explain how a group member is removed quickly when their role changes.
Confirm agency-wide operations and evidence
Once the final wave is complete, the agency moves the assistant into routine operations with a named service owner, a patch and model update procedure, and recurring access reviews. The partner hands over runbooks, configuration baselines, the control mapping with evidence references and a list of residual risks accepted by the agency. The step is done when the service is handled by the agency's own operations team or a retained support provider, and the evidence pack is stored where auditors and inspectors can request it. Confirm in writing which update and support responsibilities remain with the partner under any retainer.

Evidence auditors and oversight bodies will ask the agency for

This table lists the evidence items a federal, state or criminal justice assessor is likely to request for a staff chat assistant, the component that produces each item and what the reader should confirm before the assessment. Use it to plan evidence collection from the start of the build, not at assessment time. DeployOpen prepares this evidence, but independent assessors and authorizing officials issue the determinations.

Evidence item	Component that produces it	What the assessor checks
System security plan and control mapping	Agency documentation built from the SP 800-53 baseline selected for the system	Each control is mapped to a specific configuration, owner and test result
Encryption configuration for CJIS SC-13 and SC-28	TLS termination, database and file storage settings, key management system	FIPS 140-3 validated modules or FIPS 197 AES at 256-bit keys where CJI is stored or transmitted
Vulnerability scan reports for RA-5	Scanner run against Open WebUI, Ollama or vLLM hosts, database and container images	Scans run at least monthly, with findings tracked to remediation and accepted risk recorded
Access and group membership exports	Identity provider groups, Open WebUI SCIM 2.0 provisioning and role assignments	Only authorized users hold access, and leavers are removed on the expected schedule
Chat and admin audit logs	Open WebUI and the reverse proxy, retained in the agency log platform	Administrative actions and access to conversations are attributable to named accounts
Retention settings for chats, uploads and logs	Open WebUI configuration, database retention jobs and log storage policy	Retention matches the records schedule and the agency's policy for criminal justice data
Data classification rules for pasted and uploaded content	Acceptable use policy, user banner, and any input filtering or guardrail layer	Restricted data categories are defined, communicated to users and enforced where possible
Accessibility test results against WCAG 2.1 AA	Accessibility audit of the interface, plus fixes tracked in the change record	Tests cover the interface and any custom pages, with remediation dates before the applicable deadline
Branding and license records	Open WebUI License terms, enterprise license agreement or written permission on file	Branding changes and user counts above 50 users in a rolling 30-day period are covered by a valid license
Model and component inventory with licenses	Software bill of materials and procurement records for models, inference and interface components	Each open-source component has a recorded version, license and approved source for the agency's procurement rules

What agencies ask before hiring a partner for a private LLM chat

What drives the cost of a private LLM chat for a government agency?

Cost is driven by who is paid for what, and most of it is labor and hardware rather than software. The main components are GPU servers or accredited hosting capacity for the models, the chat interface and any enterprise license, integration work with the agency's identity provider and document stores, security testing, accessibility work against WCAG 2.1 Level AA, and ongoing operations.

One-time: architecture, integration, hardening, accessibility remediation and security testing
Recurring: hardware or hosting, support, patching, model updates and staff time

Ask any partner for a quote that separates these lines and states which items depend on your classification and hosting decisions.

How long does a public sector private LLM chat project take?

Duration depends on the scope you agree, how many approvals sit in the path, and whether the hosting environment already exists. A pilot limited to one agency group with existing infrastructure moves faster than a multi-agency rollout that needs a new accreditation boundary. Approvals for data classification rules, security assessment and procurement often set the critical path more than the technical build does.

DeployOpen agrees timelines in writing as part of a fixed-scope engagement, so ask for a schedule that lists each approval gate and who owns it.

What does the agency need to provide before a partner starts?

The agency has to supply the decisions and access that only it can give. The partner can build the platform, but cannot define what staff may paste into it.

Data classification rules that say which categories of information may be entered, uploaded or retained
Identity provider access for OIDC or LDAP configuration, group mapping and SCIM provisioning
Named approvers for hosting location, security assessment and accessibility sign-off
Infrastructure access in the agency's own cloud account, data center or enclave

Missing classification rules are the most common reason a pilot stalls before any model is installed.

Can our in-house staff run a private LLM chat after launch?

Yes, if the agency has people who can operate Docker or Kubernetes, manage PostgreSQL and Redis, patch the chat interface and inference servers, and watch logs and capacity. Open WebUI runs on Docker Compose, Kubernetes with Helm, or pip, but a multi-replica deployment needs PostgreSQL, Redis and shared file storage, which adds operational work. Many agencies split the work: internal staff own day-to-day operations and access control, while a partner handles upgrades, hardening and support under a separate retainer.

Ask the partner which tasks they would hand over, and what runbooks and documentation you would receive at handover.

When does an agency need an Open WebUI license?

The Open WebUI License requires a license only if you want to alter or remove Open WebUI branding in a deployment that exceeds 50 users in any rolling 30-day period. Below that threshold, the branding clause does not apply under the license terms. Above it, you can either keep the branding, obtain written permission from the contributors, or buy an enterprise license that allows white-labeling and custom branding. Enterprise pricing is not published, so the agency should request a quote with its expected seat count.

Read the Open WebUI License with your legal team before the user count is set, because the threshold is measured on a rolling basis.

How do FedRAMP and CJIS requirements change the work?

They change where the system may run, which controls must be evidenced and how much testing is needed. FedRAMP scope covers cloud products that create, collect, process, store or maintain federal information on behalf of an agency, so a hosted assistant that handles federal data may fall into scope, while a system used only for one agency's own operations on cloud infrastructure may not. The FBI CJIS Security Policy version 6.1 took effect on 26 June 2026 and requires FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys for criminal justice information in transit and at rest outside physically secure locations, plus vulnerability scanning at least monthly.

Confirm your scope with the agency's security office before design starts, because it determines the hosting model.

Does the chat interface need to meet accessibility standards?

Yes, for state and local government entities covered by the ADA Title II web rule, web content and mobile apps must meet WCAG 2.1 Level A and AA, including content provided through contracts or licenses. The compliance date is 26 April 2027 for entities serving 50,000 or more people and 26 April 2028 for smaller entities and special districts, according to 28 CFR 35.200. Accessibility testing should cover the chat interface, administration screens and any file upload flows, and it adds effort to the build.

Ask partners how they test with assistive technology and whether accessibility work is in the written scope.

What security testing should an agency expect before go-live?

Expect testing of the application, its APIs, the infrastructure and the AI-specific attack paths, including prompt injection and data leakage through uploaded files. DeployOpen runs application and API testing, cloud and infrastructure testing, AI system and agent testing and red teaming, and it prepares systems, controls and evidence for compliance assessments. It does not issue audit reports, attestations or certifications, so the agency still needs its assessor or certification body for those decisions.

Ask for the test scope in writing, the findings format and how retests are handled.

Questions to ask an implementation partner before a public sector contract

Start with the control baseline, because it decides what the partner must build and document. Ask how they map each component of the assistant (the Open WebUI or LibreChat front end, the model server, the database, the file store and the logs) to controls in NIST SP 800-53B, and which baseline they assume: the low, moderate and high baselines select 149, 287 and 370 controls and enhancements respectively. Ask for the mapping in the spreadsheet or OSCAL form they will hand over, not a summary. Then ask whether the scope includes the controls your authorizing official will inherit from the hosting environment and which ones the partner will document as their own.

Next, press on the CJIS and scope questions if any criminal justice information could reach the chat. Ask how the partner meets the FBI CJIS Security Policy version 6.1 requirements for encryption, which now call for FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys for data in transit and at rest outside physically secure locations. Ask how they configure TLS between the browser, Open WebUI and the model endpoints, where keys are held, and how vulnerability scanning at least monthly is scheduled and evidenced. Also ask whether they have delivered inside a FedRAMP or CJIS boundary before, and how they decide whether a component is in scope at all, since a system used only for one agency's operations on agency infrastructure may fall outside FedRAMP scope under the M-24-15 scope rules.

Finish with accessibility, branding and release control. Ask how WCAG 2.1 Level A and AA conformance is tested for the interface, including keyboard navigation, screen reader behavior with streaming responses and the admin screens, and whether they run automated scans and manual testing with assistive technology. For branding, ask whether the agency will operate above 50 users in any rolling 30-day period, which triggers the Open WebUI License branding clause, and how the partner will document the enterprise license, written permission or other exception that applies. Last, ask how logs are retained for the period your records policy requires, and how model and Open WebUI releases (the project published v0.12.3 on October 16, 2026) are tested in a staging copy and approved by your change board before they reach users.

How DeployOpen scopes and delivers a public sector assistant

DeployOpen plans, implements, hardens, documents and supports self-hosted open-source assistants inside the agency's own infrastructure, whether that is a cloud account, a data center or an air-gapped network. The work starts with a written scope that sets which data classes the assistant may handle, which users and groups it serves, and which evidence the agency needs for its assessments.

How we scope

Written scope fixes the user groups, expected concurrency and the data classes allowed in prompts and uploads.
We record which hosting location, operator and sovereignty conditions the agency must meet before go-live.
Branding and license terms for Open WebUI are confirmed up front, including whether the user count exceeds 50.
Engagements are fixed-scope, with deliverables and timelines agreed in writing before work begins.

How we build

We deploy Open WebUI or LibreChat with Ollama or vLLM inside the agency's environment, with SSO, groups and role-based access configured.
Model endpoints are kept off public networks, and cloud features are disabled where the agency's policy requires it.
Accessibility of the interface is checked against WCAG 2.1 Level AA as part of acceptance testing.
Security testing covers the application, its APIs and prompt injection exposure, with findings tracked to closure.

How we hand over

Deliverables include architecture documentation, configuration records, access model and runbooks for the agency's operators.
Chat retention, logging and audit settings are documented so the agency can show how records are kept.
Compliance readiness material maps the system to the agency's chosen control framework, without issuing any audit opinion.
Knowledge transfer sessions train the agency's administrators on user provisioning, model updates and backups.

How we support

Ongoing support, upgrades and continuous security testing are available as retainers after handover.
Upgrades to Open WebUI, Ollama or vLLM are tested in a staging copy before they reach staff.
Backups and restore tests cover chat history, uploads and configuration on the agency's own storage.
Support scope, response expectations and change control are defined in the written retainer agreement.