Why a RAG knowledge assistant for the public sector needs a different design
Caseworkers and policy staff in agencies spend long periods searching large, frequently amended bodies of legislation, guidance and case procedures, and a self-hosted RAG assistant with citations is feasible, but the jurisdiction, data classification and records rules that apply to the agency shape the platform choice more than the chat features do. In the short answer, Open WebUI or RAGFlow paired with a self-hosted vector store are the usual starting points, with Open WebUI's retrieval support for vector databases such as Qdrant and PGVector and RAGFlow's Apache-2.0 license being the main reasons to shortlist them. Dify is harder to justify for a public body that might later offer the assistant to other agencies or contractors, because its modified Apache License 2.0 requires a commercial license to run a multi-tenant service from the source code. Open WebUI carries its own constraint: its license restricts altering or removing branding in deployments above 50 users in any rolling 30-day period unless an enterprise license or written permission applies, so a department-wide rollout may need that decision made early.
Before any platform is chosen, the agency must confirm which regime applies. A federal or state body may fall under the NIST SP 800-53 control baselines through FISMA, a criminal justice body may need to meet the FBI CJIS Security Policy in its current version, and a body that handles controlled unclassified information may need to apply NIST SP 800-171 to non-federal systems. An EU public body may fall under NIS2 or its national transposition, and any public web content the assistant exposes may need to meet WCAG 2.1 Level A and AA under the ADA Title II rule. Each regime changes the hosting location, encryption requirements, logging depth and who can see which documents, so the assumptions that work for a general enterprise pilot often do not carry over unchanged. Ask the legal, records and security owners to write down the applicable regimes, the highest classification of the source documents, and the retention schedule that governs the answers and logs before the solution architecture is drafted.
Which public sector requirements shape the build
This table maps each public sector requirement to its source, the evidence an assessor expects and the build decision it forces on a RAG assistant. Read the applicability column first, because several items apply only in certain jurisdictions or agency types, and confirm them with your legal and compliance owners before design starts.
| Requirement | What it asks for and where it applies | Build implication |
|---|---|---|
| NIST SP 800-53 Rev. 5 and SP 800-53B baselines | US federal control catalog (Release 5.2.0 current). SP 800-53B baselines select 149 (low), 287 (moderate) and 370 (high) controls and enhancements. Applies to US federal systems and often becomes the reference for state and local programs. | Confirm the impact level of the corpus and the assistant with your authorizing official, then map each control to a component or procedure before build starts. |
| FISMA and Inspector General assessment | FISMA, listed by NIST as a law behind SP 800-53, requires agencies to protect their systems. Inspectors General independently assess agency security and report to OMB. Applies to US federal agencies. | Keep assessor-ready evidence: system boundary, configurations, access reviews and logs, because an independent reviewer may test them directly. |
| FedRAMP scope under OMB M-24-15 | Covers cloud products and services that create, collect, process, store or maintain federal information for an agency. A system used only for one agency's operations on cloud infrastructure, not offered as a shared service, is out of scope. Applies to US federal use of cloud services. | Decide whether the hosted vector store, model endpoint or hosting provider falls in scope. Confirm the scoping position with the agency's FedRAMP or security office. |
| CJIS Security Policy v6.1 encryption and scanning | For criminal justice information, v6.1 (effective 26 June 2026) requires FIPS 140-3 certified modules or FIPS 197 AES with at least 256-bit keys in transit and at rest outside physically secure locations, plus vulnerability scanning at least monthly. Applies only where CJI is in scope, typically law enforcement and courts. | Keep case files and criminal justice content out of general indexes unless required. Specify validated cryptographic modules for the vector store, object storage and backups, and schedule monthly scans. |
| 32 CFR 2002.14(h)(2) and NIST SP 800-171 for CUI | Federal agencies using non-federal systems for controlled unclassified information must apply NIST SP 800-171 requirements, unless the CUI category sets other safeguards. Applies to US federal agencies and their contractors handling CUI. | If CUI is indexed, the self-hosted stack, admin access and logs must meet SP 800-171 requirements. Confirm which CUI categories exist in the corpus before ingestion. |
| ADA Title II web and app accessibility (WCAG 2.1 AA) | State and local government web content and mobile apps, including contracted content, must meet WCAG 2.1 Level A and AA from 26 April 2027 for entities serving 50,000 or more people, and from 26 April 2028 for smaller entities and special districts (28 CFR 35.200). Applies to US state and local government. | Treat the chat interface and citation display as public-facing web content. Confirm your population category and test keyboard use, screen reader output and citation links before the applicable date. |
| NIS2 scope for public administration (EU) | NIS2 covers central government entities regardless of size and lists public administration as a high-criticality sector. Member States may extend it to local administration. It does not apply to entities in national security, public security, defense or law enforcement. Applies to EU public bodies as set by national transposition. | Confirm whether the agency is in scope under its national law. If it is, document risk management, incident handling and supplier security for the self-hosted stack. |
| EU AI Act Article 12(1) event logging | High-risk AI systems must technically allow automatic recording of events over their lifetime. Applies only if the assistant falls within a high-risk use under the Act, so classification must be confirmed. | Design the assistant to log queries, retrieved sources, model versions and outputs from the start. Confirm the classification with legal counsel. |
| EU AI Act Article 19(1) log retention | Providers of high-risk systems must keep the logs for at least six months unless other law provides otherwise. Applies alongside Article 12 when the Act applies. | Set retention for logs to meet the longer of the AI Act period or the agency's records schedule. Records retention rules often exceed six months, so confirm the period with the records officer. |
What data and systems the assistant touches in government
The corpus for a public sector assistant usually mixes statutes and regulations, agency guidance, case procedures, internal manuals and records that fall under retention schedules. Each class carries different handling rules, so the first decision is document classification before anything is indexed. Classification should assign each file a sensitivity label, an owner, a records series and a permitted audience, and it should be recorded in the document management system rather than in a spreadsheet kept by the project team. Restricted manuals, investigative case files and anything marked as controlled unclassified information should stay out of the shared collection entirely. Keep them in separate collections with their own access groups, or leave them out of scope for the first release. Confirm with your records officer and security office which markings apply to your agency, since the marking scheme differs between jurisdictions and between federal, state and local bodies.
Records retention rules reach into the assistant in three places that teams often overlook. Source documents follow the retention schedule, so when a record is destroyed, its chunks and embeddings must be removed from the vector store too. Chunks and embeddings are derived copies, so they need their own deletion job tied to the source identifier. Chat logs and the retrieved passages stored with them are records in their own right if the agency's policy treats assistant interactions as official communications. Ask your records office how long those logs must be kept and whether they can be purged on schedule. Connectors to document management systems should pass the source identifier, version and access control list into every chunk, so that the retrieval layer can filter by the user's existing permissions. Do not rely on the assistant's own group model to replace them. Test this with a user who lacks access to a restricted folder and confirm that neither the answer nor the citation exposes its content.
Sovereign hosting options for models and vector stores
In an agency-owned data center, the agency controls the hardware, the hypervisors, the network and the staff with access to the stack. Models usually run on Ollama for smaller open-weight models or on vLLM where the agency has GPU capacity and needs higher concurrency. The vector store is typically Qdrant or PGVector on the same internal network, with application traffic kept inside segmented zones. Because no third party hosts the service, a FedRAMP question does not arise for this deployment. Confidentiality still depends on the agency's own controls, so the design must show them.
- Model serving: Run Ollama or vLLM behind an internal OpenAI-compatible endpoint, reachable only from the assistant and the gateway.
- Vector storage: Place Qdrant on block storage with a POSIX-compatible file system, on SSD or NVMe where possible. Do not mount NFS or object storage under Qdrant.
- Cluster sizing: Qdrant's distributed docs recommend at least three nodes for production so quorum survives a node loss. Ask the partner how that maps to the agency's hypervisor and storage design.
- Secure defaults: Qdrant instances listen on all interfaces with no authentication by default, so bind them to internal addresses, enable API keys and TLS, and enable JWT collection permissions and audit logging.
- Backups: Plan snapshot and restore for vector collections alongside the document store, and test restores on a separate host.
Confirm with the agency's infrastructure team which hardware, storage tier and backup platform the data center already supports before any model or vector size is chosen.
A government cloud account places the stack in a cloud tenancy operated by the agency or a provider. The hosting decision is now shaped by the provider's authorization status and by who holds the federal information. FedRAMP scope covers cloud products and services that create, collect, process, store or maintain federal information on behalf of an agency. Systems used only for a single agency's operations on cloud infrastructure, not offered as a shared service, fall outside that scope, so the agency must determine which category its deployment falls into. The agency must verify any provider authorization itself and record the baseline it relies on.
- Authorization check: Where the cloud service holds federal information, confirm the provider's FedRAMP authorization and baseline against the current FedRAMP listings before the design is approved.
- Model placement: Run Ollama or vLLM on GPU instances inside the tenancy, and keep prompts, files and logs inside the account's storage.
- Vector storage: Run self-managed Qdrant on block volumes, not on the cloud's object storage or network file share, since Qdrant cannot run on either.
- Managed Qdrant: Qdrant Cloud is a separate service with its own authorization position. Confirm that status before it is considered for any federal information.
- Network controls: Use private subnets, restricted egress and customer-managed keys, and document which provider services touch assistant data.
Ask the partner to produce the authorization evidence for each cloud service in the data path and to flag any service that falls outside the agency's chosen authorization boundary.
An air-gapped network has no route to the internet or to external model providers. Every model weight, container image, package and vector update must enter through a controlled transfer process. The assistant then runs fully inside the isolated zone, which suits agencies handling the most sensitive reference material. Ingestion, embedding, vector search and generation all run on local hardware, so model choice and index sizing must be settled before the network is sealed.
- Local models: Serve open-weight chat and embedding models with Ollama or vLLM from internal registries, and verify each artifact before import.
- Vector store: Run Qdrant or PGVector on local block storage. Size the disks for the index and for snapshot copies, because replication multiplies storage at a factor of 2.
- Update path: Define a one-way transfer procedure for new documents, model versions and patches, with hashing, approval and a record of each transfer.
- Identity and access: Use local identity services and internal certificates, since external SSO endpoints will not be reachable.
- Operating model: Plan for offline support, spare parts and staff who can rebuild the stack without internet access.
Confirm with the agency's security team which transfer and approval steps are allowed, and whether the network's accreditation covers the chosen components.
Which open-source platforms fit a public sector RAG assistant
A public sector assistant has to answer from legislation and procedures while respecting access rules, keeping records and staying under the agency's control. The platforms below cover the interface, document parsing, the model gateway, observability and the vector store, and each one brings a licensing or security condition that changes what an agency can deploy and how.
Open WebUI for staff access
Open WebUI provides the chat interface, with OIDC, LDAP and SCIM 2.0 provisioning, role-based groups and API keys documented for it. Its retrieval features support 13 vector databases, including Qdrant, with hybrid search and reranking. The branding clause of the Open WebUI License applies to deployments above 50 users in any rolling 30-day period unless the agency holds an enterprise license or written permission. Confirm that clause with the licensing team before rollout, and ask the vendor what the enterprise license covers for white-labeling.
RAGFlow for document parsing
RAGFlow is an Apache-2.0 RAG engine, and its document parsing matters most for legislation, scanned forms and case files with tables and numbered clauses. Parsing quality decides whether a citation points to the right article, so test it on a sample of the agency's own documents before choosing it. Check how it preserves document metadata such as effective dates and amendment status, since these determine which version of a law an answer may cite.
AnythingLLM for single-team pilots
AnythingLLM is MIT licensed and suits a single department testing its own guidance collection before a wider rollout. Its workspace model keeps one team's documents separate from others, which is simpler than building that separation by hand. It is a weaker fit when many departments need group-based permissions that mirror the agency's directory structure, so check its access controls against the identity model the agency will require.
Dify under license review
Dify is licensed under a modified Apache License 2.0 that requires a commercial license to run a multi-tenant service from the source code. An agency's internal assistant serving its own staff is a different case from a shared service for several agencies or a public portal, so the license terms must be read against the actual deployment model. Dify is a candidate for agency-internal use only after legal review confirms the terms apply as intended.
LiteLLM and Phoenix for gateway and tracing
LiteLLM sits between the assistant and model endpoints, providing routing, logging and guardrail hooks in one place that the agency controls. Its enterprise directory uses different terms, so confirm which features the agency needs fall outside it. Phoenix, for tracing, is under the Elastic License 2.0, which bars offering it to third parties as a hosted or managed service. An agency running it for its own staff is generally not offering a managed service, but confirm this with counsel, and use Langfuse as the alternative if the terms create doubt.
Qdrant as the self-hosted vector store
Self-hosted Qdrant is not secure by default: it listens on all interfaces with no authentication, so the agency must enable API keys, TLS and JWT-based per-collection permissions before loading records. Since v1.17.0 it logs authenticated API operations in JSON, which supports the audit trail. The Helm chart is community supported and lacks enterprise features such as zero-downtime upgrades and automatic shard rebalancing, so confirm which production path the agency needs. Qdrant also requires block storage, so NFS and S3 cannot host its data.
How a public agency rolls out the assistant in phases
The sequence moves from scoping the corpus and security baseline to a tested pilot, a measured citation review, accessibility checks and a staged rollout, so each phase closes with evidence the agency can keep. Durations depend on scope and are set in a written plan agreed before work starts.
Classify and scope the corpus with records officers
Records officers and content owners list the legislation, guidance and case procedures that the assistant may index, and mark each source by sensitivity, owner, retention class and current version. Anything containing case-specific personal data or criminal justice information is excluded or handled in a separate, later decision. The output is a signed corpus register that states what is in scope, what is excluded and why. The step is done when every indexed source has a named owner and a retention rule.
Set the security categorization and baseline
The agency's security team and DeployOpen review the system's impact level and select a control baseline from NIST SP 800-53B, or the baseline the agency's own authority requires, then document how the assistant's components inherit or implement those controls. Hosting, key management and log retention are decided here, before any model or vector store is deployed. The output is a categorization record and a control mapping that the agency's authorizing official can review. The step is done when each open control has an owner and a planned implementation.
Build and pilot on public guidance
The team deploys the chosen stack inside the agency's approved environment, ingests only public guidance and legislation, and runs permission tests that confirm the assistant cannot return content from restricted collections to users outside those groups. Test accounts for each role are used to attempt retrieval of restricted items, and every failed attempt is logged as a defect. The output is a pilot environment and a test report listing pass and fail results. The step is done when no restricted test document appears in any answer or citation.
Evaluate citation accuracy against a reviewed question set
Subject matter experts from the agency write a question set drawn from real service questions, and a second reviewer approves it before testing begins. The team runs each question, then checks whether each answer is supported by the cited passage and whether the cited passage is the correct, current version. Results are scored and failures are traced to chunking, retrieval, reranking or prompt causes. The output is an evaluation report with error categories and agreed release thresholds set by the agency. The step is done when the agency accepts the report and the thresholds in writing.
Test accessibility against WCAG 2.1 AA
Because the ADA Title II rule applies WCAG 2.1 Level A and AA to state and local government web content and to content provided through contracts, the interface is tested before any staff or public use. Testers check keyboard operation, screen reader output for answers and citations, color contrast and error messages, and the team fixes defects in the front end or configuration. The output is an accessibility test record with each finding and its resolution. The step is done when open defects are either fixed or recorded with a remediation date in the agency's plan.
Roll out in stages to case teams
The assistant is opened first to a small group of case team staff chosen by the business owner, with groups mapped from the agency's identity provider and access limited to the collections those teams need. Usage, citation feedback and any incorrect answers are reviewed weekly, and access widens only after the owner signs off on the results. The output is a rollout record showing each group admitted, the evidence reviewed and the decision taken. The step is done when the agency confirms that operating procedures, user guidance and escalation paths are in place.
Re-index and review on a fixed cycle
After go-live, the team re-indexes changed legislation and guidance on a schedule the records owner sets, removes superseded versions from retrieval, and reruns the reviewed question set after every material change. Logs are kept under the agency's retention rules, and access reviews repeat on the cycle the security team defines. The output is a maintenance runbook and a dated change log that shows which source version each answer used. The step is done when the agency can show, for any answer, the source version that was retrieved at the time.
Evidence auditors and oversight bodies ask for
This table maps each piece of evidence that assessors and oversight bodies commonly request to the component that produces it in a self-hosted RAG assistant. Read it as a checklist for design decisions: if a component cannot produce its row, the evidence has to come from another control or be built deliberately. DeployOpen prepares this evidence, but auditors and certification bodies are the ones who issue any opinion or certification.
| Evidence | Component that produces it | What the reader should confirm |
|---|---|---|
| Access logs and authentication records | Open WebUI authentication and audit output, plus Qdrant JSON audit logs of authenticated API operations (v1.17.0 and later) | Log retention period, central log shipping, and whether Open WebUI logs cover every chat and knowledge action needed |
| Collection-level access control evidence | Qdrant JWT permissions per collection (v1.9.0 and later) and Open WebUI role and group settings | Export of role assignments and a sample test showing a user without rights cannot retrieve restricted passages |
| Model and corpus version records | Model registry entries, embedding model identifiers, and document source snapshots recorded at each indexing run | Whether each answer can be traced to the model build and the document versions active at that time |
| Citation and answer evaluation results | Retrieval test sets, citation accuracy checks and answer scoring kept with the release record | Acceptance thresholds agreed with the agency and a dated record of each evaluation run |
| Records retention settings | Configured retention for prompts, chat history, uploaded files and indexed content in Open WebUI, the database and object storage | Alignment with the agency's records schedule and whether deletion removes vectors as well as source text |
| Encryption configuration evidence | TLS settings for REST and gRPC in Qdrant, database and storage encryption, and key management records | Whether modules and algorithms match the applicable CJIS Security Policy or agency requirement, such as FIPS 140-3 validated modules |
| Vulnerability scan reports | Scheduled scans of the Open WebUI, Qdrant, database and host estate with tracked remediation | Scan frequency required by the authorizing body, such as monthly scanning under the CJIS Security Policy |
| WCAG test results | Accessibility test reports for the Open WebUI front end and any agency-built interface | Target level, WCAG 2.1 Level AA as referenced by the ADA Title II rule, and the compliance date that applies to the agency |
| EU AI Act Article 12 and 19 logs | Automatic event logs covering the system lifetime, kept for at least six months under Article 19(1) | Whether the agency is a provider or deployer, and whether other law requires longer retention |
What agencies ask before hiring an implementation partner
What drives the cost of a public sector RAG assistant?
Cost follows the scope of the document estate and the authorization work, more than the software itself, which is open source. The main drivers are corpus size and the number of source systems, how much classification and records-retention mapping is needed before anything is indexed, how permissions from the agency's identity system are carried into retrieval, the evaluation work to test answers against agency-approved questions, and ongoing support. Ask each partner for a written scope that lists these items separately, which costs are one-time (build, integration, evaluation baseline) and which recur (hosting, model compute, upgrades, support retainers), and what changes if the corpus grows. DeployOpen does not publish prices and quotes from a written scope.
Can our agency build a RAG assistant in-house?
Many agencies can, provided they have engineers who run containers and databases, security staff who can review the configuration, and records and legal staff who can define the rules for what is indexed. The harder parts are rarely the chat interface. They are permission-aware retrieval, parsing scanned and legacy documents, re-indexing when a procedure changes, and building an evaluation set that records officers accept. Confirm your internal skills in each area before deciding. A partner can build the platform, document it for your staff and hand over runbooks, or run it under a support retainer, and the written scope should state which of these applies.
How long does a public sector RAG rollout take?
The timeline depends on the sources to be connected, the approval path for the document corpus, and how many authorization reviews the agency requires before go-live. Agencies that have already classified their documents and defined access groups usually move faster than those that must do both during the project. Ask a partner for a phased plan with a pilot group, a defined evaluation gate and a production date that depends on the agency's own review cycles. Confirm whether your security or accreditation office must approve the platform, the model host and the vector store as separate components, since each can add time.
What do records officers, security staff and legal reviewers have to do?
Each group owns a decision that the technical team cannot make for them. Records officers define which documents are records, how long they are retained and which versions the assistant may cite. Security staff review the hosting boundary, access groups, logging and key management. Legal reviewers confirm how answers are presented, what disclaimers apply and whether any corpus contains material with special handling rules. Expect each group to review samples, sign off on the indexing rules and attend evaluation sessions. Ask the partner to name these reviews in the plan so that staff time is scheduled before the pilot starts, not discovered during it.
Does the Open WebUI license apply to our agency's user count?
The Open WebUI License keeps the BSD-style terms but adds a branding clause in clause 4, which prohibits altering or removing Open WebUI branding except in listed cases. Per the supplied license documentation, one exception covers deployments with no more than 50 end users in any rolling 30-day period. Deployments above that count must either keep the branding, obtain written permission, or hold an enterprise license that allows white-labeling. Open WebUI does not publish enterprise pricing, so ask its sales team for a quote based on your expected seat count, and have your counsel confirm how the clause applies to your deployment model before you commit.
Can a RAG assistant meet CJIS Security Policy or CUI requirements?
The assistant can be built to support those requirements, but it does not meet them by default, and no open-source component certifies compliance. For criminal justice information, the FBI CJIS Security Policy version 6.1 requires FIPS 140-3 validated modules or FIPS 197 AES with at least 256-bit keys for CJI in transit and at rest outside physically secure locations, plus vulnerability scanning at least monthly. For CUI on non-federal systems, agencies apply NIST SP 800-171 under 32 CFR 2002.14. If a cloud service is involved, FedRAMP scope applies. Confirm with your CJIS Systems Officer or security office which controls apply, then ask the partner to map each control to a configuration and evidence item.
Which Qdrant settings matter most for a public sector deployment?
Start with the default state: self-deployed Qdrant listens on all network interfaces and has no authentication configured, so the partner must enable API keys, TLS and access control before any document is loaded. Per the supplied Qdrant security documentation, admin and read-only API keys, TLS for REST and gRPC, JWT-based per-collection permissions and JSON audit logging of authenticated operations are available in the open-source engine. Qdrant also needs block storage and cannot run on NFS or S3. For production, the distributed docs recommend at least three nodes so quorum survives a node failure. Ask the partner to show these settings in your environment.
What to ask a partner before a public sector RAG build
Start with the index, because that is where restricted material gets in. Ask the partner how classified and restricted documents are kept out of the vector store before ingestion, not filtered at answer time. The answer should name a classification check in the ingestion pipeline, a documented rule for which collections agency staff can reach, and a test that proves a restricted file never produces a chunk. Ask who signs off the collection list and how a document's classification changes after it is indexed. For the Qdrant store, ask how the partner sets API keys, read-only and admin roles, and TLS on REST and gRPC, and how they confirm the instance does not listen on open interfaces, since the Qdrant security documentation states that self-deployed instances are not secure by default. Ask whether per-collection JWT permissions and JSON audit logging are enabled, and where those logs are stored.
Next, ask how citation accuracy is tested rather than assumed. A credible partner will describe a fixed set of questions drawn from the agency's own legislation, guidance and case procedures, with expected source passages, and will score whether each cited passage actually supports the answer. Ask how prompt injection from documents is handled, including instructions hidden inside uploaded files, and whether retrieved text is treated as data and never as instructions to the model. Ask which rules the team applies to Open WebUI license compliance. The Open WebUI License restricts altering or removing branding for deployments above 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written permission, so the partner should show the user count, the branding decision and the written position for the agency. Finally, ask for records retention and erasure procedures covering source files, chunks and embeddings, and for the evidence pack they hand over at the end, such as configuration exports, test results, access reviews and log samples, with a plain statement of what the evidence does not prove. Independent auditors make the assessment, so the partner should only prepare that evidence.
