RAG assistant requirements checklist for buyers

How to use this RAG knowledge assistant requirements checklist

Use this checklist to write the requirements for a RAG knowledge assistant before you choose between RAGFlow, AnythingLLM, Dify or Open WebUI. Each requirement below answers one question: what the assistant must do, for whom, and on which documents. Every requirement carries a verification step, a check you can run during evaluation or pilot to confirm the requirement is met, so the list doubles as an acceptance test. Work through it in order, record a decision or an open question against each item, and turn the result into a written scope that a partner can price against.

The platforms named here are open source and run in your own infrastructure, so the requirements also decide where documents, embeddings, prompts and logs live. Expect to resolve questions such as which connectors you need, whether answers must respect source-document permissions, which vector database holds the embeddings, and how often the index is refreshed. Note the licensing questions early as well. Open WebUI and Dify carry license terms that depend on user count and tenancy, so the checklist should record expected seats and whether the assistant will serve customers or only staff.

DeployOpen plans, implements, integrates and tests self-hosted platforms in your environment, including retrieval quality and security testing of the assistant. It does not issue audit reports, attestations or certifications. Where a requirement touches a compliance framework, the checklist produces the systems, controls and evidence that an independent auditor or certification body would review, and your compliance lead should confirm which framework applies and what evidence it expects.

Functional requirements a RAG knowledge assistant must meet

This table lists the retrieval requirements that matter most for a document-grounded assistant, why each one matters, and how to test it in a proof of concept. Use it to turn a requirements list into a test set with known source passages before choosing a platform or partner. The main takeaway is that answer quality can only be judged against questions whose correct source passages you already know.

Requirement	Why it matters	How to verify in a proof of concept
Document sources and connectors	Answers are only as complete as the corpus. Each source system (file shares, wikis, document management, ticketing) needs a connector that keeps document identity, version and modified dates.	Connect each source in scope and confirm that a sample of files arrives with stable IDs, titles, owners and timestamps. Confirm which connectors the chosen platform provides and which need custom work.
File types and scanned content	Policies, manuals and contracts often sit in PDFs, slide decks, spreadsheets and scanned images. Text that the parser cannot read will never be retrieved.	Build a test set that includes scanned pages, tables and image-heavy slides. Check whether each file type yields readable text, and record which documents need OCR or manual handling.
Chunking and citation needs	Chunk boundaries decide whether an answer finds the right clause. Citations must point to a document, section and page so a reviewer can check the claim.	For each test question, check whether the retrieved chunk contains the known passage and whether the citation links to the correct document version and page. Test tables and long clauses separately.
Model choice and data handling	The embedding model sets retrieval quality, and the generation model sets how faithfully answers follow retrieved text. Both must run where the organization allows its data to go.	Compare candidate embedding and generation models on the same test set. Confirm where each model runs, whether prompts or documents leave the environment, and which licenses apply to the models and the platform.
Answer quality evaluation	Without a fixed test set, teams cannot tell whether an upgrade, a new model or a re-index made answers better or worse.	Write questions with known source passages and expected answers. Score retrieval (correct passage found) and generation (answer supported by the passage, citation correct). Repeat the run after every change.
Refresh frequency and re-indexing	Stale policies are a common source of wrong answers. Deleted or superseded documents must also leave the index, or the assistant may cite withdrawn guidance.	Edit, add and delete a set of test documents at the source. Measure how long each change takes to appear in answers, and confirm that superseded versions and deleted files no longer return.

Why source permissions must carry into every answer

A RAG assistant can leak content that a user could never open in the source system if retrieval ignores source access control. The language model only sees the chunks the retriever returns, so a chunk from an HR investigation or a restricted deal folder becomes readable the moment it ranks highly for a question, even if the asking user has no rights to the original file. The requirement is that every chunk keeps the access control list of its source document (or a reference to it), that the list is refreshed when the source changes, and that retrieval filters candidates by the asking user's identity before anything reaches the prompt. Filtering only in the user interface, or only when the answer is displayed, does not meet this requirement because the model has already read the restricted text.

Ask each candidate platform how it stores and enforces these rules. Open WebUI documents groups, role-based access control and per-resource permissions, and its knowledge features are the place to test whether a collection-level grant also constrains which chunks come back for a given user. Qdrant offers admin and read-only API keys, and JWT-based access control with per-collection read or write permissions, which can scope what a service account may query, but it does not map individual documents to source ACLs on its own. In either case, the application or a filtering layer must put the user's identity into the query. Dify's workspace and app permissions govern who can open an app, which is a different control from who can retrieve which document. Confirm with each vendor, in writing and against your own configuration, how a query is restricted at chunk level.

Test this before any pilot content is indexed. An illustrative test: create two test users with different source permissions, index one restricted document with a distinctive phrase, then ask questions that should surface it for the unauthorized user and check the answer, the citations and the retrieval logs. Repeat after a permission change in the source system to confirm the index picks it up within your agreed refresh window. Record the results, since they become evidence for access control reviews. A partner should propose this test as a deliverable, with the permission mapping documented per source system.

Security requirements for the assistant, index and model path

A RAG assistant touches the vector index, the application layer, the model path and the source documents, and each one needs its own controls. The cards below name the checks a buyer should verify before any document is indexed, so that access, keys, network paths and logs hold up under review.

Secure defaults for the vector database

Self-deployed Qdrant is not secure by default: it listens on all network interfaces and has no authentication configured. Verify that the index is reachable only from the application's network segment, that admin and read-only API keys are set, and that TLS is enabled for REST and gRPC. Confirm that per-collection read and write permissions are assigned to each service account, so the ingestion job cannot be used to query the index and the chat service cannot write to it.

SSO and MFA for every user path

Staff reach the assistant through a web interface, admins reach the platform console, and both must sit behind the same identity provider with multi-factor authentication. Check which protocols the chosen platform supports, such as OIDC, LDAP or SAML, and whether user and group provisioning uses SCIM 2.0 so that offboarding removes access to collections as well as to the chat. Confirm whether SSO scopes can be set separately for workspace access and for published web apps.

Encryption keys and secret management

Application secrets, session signing keys and model provider credentials must be generated per environment and stored outside container images. Verify where the platform keeps its signing key, how workspace encryption keys are stored (local storage or a key vault), and what happens if the key is rotated, since some platforms log every user out when the secret changes. Ask who holds the backup of the key material and how it is tested during restore.

Network isolation for outbound requests

A RAG assistant makes outbound calls when it fetches web content, calls connectors or reaches a hosted model API. Verify that outbound traffic from the code sandbox and tool calls passes through a proxy that blocks private and internal address ranges, which limits server-side request forgery. Confirm that the model endpoint is either inside the network or reached only through an approved egress path, and that each egress route is listed in the architecture document.

Audit logging of queries and retrievals

Record who asked which question, which collections were searched, which chunks were retrieved and which sources were cited, along with the model version used. Verify that logs are written to storage the platform team cannot silently edit, and that the retention period matches the records policy. Where the assistant may be classed as high-risk, confirm that the logging design meets the automatic event recording and six-month retention duties in the EU AI Act.

Guardrails against injection and poisoned documents

Documents can carry instructions meant to override the assistant, and an uploaded file can plant false facts that are later cited as authoritative. Verify that ingestion restricts who can add documents to each collection, that uploaded content is scanned for embedded instructions and hidden text, and that retrieved text is passed to the model as data, not as system instructions. Confirm that an output check flags answers with no supporting citation, and that a named owner approves changes to high-impact collections.

Choosing the vector database, model and platform for your constraints

The vector store holds embeddings derived from your documents, so it inherits their sensitivity and must be inside the same trust boundary as the source files. Three paths are common. Self-hosted Qdrant is a dedicated engine with API keys, per-collection JWT permissions and JSON audit logging of authenticated operations. pgvector runs inside PostgreSQL, which many teams already back up, monitor and patch. The built-in default of a platform, such as Weaviate in Dify, is the fastest to start but ties you to that platform's operating model.

Qdrant security defaults: self-deployed instances listen on all interfaces with no authentication until you configure it, so network policy, TLS and API keys are part of the build, not optional extras.
Storage requirements: Qdrant needs block storage with a POSIX-compatible file system and does not run on NFS or S3, so confirm that your storage class meets this before sizing.
Replication cost: in distributed mode, a replication factor of 2 doubles storage, and the Qdrant docs recommend at least three nodes for production quorum.
Access control: confirm whether your chosen store can filter results by the permissions of the source document, because permission-aware retrieval depends on it.
Support path: Qdrant's community Helm chart lacks enterprise features such as zero-downtime upgrades, so decide early who will operate upgrades.

Ask your implementation partner to show how permission metadata is written to each chunk and tested before any user sees an answer.

Operational requirements and how to verify each

This table sets out the operational requirements a RAG assistant must meet after launch, what each one asks of the platform and the evidence to collect. Read each row as a test you run before go-live: a restore test, an upgrade rehearsal in a non-production environment and a load test. The main takeaway is that refresh, backup, upgrades and scaling must be proven on your own stack, because a working demo does not show that the index can be rebuilt or the service survives a replica failure.

Requirement	What it asks for	How to verify it
Refresh and re-indexing	Changed or deleted source documents are re-parsed, re-embedded and removed from the index on a defined schedule, with deleted files gone from answers	Edit and delete a source file, run a refresh, then confirm the old text no longer appears in citations or retrieved chunks
Backup of the vector store	Vector data can be restored with its collection settings, and the source files can rebuild it if the snapshot is lost	Take a Qdrant snapshot, restore it to a clean instance and compare collection point counts and sample query results
Backup of the metadata database	PostgreSQL holds users, workspaces, knowledge base settings and application state, with a tested point-in-time or dump restore	Restore the database to a separate host, start the application against it and confirm users, permissions and knowledge bases match
Secrets and keys	SECRET_KEY and workspace encryption keys are stored and backed up under a documented process, since changing SECRET_KEY logs out all users	Restore a backup on a rehearsal host and confirm sessions, OAuth connections and stored credentials still work
Upgrade path and release cadence	Platform releases arrive often (Dify 1.16.0, 1.17.0 and 1.17.1 came between July and September 2026), so a repeatable upgrade routine is required	Rehearse each upgrade in a non-production copy with production-like data, then re-run retrieval and answer evaluation before promotion
Scaling for multiple replicas	Open WebUI needs PostgreSQL instead of SQLite, plus Redis for shared state and shared file storage for uploads and knowledge	Run two or more replicas behind a load balancer, upload a file on one replica and confirm it is retrievable from the others
Vector store deployment limits	Qdrant needs block storage with a POSIX file system, not NFS or S3, and three or more nodes are recommended for a production cluster	Confirm storage type in the target environment, then test node failure and confirm queries continue with quorum intact
Load and latency	Concurrent users and ingestion jobs share compute, so query latency and indexing throughput must be measured at expected peak	Run a load test with representative queries and a concurrent ingestion job, recording response times, errors and resource use
Monitoring and alerting	Health of the API, workers, vector store, database, queue and model endpoint is visible, with alerts on failed ingestion and stale indexes	Stop a worker and a model endpoint on purpose, then confirm alerts fire and the failure appears in logs with timestamps

Questions buyers ask before writing RAG requirements

How much cost and effort does requirements work add to a RAG assistant?

Requirements work is a smaller share of the total than the build, but its size depends on how many sources, identity systems and document types the assistant must cover. The cost has three parts: the time your own staff spend on document owners, identity admins and access rules; the consultant or partner time to write and review the requirements; and the later cost of fixing gaps that were not captured. Costs that recur include re-indexing, model and vector database upgrades, and evaluation runs after each change. No public figure for this work is supplied here, so ask any provider for a fixed-scope quote that lists each source, each document type, each permission model and each evaluation round as separate line items. Ask which of those items is one-time and which recurs.

How long does a RAG proof of concept take to deliver?

The duration depends on scope, including the number of sources, the file types, whether permissions must carry into answers, and how many evaluation rounds the business requires. A proof of concept on a narrow document set with one identity provider is shorter than one spanning several repositories and scanned archives. No duration is supplied for this work, so request a written timeline with the scope, the milestones and the acceptance tests before work starts. Most engagements at DeployOpen are fixed-scope, so the agreed scope and timeline are fixed in writing before the work begins.

What does a RAG requirements project need from our team?

The project needs three groups of people and one set of artifacts. Document owners confirm which sources are authoritative, which are outdated and which must stay out of the index. Identity admins explain how groups and attributes map to each source's access rules, because answers must respect those permissions. Business users write a test question set with expected answers and the source documents that should be cited. The artifacts are the source inventory, the permission map and the test set. Ask any partner to name these roles and the hours each will need before signing, since weak test questions make evaluation results meaningless.

Can our internal team write RAG requirements without an outside partner?

Yes, a capable internal team can write the requirements, provided it has access to document owners, identity administrators and someone who can design an evaluation. The harder parts are usually permissions and evaluation: mapping source-system access into retrieval filters, and building a test set that reflects real questions and includes adversarial cases such as prompt injection hidden in documents. An outside partner adds value when the team lacks experience with self-hosted vector databases or security testing of the assistant. Whichever path you choose, confirm who owns the final requirements document and the acceptance criteria.

How do we choose between Open WebUI, AnythingLLM, Dify and RAGFlow?

Start from licensing and deployment shape, then from features. Open WebUI is distributed under the Open WebUI License, whose clause 4 prohibits altering or removing branding except for deployments with no more than 50 end users in any rolling 30-day period, with written permission, or under other listed exceptions. Dify uses a modified Apache License 2.0 that forbids operating a multi-tenant environment from the source code without written authorization. AnythingLLM and RAGFlow are licensed under MIT and Apache-2.0 respectively. Then compare the features you need: SSO and SCIM provisioning, citation display, permission handling and connectors. Confirm each capability against the current release, since platforms change quickly.

Does the EU AI Act log retention rule apply to a RAG assistant?

It applies only if the assistant is a high-risk AI system under the Act, which depends on its use case and your legal review. If it is, Article 12(1) requires the system to technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. For a RAG assistant, the requirements should specify which events are logged, such as queries, retrieved source identifiers, cited passages and model versions, and where those logs are stored. Confirm with counsel whether your deployment is high-risk and whether sector rules require longer retention.

Which vector database and access controls should the requirements specify?

The requirements should name the vector database, its deployment mode and its access model before any build starts. Qdrant is licensed under Apache-2.0 and self-hostable, but by default self-deployed instances listen on all interfaces with no authentication, so the requirements must mandate API keys, TLS and per-collection permissions. Qdrant also needs block-level storage and cannot run on NFS or S3, and a production cluster is recommended with at least three nodes. Ask the partner to show how each collection maps to a source permission set and how audit logs of API operations are kept.

How to test answer quality before go-live

This sequence turns the requirements into evidence: it shows whether the assistant finds the right passages, respects source permissions, cites correctly, resists planted instructions in documents, and stays validated after every change. Each step produces a record that the client can keep for model risk, security and compliance reviews.

Build the question set with expected sources
Work with the business owners of each document collection to write a set of realistic questions that staff actually ask, and for each question record the passage or document that should support the answer. Include questions the corpus cannot answer, so the team can see whether the assistant declines or invents an answer. The client supplies the subject matter experts, the document owners and the approved source list. The step is done when the set is version-controlled, reviewed by a second business owner and covers every collection in scope.
Run retrieval tests for recall
Run each question through the retriever alone, before the language model is involved, and check whether the expected passage appears in the top results. Compare results across chunking settings, embedding models and whether hybrid search or reranking is enabled, so the team can see which configuration retrieves the right material. This step shows whether the problem is retrieval or generation, which matters because the fix differs. The output is a recall table per configuration, with misses listed by document type, such as scanned PDFs or tables, so the parsing gaps are visible.
Test permission filtering in retrieval
Create test users that represent each group and access level in the source systems, then run the same questions as each user and confirm that restricted passages never appear in results, citations or the generated answer. Include negative cases where a user asks about a document they cannot open, and check that the answer does not reveal its existence. The client supplies test accounts, the group mappings from the identity provider and the names of restricted documents. The step closes when every test user's results match the expected access matrix and the failures, if any, are logged as blocking defects.
Score citations and answer accuracy
Review each generated answer against the expected source and grade it for factual accuracy, completeness, and citation quality, meaning the cited passage actually supports the claim it is attached to. Use a scoring rubric agreed with the business owners, and have two reviewers score a sample so disagreements can be resolved and the rubric tightened. Where the team uses an evaluation tool such as Langfuse or Phoenix, store the scores with the trace so they can be compared over time. The output is a scored baseline with recorded reviewer agreement and a list of failure categories.
Test prompt injection through a planted document
Add a test document to a non-production index that contains instructions aimed at the assistant, such as an instruction to ignore its rules, reveal other documents or send content to an external address. Ask questions that cause the planted passage to be retrieved, then check whether the assistant follows the embedded instruction, exposes restricted content or cites the planted file as authoritative. Repeat with the planted text placed in tables, headers and scanned text where parsing may differ. The client confirms which guardrails and output controls are in scope. The step is complete when each attempt is recorded with its result and any successful injection has a documented remediation and a retest.
Set a re-test gate for every change
Define the changes that trigger a full or partial re-run of the test set: a new model or model version, a new embedding model, a change to chunking or reranking, a new document source, a permission mapping change, or an upgrade of the assistant platform or vector database. Put the gate in the change process so no change reaches production until the affected tests pass and the results are filed. The client decides who approves releases and where test evidence is stored. The output is a written gate procedure with pass thresholds agreed in advance, which is what auditors and model risk reviewers usually ask to see.

Questions that separate platforms and implementation partners

The strongest signal in any evaluation is how a platform or partner answers specific questions, not how broad their feature list looks. Ask how source permissions are enforced at retrieval time, not only at the front-end chat. A credible answer names the mechanism: the connector copies each document's access control list into chunk metadata, every query filters vector and keyword results against the asking user's identity before ranking, and the filter is tested with accounts that should see nothing. For a self-hosted Qdrant store, ask which of its controls they would use, such as per-collection read or write permissions through JWT-based access, and whether the API keys are scoped. Qdrant's security documentation states that self-deployed instances are not secure by default, so a partner who skips authentication setup in their answer is a warning sign.

Next, ask how retrieval and answers are tested, and what the evaluation set looks like. Look for a versioned set of real questions drawn from the target users, expected source passages for each one, and metrics for both retrieval (did the right chunk appear) and generation (is the answer supported by the citation). Ask who labels the expected answers and how often the set is rerun after a model, chunking or reranking change. Ask how document re-indexing is handled: which events trigger it, whether deleted or superseded files are removed from the vector index, and how a partner proves that a withdrawn policy no longer appears in answers. Ask how prompt injection arriving through documents is handled, including how retrieved text is separated from instructions and what output controls exist. Finally, ask which license limits apply to the chosen components. Open WebUI's branding clause changes obligations once a deployment exceeds 50 users in any rolling 30 day period, and Dify's license forbids operating a multi-tenant environment from the source code without written authorization. Ask the partner to name the license for every component, including the vector store, embedding service and any connector, and to state in writing which terms apply to your user count and business model. Ask for the partner's answers in writing, and compare them with the RAG knowledge assistant RFP questions page for the full question set.

How DeployOpen scopes and delivers a RAG knowledge assistant

DeployOpen plans, implements, customizes, integrates, hardens and documents self-hosted open-source platforms inside the client's own infrastructure. For a RAG knowledge assistant, that means turning the requirements in this checklist into a written scope, building the ingestion, retrieval and access controls in your environment, and handing over the configuration, tests and runbooks your team needs to operate it.

How we scope

Inventory document sources, file types, scanned content and the systems that hold source permissions
Agree in writing which collections, user groups, models and vector database options are in scope
Define answer quality criteria, refresh frequency and guardrail requirements before any build work starts

How we build

Configure the chosen platform, such as RAGFlow, AnythingLLM, Dify or Open WebUI, with the self-hosted vector database in your infrastructure
Build connectors and permission mapping so retrieved passages respect the access rules of their source documents
Test retrieval and answer accuracy against a question set agreed with your business owners, including prompt injection cases in documents

How we hand over

Deliver architecture, configuration, data flow and license limit documentation for your architecture and compliance reviewers
Record evaluation results, log settings and retention choices as evidence for internal and external assessments
Train operators on re-indexing, source onboarding, model changes and incident steps, then transfer administrative access

How we support

Offer ongoing support, upgrades and continuous testing as a retainer after handover
Re-run agreed evaluation sets after platform upgrades, model swaps or significant changes to the document corpus
Keep a change record so compliance teams can trace what changed in the assistant and when

RAG knowledge assistant requirements checklist