RAG knowledge assistant RFP questions

A strong partner names the enforcement point and shows it working. Permissions are read from the source system (file share, wiki, document management system or helpdesk), stored as access metadata on every chunk, and filtered before any text reaches the model. The answer should specify which identity system supplies group membership, how the assistant authenticates users (for example through OIDC or SAML), and whether filtering happens inside the vector database query or in a service in front of it. Ask for a test where a user without access to a folder asks a question whose answer lives only there, and confirm that no snippet, citation or summary leaks. Open WebUI documents role-based access control with groups and per-resource permissions, and Dify Enterprise documents separate SSO scopes for workspace and web app access, so ask how these are configured for the chosen platform.

The partner should describe a sync job that re-reads permissions on a defined schedule and on change events where the source supports them, then updates access metadata on affected chunks without a full re-index. Ask how long a revoked user can still see content after removal in the source, and whether the design fails closed when the sync is delayed or errors. Ask whether group membership is resolved at query time from the identity provider or cached, and how cached entries expire. A good answer includes a re-test after each change: a scripted set of users and documents whose expected visibility is checked against the live index. Confirm that deleted source files also remove their chunks and embeddings, not only their titles.

Retrieval testing starts with a question set that the client owns, written by subject experts, with each question linked to the documents and passages that should answer it. The partner should measure whether the correct source appears in the top results for each question, and report the results per collection and per user role. Ask for the size and make-up of the set, how it covers edge cases such as scanned tables, superseded policies and questions with no answer in the corpus, and how it is versioned. The test set should be kept separate from anything used to tune chunking or reranking, so the scores are not inflated. Ask who maintains the labels after launch and how new documents add new test questions.

Accuracy and citation correctness are scored separately. Accuracy checks whether the answer is supported by the retrieved text and matches the expected answer, typically through human review of a sample plus automated judgments that the client can audit. Citation correctness checks that each cited passage actually contains the claim it supports, and that the link opens the current version of the document. Ask the partner to name the evaluation tooling and where the scores are stored. Langfuse and Arize Phoenix both offer tracing and evaluation features, and Phoenix is licensed under the Elastic License 2.0, which restricts offering it as a hosted service, so confirm the licensing fits the deployment. Ask for the pass criteria agreed before launch and the process for a release that falls below them.

Failures should be logged as structured records that link the user query, the retrieved chunks with their source and version, the permission filter applied, the model and prompt version, the answer, and the citations shown. The partner should explain how logs are stored in the client's infrastructure, who can read them, how long they are kept, and how personal data in queries is handled. Ask whether logs cover refusals, empty retrievals and user feedback, and how reviewers triage them into test set additions. For systems that may be high risk, Article 12 of the EU AI Act requires technical capability for automatic event logging over the system's lifetime, and Article 19 requires providers to keep those logs for at least six months unless other law provides otherwise. Ask for a sample record of a logged query so you can judge completeness before signing.

DeployOpen does not publish prices, and no supplied figure gives a cost for evaluation work, so ask for a fixed-scope quote built from written deliverables. The main cost drivers are the number of document collections, the size and complexity of the labeled question set, how much expert time the client must give for labeling and review, the number of user roles to test for permissions, and how many evaluation cycles run before launch. Recurring costs include re-running the test set after each model, prompt, or index change, and maintaining labels as documents change. Ask the partner to separate one-time setup from ongoing evaluation, and to state what the client must provide, such as subject matter reviewers and test accounts for each role.

Yes, if the platform exposes the data and configuration needed to test, and the client has someone who can own the test set. Open-source components such as Qdrant document API keys, JWT-based per-collection permissions and JSON audit logging of API operations, which give a team concrete points to test. The practical requirement is ownership: a named person must maintain the labeled questions, run the permission test users after every identity change, and review logged failures. Ask the partner what runbooks, scripts and test accounts will be handed over, what training is included, and whether a retainer for continuous testing is needed for the first months. The engagement should document each test so the client can repeat it without the partner.

The main drivers are the number of document sources, how complex the permission model is, how clean the documents are, and how deeply the answers are evaluated. Each source adds connector work, parsing and a mapping of who may see what. Permissions that follow a simple folder structure cost less to carry into answers than permissions that vary by document, matter, or business unit. Scanned PDFs, inconsistent templates and duplicate policies add cleanup before indexing. Evaluation depth matters as well: a small test set checked by hand costs less than a graded question set with repeated runs after each change. Ask the partner to itemize each of these in the written quote, because the number belongs to that scope, not to a generic estimate.

The initial build is one-time: platform setup, connectors, ingestion pipelines, permission mapping, integration with the identity system, and the first evaluation round. Recurring work includes re-indexing when documents change, retesting answers after model, prompt or platform updates, applying upgrades to the platform and the vector database, monitoring, and backups. Retainers usually cover the recurring part, and their scope should list response expectations, upgrade cadence and how many re-evaluation runs are included. Also ask who pays for model inference and hosting, since those costs are usually billed by the infrastructure owner rather than the implementation partner. The written quote should separate these lines so the recurring cost is visible before the build starts.

The client needs to supply access and decisions that only it can make. The most common inputs are:

• Read access or a service account for the identity system, so group membership can be mapped to source permissions.
• Named document owners for each source, who confirm which content is current and who may see it.
• A test question set with expected answers and the source documents that should support them, written by people who know the work.
• Decision makers for permissions, including rules for restricted material and who approves exceptions.

If these inputs arrive late, the schedule slips, so ask the partner to list them with owners and dates in the scope document.

Often yes, if the team already runs containerized services, manages identity integration and has someone who can own answer quality. The open-source platforms covered on this page are documented and can be deployed by an experienced platform team. The harder parts are usually permission carry-over, evaluation and ongoing re-indexing, because they require repeated testing and documented decisions. Ask whether the team can staff those roles for the life of the system, not only the first release. A partner can handle the build, the handover documentation and a retainer for upgrades and retesting, while your team keeps operational control. Confirm the division of responsibilities in writing before choosing either path.

Most DeployOpen engagements are fixed-scope, and the scope, deliverables and timeline are agreed in writing before work begins. The timeline depends on the number of sources, the identity integration, the state of the documents and the evaluation work agreed in scope. The written scope should name milestones, the client inputs each milestone depends on, and what counts as acceptance. Ask for the milestone list and the dependencies rather than a single date, since a date without dependencies is not a plan. DeployOpen does not publish a price list, so cost and duration come from that written scope.

It can. Open WebUI is distributed under the Open WebUI License, which restricts altering or removing its branding in deployments above 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written permission. Dify's license requires a commercial license to operate a multi-tenant service from its source code. If the assistant serves a large internal population or customers, the licensing review should happen before platform selection, and any enterprise license cost should appear as a separate line in the quote. Ask the partner for a written licensing note that matches your user count and deployment model, and have your legal team confirm it.

A credible partner should describe a test process that runs before go-live and after each significant change. The process typically includes a question set drawn from real work, expected source documents for each question, scoring of whether answers are correct and cited, and a record of failures with the decision taken on each. Ask whether the evaluation covers permission tests, such as confirming that a user without access to a restricted document never receives its content in an answer. Evaluation depth is a cost driver, so ask how many questions, how many scoring rounds and who signs off the results. The evidence should be kept in a form your auditors and risk team can review.