Why a RAG knowledge assistant needs a platform decision first
Staff are searching scattered internal documents, and the security team needs to control who sees what before anything goes live. For a retrieval-augmented assistant, the open-source choice is usually between RAGFlow for document-heavy retrieval, AnythingLLM for small teams and quick starts, and Dify for workflow-driven applications. The right fit depends less on answer quality demos than on three things: how permissions are enforced at retrieval time, what the license allows you to do with the software, and who will operate the platform after the pilot. Those answers should be settled before a document corpus is loaded, because re-ingesting content to fix access rules is expensive and error-prone.
Start by mapping your document sources and their current access controls. Confirm whether each platform can apply access rules at the level of the knowledge base, the document or the chunk, and whether it can read group membership from your identity provider or needs those mappings maintained by hand. Next, check the license terms against your intended use. Dify's modified Apache License 2.0 requires a commercial license to run a multi-tenant service from the source code, while RAGFlow and AnythingLLM are published under Apache-2.0 and MIT. Ask your legal team to review the actual license text for the version you plan to deploy, not a summary.
DeployOpen plans, implements, hardens and supports these platforms inside your own cloud accounts, data centers or air-gapped networks, so the data never leaves your control. A written scope should name the platform, the document sources in scope, the identity integration, the evaluation method and who runs the system at handover. Treat these as decisions to record in that scope, not details to settle during build.
What to agree on before choosing a RAG platform
A RAG knowledge assistant is only as safe and useful as the decisions made before the platform is picked. The six requirements below connect document scope, access rules, hosting location, quality measurement, logging and ownership, and each one narrows which open-source option can realistically meet it.
Document sources and formats
List every repository the assistant must read, such as file shares, wikis, ticketing systems, email archives and scanned PDFs, and the formats in each. Scanned or table-heavy documents need OCR and layout parsing, and not every RAG engine handles them equally well. Ask each candidate which connectors it ships, which it expects you to build, and how it handles updates and deletions in the source. Sources with frequent edits need an incremental sync schedule, not a one-time import.
Permission model mirrored from sources
The assistant must never show an answer built from a document the asking user cannot open in the source system. Decide whether permissions are enforced at ingestion (copying access control lists into the index), at query time (filtering results against the live identity), or both. Confirm whether the candidate can apply document-level filters from your identity provider groups and how it behaves when a source permission changes after indexing.
Data residency and model hosting
Decide where documents, embeddings, prompts and answers may be stored and processed, and whether models run in your cloud account, a data center or an air-gapped network. The answer determines whether you need a self-hosted model server such as Ollama or vLLM or can call an external model API through a gateway. Confirm the license of the model you choose, since open-weight model terms can differ from the license of the platform around them.
Answer quality targets and measurement
Define what a good answer means for your users, such as citation accuracy, refusal when the source is missing, and correctness on a fixed set of real questions from your business. Build a test set with expected sources and score retrieval and generation separately. Check whether the platform exposes retrieval results, chunk scores and citations in a form your evaluation tooling can read, because a black-box answer cannot be measured or improved.
Audit logging and retention
Specify which events are recorded: who asked what, which documents were retrieved, which answer was returned, and which permission check applied. Record-keeping duties can apply to some AI systems, and Article 12 of the EU AI Act requires high-risk systems to allow automatic event logging over their lifetime. Article 19 sets a minimum retention of six months unless other law requires longer. Confirm whether the platform writes these logs natively, and where they can be exported to your log store.
Named owner for the assistant
Name one accountable owner who approves source onboarding, reviews quality reports, handles permission incidents and decides when content is removed from the index. Identify who operates the platform upgrades, backups and model changes, and whether that is your team, a managed partner or a retainer. Ownership gaps are the most common reason an internal assistant stops being maintained after its pilot, so the answer should be written into the scope before selection.
RAGFlow, AnythingLLM and Dify compared on licensing and fit
This table compares three open-source retrieval-augmented generation platforms on best fit, licensing constraints, the permission model to confirm and operating effort. Read the licensing row first, because Dify's terms can rule it out for a multi-tenant service before any technical comparison matters. The main takeaway is that licensing and permission depth often decide the shortlist, and each row should be verified against the current license text before procurement.
| Criterion | RAGFlow | AnythingLLM | Dify |
|---|---|---|---|
| Best fit | Document-heavy knowledge bases that need detailed ingestion, parsing and retrieval tuning | Smaller teams or departments that want a packaged workspace with document chat and a simple setup | Teams building assistants and workflows on one platform, with an application builder for non-engineers |
| License | Apache-2.0 | MIT (v1.18.0 published October 13, 2026) | Modified Apache License 2.0 (1.17.1 published September 10, 2026); a commercial license is required to operate a multi-tenant service from the source code |
| Licensing risk for internal use | Low for internal deployment under Apache-2.0 terms; confirm notice and attribution duties with counsel | Low for internal deployment under MIT terms; confirm the terms of any bundled third-party components | Confirm whether your deployment counts as a multi-tenant service; if it does, a commercial license is needed |
| Permission model to confirm | Confirm how knowledge bases map to user groups, whether document-level access is enforced at retrieval, and how SSO groups are synced | Confirm workspace-level versus document-level access, user roles and whether retrieval respects them | Confirm workspace roles, app-level access, dataset permissions and whether retrieval respects them for each end user |
| Operating effort | Expect more work on parsing pipelines, model configuration and tuning retrieval quality; confirm the upgrade path between releases | Lower setup effort for a single team; confirm how it scales when many workspaces and users share one instance | Moderate effort; more components to run, plus ongoing review of applications and workflows built by users |
| Front end note | Usually paired with its own interface; confirm whether a separate chat front end is needed | Includes its own chat interface, so a separate front end is often unnecessary | Includes its own app interface; confirm how it fits with an existing portal or chat tool |
| Open WebUI as a front end | If Open WebUI is considered, check its license first: clause 4 limits removing or altering its branding beyond listed exceptions, including deployments with no more than 50 end users in any rolling 30 day period | Not applicable as a bundled option; check license terms before combining it with other tools | Not applicable as a bundled option; check license terms before combining it with other tools |
Reference architecture components and how they connect
A RAG knowledge assistant is a chain: documents enter through ingestion, get stored as searchable chunks, are retrieved for each question, and reach a model through a gateway that records what happened. Evaluation and tracing sit alongside the chain so that answer quality and access decisions can be reviewed after the fact.
Ingestion and parsing
This layer pulls documents from file shares, wikis, ticketing and document management systems and converts PDFs, slides, scanned files and tables into text the engine can index. It is where source permissions must be captured, because a chunk that loses its source access list can surface to users who should never see it. Check which file types the chosen engine parses natively, how it handles scanned pages without OCR, and whether it re-ingests on document change or only on a schedule.
Chunking and embedding store
Chunking decides how documents are split, and the embedding store holds the vectors used for similarity search. Chunk size and overlap strongly affect whether answers cite the right passage, so they belong in the evaluation plan rather than in defaults. Confirm which vector databases the engine supports, where the store runs (inside your network or a managed service), and that each chunk keeps metadata for source, owner, and access groups.
Retrieval and reranking
Retrieval selects candidate chunks for each question, and reranking reorders them before they reach the model. Combining keyword and vector search usually matters more for internal documents with product codes, policy numbers and names than either method alone. Ask the engine vendor or your implementation partner to show how the permission filter runs: it must apply before chunks are passed to the model, not after the answer is written.
Model serving for local models
Ollama suits single-host or small-team serving of local models, while vLLM is an inference and serving engine typically used where throughput and concurrency matter more. Both keep prompts and documents inside your infrastructure when the models run there. Size GPU or CPU capacity from measured load in your pilot, and confirm the model licenses separately from the software license, since model terms vary by model.
Gateway for keys, budgets and logs
A gateway such as LiteLLM or Apache APISIX sits between the assistant and every model endpoint. It issues keys per team or application, enforces spend or token budgets, routes between local and external models, and writes request logs that support record-keeping. Decide early whether external model providers are allowed at all for each document classification, because the gateway is the control point that enforces that rule.
Evaluation and tracing
Langfuse and Arize Phoenix record traces of retrieval and generation so teams can see which chunks supported each answer and score answer quality against a test set. Langfuse is MIT licensed except its ee directories, so review those paths before enabling them. Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service, so confirm that your internal use or delivery model is allowed before choosing it.
How ingestion and document permissions work in practice
The assistant must respect the access rights of the source system, so a user should only ever see answers built from documents that user could open in SharePoint, a file share, Confluence or the ticketing tool. In practice this means every chunk written to the index carries access metadata copied from the source (owner, group or ACL identifiers, and a document ID), and the retrieval step filters on the requesting user's identity before any text is passed to the model. Filtering after generation, or relying on the chat interface to hide answers, is not an acceptable control. Confirm with each candidate platform (RAGFlow, AnythingLLM and Dify) whether it stores source ACLs at ingestion, whether it can apply a per-user filter at query time, and whether that filter is enforced in the API as well as the UI.
Permissions and content both change after ingestion, so the sync design matters as much as the first load. Ask each platform how it detects deleted files, moved files and permission changes, whether it uses incremental sync or full re-indexing, and how long a revoked user can still retrieve a chunk under the default schedule. Scanned documents need a separate decision: without OCR, image-only PDFs produce no searchable text, and OCR output carries errors that affect retrieval. Confirm which OCR engine each platform uses, whether OCR output is stored alongside the original, and who reviews the quality of a sample before the corpus goes live.
Test permissions as a release gate, not a one-off check. For a set of test accounts that cover each group, pick documents that each account cannot open in the source system, then ask questions whose answers appear only in those documents and confirm that no chunk, citation or summary of them is returned. Repeat the test after a permission is removed and after a file is deleted, and include the retrieval logs in the evidence pack. Illustrative example: a contractor removed from a finance folder should get no answer that quotes it, even on the next day's sync. The test plan, the sync interval and the owner of each source connector should be written into the statement of work before the pilot starts.
Build in-house, work with a partner, or buy a SaaS assistant
This table compares building on open-source RAG platforms in-house, working with an implementation partner on the same open-source stack, and buying a commercial SaaS assistant. Read each row against your own data residency rules, the team you can staff, and how you want to split one-time and recurring costs. The main takeaway is that open source moves license cost to zero but not operating effort, so the decision turns on who carries upgrades, evaluation and support.
| Factor | In-house build on open source | Open source with an implementation partner | Commercial SaaS assistant |
|---|---|---|---|
| Control over documents and embeddings | Full control. Documents, vector indexes and logs stay in your cloud account or data center. | Full control in your environment, with the partner working inside your accounts under agreed access rules. | Depends on the vendor's hosting region, subprocessors and retention terms. Confirm them in writing. |
| Skills needed | Platform engineering, vector search, identity integration, and someone who owns retrieval evaluation. | Your team owns business decisions and content. The partner supplies engineering and evaluation capacity. | Mostly administration, content owners and prompt or policy configuration. |
| Effort to reach pilot | Sets the timeline through internal capacity. Scope the ingestion connectors and permission model first. | Defined in a fixed-scope written plan, with deliverables and timelines agreed before work starts. | Usually shorter for setup, but connector and permission gaps may still need custom work. Ask the vendor to confirm. |
| Upgrade burden | Your team tracks releases of the RAG engine, model server, vector store and connectors, and tests each upgrade. | Upgrades can be included in a support retainer. Confirm the upgrade cadence and test scope in the contract. | Handled by the vendor. Confirm how feature changes that affect answers are announced and tested. |
| Answer quality evaluation | Your team builds the test set, scoring and regression checks. | The partner can build and maintain the evaluation set with your content owners, and report results over time. | Vendor-provided metrics vary. Ask what you can export and whether you can run your own test set. |
| One-time costs | Architecture, connector build, permission mapping, evaluation set, security review and pilot. | The same activities, paid as scoped engagement work, plus knowledge transfer to your team. | Configuration, connector work, integration with identity and a validation phase. Vendor professional services may apply. |
| Recurring costs | Hosting, model usage, storage, backups, staff time for upgrades and quality reviews. | Hosting and model usage in your accounts, plus any support or upgrade retainer you choose. | Subscription, usage fees and any add-on connectors or storage tiers. Ask for the full pricing basis in writing. |
| Cost drivers to ask about | Number of document sources, GPU or model hosting choice, and the size of the evaluation set. | Scope of connectors, number of permission models, and whether knowledge transfer is included. | Seat count, query volume, indexed content volume and premium connectors. Confirm each unit price. |
| Best fit | Teams with platform engineering depth and strict data control who can run evaluation continuously. | Regulated organizations that want open-source control but lack capacity for build and ongoing operation. | Organizations with low data sensitivity, few custom sources and acceptance of vendor hosting terms. |
How a RAG assistant rollout runs from pilot to production
The sequence moves a knowledge assistant from a scoped pilot on one document domain to a monitored production service, with each phase closing on a named output that the sponsor can review before the next phase begins.
Scope the use case and inventory sources
The owning business function and the DeployOpen team agree on the questions the assistant must answer, the user groups it serves and the document repositories it may read. Each source is recorded with its owner, format, access model, sensitivity classification and refresh pattern. Platform choice between RAGFlow, AnythingLLM and Dify is confirmed here against licensing and hosting constraints. The output is a written source inventory and a scope document that names what is excluded. Ask the implementation partner to show the scope and timeline in the written scope before work starts.
Build the pilot on one document domain
The team deploys the chosen platform in your infrastructure, connects one document domain such as a single policy library or product manual, and configures ingestion, chunking, embeddings and the model endpoint. Pilot scope is deliberately narrow so that retrieval failures are traceable to a small, known corpus. The output is a running pilot environment and a documented ingestion configuration that another engineer could rebuild from the runbook.
Create the evaluation set before tuning
Subject matter experts write a set of representative questions with the expected source passages or answers, including questions the assistant should decline or escalate. The team scores retrieval (whether the right passage is returned) and answer quality (whether the response is grounded in that passage and correct) for each pilot configuration. Changes to chunking, retrieval settings or prompts are accepted only when the scores are recorded against the same set. The output is a versioned evaluation set with baseline scores and a list of known failure categories.
Test permissions and document-level access
The team checks that the assistant honors the access rights of the source system, so a user cannot receive an answer drawn from a document they could not open directly. Test accounts from each user group run a fixed set of queries, including attempts to retrieve restricted content through paraphrase or follow-up questions. Synchronization of group membership and document permissions is verified after a change in the source system. The output is a permission test report with pass and fail results, and a documented decision on how access is enforced when source permissions cannot be mapped cleanly.
Complete the security review and logging setup
The security team reviews authentication, secrets handling, network paths to the model endpoint, prompt injection exposure from ingested documents, and administrative access. Logging is configured for queries, retrieved sources, responses and user identity, with retention set by your records and privacy obligations. If the assistant falls within an AI regulation's record-keeping scope, the log design is checked against that requirement with counsel. The output is a signed security review with open findings tracked, and a log schema with an assigned retention period.
Roll out to named teams in controlled waves
Access is granted to named teams in stages, each with an owner who reports unanswered questions, wrong answers and access complaints. The evaluation set is rerun at each wave, and new failure categories are added to it. Expansion to another document domain happens only after the prior domain meets the agreed acceptance criteria. The output is a rollout log that records each wave, the users admitted, the scores at that point and the issues closed or accepted.
Run production operations with monitoring
Operations staff monitor query volume, latency, error rates, model and embedding usage, retrieval hit quality and user feedback. Alerts route to the team that owns the platform, and the evaluation set is rerun on a schedule and after any model or platform upgrade. Platform updates are tested in a non-production copy before they reach users. The output is a operations runbook, a monitoring dashboard with agreed alert thresholds, and a named support owner for your organization.
Maintain the document refresh process
Documents change, so the team defines how updates, deletions and new versions reach the index, who approves new sources, and how stale content is retired from answers. Re-ingestion is tested against the evaluation set so that a refresh does not silently degrade answers. Removed or restricted documents are confirmed as no longer retrievable. The output is a written refresh procedure with owners, a reconciliation report comparing source and index, and a schedule agreed in the operations contract.
Questions buyers ask before hiring help for a RAG assistant
What drives the cost of a RAG knowledge assistant implementation?
Cost is driven mainly by the number and messiness of source systems, the permission model you need to honor, the volume of documents to ingest, and how much evaluation work is required before go-live. Some costs are one-time, such as design, integration, connector work, and initial evaluation sets. Others recur, such as compute for embedding and inference, storage for vector and document indexes, model API charges if you use hosted models, log storage, and support. Platform licenses differ: RAGFlow and AnythingLLM are published under Apache-2.0 and MIT, while Dify carries a modified Apache License 2.0 with conditions on multi-tenant hosting. Ask any implementation partner for a quote that separates one-time implementation, the recurring infrastructure you will pay directly, and any retainer for upgrades and support. DeployOpen does not publish a price list; a quote follows from a written scope.
How long does a pilot for a RAG assistant take?
The pilot length depends on how many document sources you connect, whether access rules must mirror existing permissions, how many people will test answers, and how quickly your team can review results. Pilots that use one or two well-governed document stores move faster than pilots that must reconcile several wikis, file shares and ticketing systems with different permission models. Agree the phases and their exit criteria in writing before work starts, since most engagements are fixed-scope with deliverables and timelines set out in advance. Ask the partner which source systems are in scope, what evaluation results must be met to move from pilot to production, and who signs off each phase.
What does the work need from our own team?
Expect to supply several roles: a business owner who defines which questions the assistant must answer and which content is authoritative, an identity or access administrator who can confirm how permissions are stored in source systems, an infrastructure team that provides the hosting environment in your cloud account, data centers or air-gapped network, and reviewers who label expected answers for evaluation. Security and compliance staff should review data flows, retention settings and logging design early rather than at the end. Ask the partner to list these roles and the hours each needs per phase, and confirm who owns the source documents once the project ends.
Can our team run a RAG assistant without a partner?
Yes, a capable platform team can run a RAG assistant without an external partner, provided it has skills in container operations, identity integration, vector and document storage, evaluation methods and incident response. The harder parts are usually ingestion pipelines that keep content current, permission mapping that stays correct when source access changes, and a regular evaluation cycle after each model or prompt change. Consider whether your team can absorb upgrades to the chosen platform and maintain backups. If not, a partner can build the system and then provide documented runbooks, upgrades and continuous support under a retainer.
How do we measure answer quality before and after launch?
Measure answer quality with a fixed evaluation set of real questions from your users, paired with expected sources or reference answers, and score retrieval and generation separately. Retrieval metrics show whether the right passages were found; generation checks show whether the answer is faithful to those passages and cites them. Run the same set after every change to chunking, embedding model, prompt or model version, and keep the results as evidence. Open source observability tools such as Langfuse (MIT, except its ee directories) and Arize Phoenix (Elastic License 2.0) can record traces and evaluation scores, though Phoenix's license prohibits offering it to third parties as a hosted service. Ask the partner which evaluation method they will use and what thresholds gate each phase.
Which license terms matter most for a RAG assistant?
Three license questions matter most. First, whether you can host the platform for internal users without restriction; Dify's modified Apache License 2.0 requires a commercial license to operate a multi-tenant service from its source code, so check whether your deployment counts as multi-tenant. Second, whether the license allows you to remove or change branding, which matters for Open WebUI because its license restricts branding changes beyond defined limits, including a 50 end user threshold in any rolling 30 day period. Third, whether supporting components carry copyleft or enterprise-only directories, such as LiteLLM's enterprise directory or the ee directories in Langfuse. Have legal counsel review the exact license file for the version you deploy, because licenses change between releases.
What happens to logs and retention under Article 12 and Article 19 of the EU AI Act?
Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow automatic recording of events over the system's lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Whether a RAG assistant is high-risk depends on its use case, so confirm classification with counsel first. If it is in scope, design logs to capture the prompt, retrieved sources, model version, and output events, with retention set to meet the longer of the six-month minimum or other applicable law. Logs contain personal data and sensitive content, so define access controls, encryption and deletion schedules before go-live. DeployOpen prepares systems and evidence for compliance assessments but does not issue audit reports or certifications; independent auditors and certification bodies make those determinations.
Compliance evidence a RAG assistant can help produce
DeployOpen prepares systems, controls and evidence for compliance assessments, but it does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations. The practical question for a RAG knowledge assistant is which records the deployment produces by default, which ones you must configure, and which ones the auditor will ask for. Start by asking your auditor which frameworks and regulations apply to the assistant, since a internal policy assistant and an assistant used in a high-risk decision process carry very different record-keeping duties.
If the assistant falls under the EU AI Act as a high-risk system, Article 12(1) requires the system to technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law sets a longer period. For a RAG assistant, the useful log records are the user identity, timestamp, query, the retrieved document chunks and their source paths or versions, the model and prompt template used, and the generated answer. Confirm with your legal and compliance team whether your deployment is in scope, and check which of these fields the platform you choose (RAGFlow, AnythingLLM or Dify) writes out of the box and which need an export or a database query to capture. Keep the retention period in your own configuration, not in the vendor's defaults.
Access control evidence and change records usually matter as much as query logs. Keep exports of the group-to-collection mappings that govern which documents each role can retrieve, the identity provider configuration that feeds them, and dated reviews of those mappings. For change records, version the ingestion pipeline settings, embedding model, chunking parameters, system prompts and evaluation results, so you can show what the assistant answered from on a given date. Plan this evidence as a deliverable from the pilot onward. Retrofitting it at production readiness is the most common reason these records have gaps.
How DeployOpen scopes, builds, hands over and supports RAG assistants
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For a RAG knowledge assistant, that means fixing the document sources, permission model and evaluation criteria in writing before the platform is configured, then handing over a system your team can operate.
How we scope
- Inventory document sources, formats and owners, and confirm which repositories are in the first ingestion wave
- Map existing identity groups and document permissions so access rules carry into retrieval results
- Agree the written scope, deliverables and timeline with you before any work starts, as engagements are fixed-scope
How we build
- Deploy the chosen platform (RAGFlow, AnythingLLM or Dify) in your environment, with ingestion, chunking and retrieval configured to your documents
- Build a labeled test question set and run answer quality evaluations, recording failures for review before each release
- Run security testing of the assistant and its connectors, including prompt injection and permission bypass attempts
How we hand over
- Document architecture, configuration, connector setup, permission mapping and evaluation results in a handover pack
- Train your platform, security and support staff to run ingestion jobs, manage access and review answer quality
- Prepare runbooks for backup, restore, upgrades and log retention so your team can operate the system after handover
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, sized to the scope you agree
- Re-run evaluation sets after platform upgrades, model changes or significant new document sources
- Keep compliance evidence current, such as access records and logs, for assessments run by independent auditors
