Why a RAG knowledge assistant for financial services needs more than a chatbot
A compliance or operations lead at a bank or insurer often has the same problem: staff search dozens of policy, procedure and product files, and the answers they find depend on who happened to locate the right version. A RAG knowledge assistant can work in this setting if retrieval respects source permissions, every answer cites its sources and every interaction is logged for review. Those three conditions are what separate a supervised tool from a general document chatbot. The platform choice follows from them: self-hosted RAGFlow, Dify or Open WebUI, each paired with a self-hosted vector database, selected according to licensing terms and how well each one can enforce document-level permissions in your environment.
Generic document chat fails in financial services for three reasons. First, information barriers (such as those separating research, advisory and trading teams, or deal teams from the wider organization) mean that a retrieval step which ignores source permissions can leak restricted content into an answer, even when the user never opens the file. Second, supervisors expect records of what was asked, what was retrieved and what was answered, which turns the chat history into a supervisory record that needs retention rules. Third, answer quality becomes a model risk question, so the bank or insurer must review how the assistant performs before and after release, under its own model risk framework. Each of these needs a named owner: a business owner for the corpus, a second line reviewer for access and logging, and an engineer who can prove retrieval filters are enforced.
Before you shortlist platforms, confirm three things with your vendor and your own architecture team: whether the platform can filter retrieved chunks by the requesting user's entitlements taken from your identity provider, whether it can export full interaction logs to your own storage, and which license terms apply to your deployment model. Dify's license, for example, restricts operating multi-tenant services from the source code, and Open WebUI's license restricts branding changes beyond small deployments, so the license review belongs in the first design decision, not after a pilot.
Which financial services obligations shape the RAG build
This table maps specific financial services obligations to the build requirements they create for a RAG assistant, along with the evidence an assessor or supervisor is likely to ask for. Read each row as a design input to settle before ingestion starts, not as a compliance finding. The main takeaway is that register entries, exit plans, data location terms, log retention and access controls are decided in the architecture and contracts, so they are cheap to design early and costly to retrofit.
| Obligation | What it asks for | Build requirement and evidence |
|---|---|---|
| DORA Article 28(3) | A register of information covering all contractual arrangements for ICT services | Record the self-hosted platform, vector database, model provider and any connector vendor as register entries, with the function each one supports and the data it touches |
| DORA Article 28(8) | Documented, tested exit strategies and transition plans for ICT services supporting critical or important functions | Keep vector indexes and document copies exportable in open formats, and test a rebuild of the index on a second environment before go-live |
| DORA Article 30(2) | Contracts state where services are provided and data processed, with advance notice of changes and data return on termination or insolvency | Pin hosting regions and data stores in the contract and runbook, and confirm how embeddings, logs and model prompts are returned or deleted at exit |
| EU AI Act Article 12(1) | High-risk AI systems must technically allow automatic recording of events over the system lifetime | Enable structured logs of queries, retrieved source identifiers, model version and answer output, confirming with counsel whether the use case is in scope as high-risk |
| EU AI Act Article 19(1) | Providers keep automatically generated logs for at least six months unless other law provides otherwise | Set log retention to at least six months and align with longer record-keeping rules that apply to the institution, such as supervisory retention periods |
| FTC Safeguards Rule, 16 CFR 314.4(c)(5) | Multi-factor authentication for any individual accessing any information system, unless the Qualified Individual approves an equivalent control in writing | Enforce MFA at the identity provider for the assistant, the admin console, the vector database and the model gateway, or keep a signed exception that names the equivalent control |
| SEC Release 33-8810, IT general controls | Program development, program changes, computer operations and access to programs and data as relevant IT general control areas | Put configuration changes to the RAG stack under change tickets, keep deployment and operations logs, and review administrator and data access quarterly |
| PCI DSS v4.0.1 scoping | Systems that store, process or transmit cardholder data, or could affect that data environment, are in scope | Exclude cardholder data from the index through ingestion filters, and if any card data can reach the assistant, bring the retrieval stack into the assessed environment |
| Permission-aware retrieval (information barriers) | Answers must respect the source document permissions and information barriers that apply to each user | Filter retrieval by the user's entitlements at query time, and test with sample users from restricted desks to show no restricted chunk or citation appears |
Which open-source platforms fit a bank or insurer, and why
Each option below handles a different part of a RAG assistant: the application layer, the retrieval engine or the vector store. For a bank or insurer, the deciding questions are license limits on how the platform is used, whether source document permissions can be enforced at retrieval time, and what security controls the platform ships with.
Dify for governed RAG workflows
Dify provides a visual builder for RAG pipelines, agents and a prompt IDE, with monitoring and API access. Its Community edition is limited to a single workspace, so multi-department use across business lines needs the Enterprise edition, which adds SSO, multiple workspaces and professional support. The Dify Open Source License bars operating a multi-tenant environment without written authorization, so a group running separate entities or subsidiaries as tenants must confirm the license position in writing. Confirm how document-level permissions are applied to retrieval, since the workspace model alone does not filter chunks by user.
RAGFlow for document-heavy retrieval
RAGFlow is an open source retrieval-augmented generation engine under Apache-2.0, which makes its license terms the least restrictive of the application options here. It is aimed at deep document parsing and retrieval over complex files such as scanned policies and tables, which suits product sheets and procedure manuals. Confirm which access control model it supports for knowledge bases, how it maps identities from your directory, and whether citations point to the exact page and passage that auditors will need to check.
AnythingLLM for team knowledge spaces
AnythingLLM is an MIT-licensed application that organizes documents into workspaces and offers a chat interface with citations. Its simple structure makes it quick to pilot for a single business unit, such as a retail banking product team. The limit for regulated use is governance depth: confirm how workspace membership is managed, whether it integrates with your identity provider, how user activity is logged, and whether access to a source document is enforced when a user asks a question that touches it.
Open WebUI for staff chat with documents
Open WebUI is a chat interface that can connect to document retrieval and local models. Its license includes clause 4, which prohibits altering or removing branding except for deployments with no more than 50 end users in any rolling 30 day period, with written permission, or under other listed exceptions. A bank or insurer with a broad staff population will likely exceed that threshold, so the branding terms need a legal review before rollout. Confirm how group-based access to knowledge collections is configured and audited.
Qdrant as the self-hosted vector store
Qdrant is an Apache-2.0 vector search engine that runs in your own infrastructure, so embeddings derived from confidential documents stay there. Self-deployed instances listen on all interfaces with no authentication by default, so API keys, TLS and network restriction must be configured before any production use. Its JWT-based access control can limit read or write permissions per collection, which is the mechanism to pair with source permissions. JSON audit logging of authenticated API operations supports record keeping. Storage must be block-level, not NFS or object storage, and production clusters should run on at least three nodes.
Permission model to confirm per platform
No application option should be accepted on its chat features alone. For each platform, ask the vendor or your implementer to demonstrate that a user who lacks access to a source file receives no chunk, citation or summary from it, including through follow-up questions. Test with restricted policies such as an internal audit file or a deal document. Record the result, since it becomes evidence for information barrier and access control reviews.
How retrieval filtering keeps information barriers intact
The vector index has to carry source-document permissions, because filtering only at the chat layer leaks restricted material. If the assistant retrieves every chunk and the interface hides some answers afterward, the restricted text has already been ranked, passed to the model and may surface through a summary, a citation title or a paraphrase. For a bank or insurer that runs information barriers between deal teams, trading desks or claims units, the control has to sit where retrieval happens. Each chunk should be stored with the access groups of its source document, and every query should be filtered against the requesting user's groups before any ranking takes place.
In Qdrant, the mechanism to design around is per-collection access control. Qdrant supports JWT-based granular access control with per-collection read or write permissions since v1.9.0, so a token can be scoped to the collections a user's groups may read. Map directory groups to collections deliberately: a group from the bank's identity provider should correspond to one collection or a documented set of them, and the mapping should be reviewed by the information barrier owner, not inferred by the platform team. Deal teams, trading desks and insurance claims units usually need separate collections because a single shared index makes per-user filtering the only barrier, which is harder to prove to a supervisor. Confirm with your security team how tokens are issued and how long they live, and whether Qdrant's JWT configuration fits your existing token service.
Permission data goes stale quickly, so decide how often group membership and document ACLs sync into the index. A user removed from a deal team should lose access to that collection at the next sync, and the gap between removal and enforcement should be a documented, tested number that compliance has approved. Test it: add a user to a restricted group, confirm retrieval and citations change, remove the user, and confirm they disappear. Also check that the audit trail is useful. Qdrant has JSON audit logging of authenticated API operations since v1.17.0, but whether those entries are enough to show which user retrieved which restricted chunk depends on how the application forwards user identity, which should be confirmed in a test environment before rollout.
How a financial institution phases a RAG rollout
The sequence moves a RAG assistant from document scoping and third-party mapping through a permission-tested build, a reviewed evaluation, and a controlled pilot before production. Each phase ends with a named output that a risk, compliance or audit reader can inspect.
Scope the corpus and classify documents
Compliance, the business owners and the security team agree which policies, procedures and product documents enter the index, and which are excluded because they hold restricted material such as deal files, customer records or cardholder data. Each source gets a classification label and an owner who approves its inclusion. The step is done when the scope is written as a signed inventory of sources, owners, classifications and excluded collections, with the reason for each exclusion recorded.
Map ICT contracts and the DORA register
Before any component is installed, the institution records each open-source platform, the hosting provider, the embedding or model endpoint and the vector database in its register of information on ICT contractual arrangements. For each component that supports a critical or important function, the team confirms where data will be processed, what notice applies to changes and how data is returned on termination. Where a contract covers a third-party service, the exit approach is documented. The output is an updated register entry set and a contract gap list that procurement and legal must close.
Build in a non-production environment
The implementation partner deploys the chosen stack, such as Dify or Open WebUI with a self-hosted vector database, into a non-production environment inside the institution's own infrastructure. Connectors pull documents with their source access control lists, and the team confirms that the vector store is not reachable from untrusted networks, since Qdrant listens on all interfaces and has no authentication by default unless configured. The output is a build record with versions, configuration, secrets handling and a network diagram signed off by the security team.
Test permission-aware retrieval
Testers sign in as users from each information barrier group and ask questions whose answers exist only in restricted sources. They check that retrieval filters out chunks the user cannot open in the source system, that citations never point to hidden documents, and that cached or summarized content does not leak across groups. Failures go back to the build phase and are retested. The output is a permission test report that lists each role, query, expected result and observed result, and that compliance can rerun after every change.
Evaluate answers against a reviewed question set
Subject matter experts and model risk reviewers write a question set drawn from real policy and product queries, with reference answers and the source passages that should be cited. The team runs the assistant against this set, scores groundedness, citation accuracy, refusals and handling of out-of-scope or conflicting policies, and reviews failures with the model risk function. The set is versioned and frozen before scoring so results can be compared over time. The output is an evaluation report with acceptance thresholds approved by the model risk owner, or a documented decision to keep working before pilot.
Configure logging and record keeping
The team turns on structured logging of user identity, query, retrieved sources, cited passages, model version and output, and defines retention with the records function. Logs are stored in the institution's own storage, access to them is restricted to named roles, and integrity controls are documented. Where the assistant may be classed as high-risk under the EU AI Act, the team checks the event recording and six-month minimum retention requirements in Articles 12 and 19 against its legal position. The output is a logging specification and a test showing that a supervisor can reconstruct a past answer from stored records.
Run a controlled pilot with one business line
One business line uses the assistant for a defined set of document collections and a named user group, with the evaluation set rerun weekly and every flagged answer reviewed. Users report wrong or unsupported answers through a documented channel, and compliance samples logged interactions. Scope expansion requires a written decision by the pilot owner and the model risk function. The output is a pilot closure report covering incidents, evaluation trends, user findings and a go or no-go recommendation for production.
Harden for production and document the exit plan
Before production, the team enables single sign-on and per-collection permissions, configures secrets management and TLS, sets backup and restore tests for the vector store and metadata database, and applies the vendor-specific security settings for each component. It also writes a tested exit plan covering export of documents, embeddings and logs, and the migration path to an alternative platform, consistent with the exit strategy expectations for critical or important ICT services. The output is a production runbook, a restore test record and an exit plan approved by the risk and third-party management functions.
Evidence supervisors and auditors ask for, and what produces it
This table maps each evidence item a bank or insurer typically needs to the component or record that generates it. Read it as a checklist of what to design for before go-live, not as proof that any control is met. DeployOpen prepares this evidence, but auditors and supervisors make the assessment.
| Evidence item | What it shows | What produces it | Where it usually fails |
|---|---|---|---|
| Register entry for each ICT service | Every contractual arrangement for the assistant's components is recorded, as DORA Article 28(3) requires | Your third-party register, populated from the contract inventory for Dify or Open WebUI, Qdrant, the model host and any gateway | Components added during a pilot are never registered |
| Contract clauses on data location and return | Where services are provided and data processed, plus data return on termination or insolvency (DORA Article 30(2)) | Signed contracts and subcontractor terms with support providers, checked against the hosting locations you document | Support access from outside the agreed region is not written into the contract |
| Exit strategy and transition plan | A tested route to move the vector store and application data if a provider fails (DORA Article 28(8)) | Documented export runbook, Qdrant snapshots and exported source files, and a recorded restore test | Snapshots exist but no one has restored them into a second environment |
| Access review for collections | Which users and groups can query each document collection, and whether that matches source permissions | Platform role exports, Qdrant API key and JWT per-collection permission settings, and dated review sign-off | Collections inherit broad access from a shared admin account |
| Audit logs of queries and answers | Who asked what, which sources were retrieved and which answer and citations were returned | Platform application logs, Qdrant JSON audit logs of authenticated API operations, and LiteLLM or Langfuse traces where deployed | Retrieval logs omit source document IDs, so citations cannot be traced back |
| Model risk validation report with test set | Answer quality measured against a fixed set of questions with known source passages | Versioned evaluation test set, run results from your evaluation tooling, and a sign-off record for each model or prompt change | Test sets are reused after tuning, so results overstate accuracy |
| MFA and authentication configuration records | Multi-factor authentication is enforced for every user of the assistant (16 CFR 314.4(c)(5) where that rule applies) | Identity provider policy export, SSO configuration for the platform, and a list of any written compensating-control approvals | Local admin accounts bypass SSO and are not covered by the policy |
What buyers ask before they hire a partner for a financial RAG assistant
What drives the cost of a financial RAG assistant engagement?
Scope drives cost more than any single component. The main drivers are document volume and file types, how complex the permission model is across source systems, the number of collections, how deep the evaluation work goes, and how many integrations are needed with identity, document management and case systems. Costs fall into one-time work (discovery, connector build, permission mapping, evaluation set creation, hardening and documentation) and recurring work (re-indexing, model and software upgrades, monitoring and evaluation runs). Infrastructure is paid to your cloud or data center provider. DeployOpen does not publish a price list, so ask for a written scope that lists each driver separately and the recurring support terms.
How long does it take to reach a pilot for a bank?
The time to pilot depends mostly on how quickly your team can supply the inputs, not on the software. The main dependencies are an agreed document set for the first collection, access to a test identity provider and group model, named document owners who can confirm what may be indexed, and a security review of the hosting environment. Model hosting choices and the vector database also matter, because the vector store must run on block storage, and Qdrant documents that it cannot run on network file systems or object storage. Ask any partner to show the pilot milestones and the client decisions each one needs in the written scope.
What must our own team provide during the engagement?
Most delays come from missing client inputs, so plan these before signing. You will need an identity provider administrator who can supply group membership and attributes for information barrier design, document owners who approve which policy and product collections are in scope, and reviewers from compliance and the business who score sample answers during evaluation. You also need an infrastructure owner for the hosting environment and a security contact for access reviews and logging requirements. Ask the partner to list these roles, the time each role is expected to commit and who signs off each stage.
Can our internal staff run the assistant after handover?
They can, if the handover is planned as a skills transfer rather than a document drop. A working team usually needs to operate the platform (upgrades, backups, user administration), maintain the ingestion pipelines and re-indexing jobs, run the evaluation set and review regressions, and answer audit requests for logs and records. Ask for runbooks, configuration documentation, restore tests and named training sessions, and confirm which tasks remain with a retainer. DeployOpen offers ongoing support, upgrades and continuous testing as retainers, so the split between internal and external ownership should be written into the scope.
How do license terms affect the choice of platform?
License terms can rule out a platform before technical comparison begins. Dify's license forbids using its source code to operate a multi-tenant environment without written authorization, where one tenant is one workspace, so a bank that serves several subsidiaries or client groups as separate workspaces should check this with counsel. Dify's self-hosted Community edition is limited to a single workspace, and multiple workspaces, SSO and a commercial license are part of the Enterprise edition, which is custom priced. Open WebUI's license requires keeping its branding unless a listed exception applies, including deployments with no more than 50 end users in any rolling 30 day period. RAGFlow and AnythingLLM are MIT or Apache licensed, so ask counsel to confirm their terms for your case.
Which parts of the work depend on our document permissions?
Permission handling is usually the largest variable in scope. Each source system has its own access rules, and the assistant must filter retrieval so that a user only receives chunks they could open in the source system, including information barrier restrictions between teams. This requires mapping groups from the identity provider, checking how permissions are read at ingestion and at query time, and testing that revoked access takes effect in answers. Ask the partner how they design retrieval filtering, how often permissions are resynchronized, and which test cases show that restricted content never appears in a citation or summary.
What evidence will supervisors and auditors ask for?
Supervisors and internal audit typically ask for the scope of indexed sources, who can access which collections, logs of queries and answers with their citations, model and software versions in use, evaluation results, and records of changes. Logging design matters early. The EU AI Act sets log-recording duties for high-risk AI systems under Article 12(1), with logs kept for at least six months under Article 19(1) unless other law provides otherwise, so confirm with counsel whether your use case is in scope. DeployOpen prepares systems, controls and evidence for compliance assessments but does not issue audit reports or attestations, which independent assessors provide.
Questions to put to an implementation partner for a bank or insurer
Start with permissions, because that is where a RAG assistant most often fails a bank or insurer. Ask the partner to show, for a sample of restricted documents, how the source system's access control list is carried into the index as metadata and enforced at query time, not only at the user interface. Ask which field in the vector store holds the permission, how it is refreshed when a source document's access changes, and whether a citation is ever shown to a user who could not open the underlying document. Ask how the partner tests information barriers: which accounts from each side of a barrier are used, which queries are run, and whether the results are kept as evidence. For a self-hosted Qdrant index, ask whether API keys, per-collection JWT permissions and the JSON audit log are configured, and who holds the keys. Qdrant does not run secured by default, so ask for the hardening record.
Next, ask how answer quality is evaluated and reviewed under the bank's model risk procedures. The partner should describe a versioned test set drawn from real policy questions, the metrics used for retrieval and for groundedness, who signs off on a release, and how a change to the chunking, embedding model or prompt triggers a re-test. Ask how prompts, retrieved passages and answers are logged, and how long the logs are kept. Where the assistant is high-risk under the EU AI Act, Article 12(1) requires automatic event logging over the system's lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law requires longer. Ask the partner which party holds the logs, where they are stored, and how a supervisor could retrieve them. Confirm with counsel which retention period applies to your institution.
Then cover third-party and resilience obligations under DORA. Ask for the contract clauses that state where data is processed, how advance notice of changes is given, and how data is returned on insolvency or termination, since Article 30(2) requires these. Ask for the exit plan and transition plan for the assistant, including how the vector index, the metadata database and the prompt history would be moved to another host, and whether that plan has been tested. Ask how the partner handles prompt injection hidden in uploaded documents, such as instructions embedded in a PDF, and which controls sit at ingestion and at the gateway. Finally, ask what evidence the partner produces, which items are excluded from scope (for example, the model provider, identity provider or document source systems), and who is responsible for each excluded item.
Deployment patterns for regulated data residency
A financial institution's assistant touches four layers: where the platform runs, where vectors are stored, where the model runs, and how the assistant connects to identity and records. Each layer has a residency decision that should be settled in writing before the design review, not after the pilot.
Hosting inside the institution's accounts
Run the assistant, its metadata database, file storage and sandbox services in the bank's or insurer's own cloud accounts, data centers or private cloud. Dify's Docker Compose stack, for example, starts the api, worker, web and plugin services alongside PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and an SSRF proxy, so every one of those components needs an owner and a location. Confirm which of them process document content and where their logs are written.
Block storage for the vector database
Qdrant needs block-level storage with a POSIX-compatible file system and cannot run on NFS or object storage such as S3. SSD or NVMe drives are recommended. Many bank storage standards default to shared NFS volumes, so confirm this with the infrastructure team before the design is approved, because a shared volume will fail the requirement silently at scale.
Three-node minimum for production
Qdrant's distributed mode uses Raft for cluster topology and splits collections into shards with replicas. The docs recommend at least three nodes for production so quorum survives a node failure, and a shard count that is a multiple of the node count. A replication factor of 2 doubles stored data, so size storage and recovery targets with that multiplier. Ask the partner how node placement maps to availability zones or data centers.
Self-hosted model endpoint
Serve the embedding and chat models from infrastructure the institution controls, using an engine such as vLLM or Ollama behind an OpenAI-compatible endpoint. Prompts, retrieved passages and answers then never leave the perimeter. Confirm the GPU or server capacity, the model licenses for commercial use in a bank, and whether the endpoint is reachable only from the assistant's network segment.
Identity, document management and logging links
The assistant should authenticate staff through the institution's identity provider using OpenID Connect or SAML, and read permissions from the document management system so retrieval respects existing access. Logs of queries, retrieved sources and answers must go to the institution's own log store. Ask the partner how the vector database's API keys, per-collection permissions and audit logs are wired into those systems.
How DeployOpen scopes and supports a financial RAG assistant
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For a financial RAG assistant, the work usually starts with a written scope that names the document sources, the identity system that supplies group and barrier attributes, the vector store and model endpoints, and the roles allowed to change retrieval filters. Most engagements are fixed-scope, so deliverables, acceptance tests and timelines are agreed in writing before work begins. Ask any partner to show how permission-aware retrieval is tested against your actual information barriers, not a sample directory.
DeployOpen also runs security testing on the assistant itself. That includes prompt injection tests against documents that have been indexed, attempts to pull restricted chunks through follow-up questions, and checks that citations point to sources the user is entitled to read. Findings should map to the controls you already run, such as access reviews and change management, so the testing output can feed your model risk file. Confirm in the statement of work who owns the test cases after handover and how regression tests are rerun when the chat front end, the embedding model or the vector database version changes.
DeployOpen prepares systems, controls and evidence for compliance assessments, including configuration baselines, access logs, retention settings and records of evaluation runs. It does not issue audit reports, attestations or certifications, and independent auditors or certification bodies make those determinations. Retainers cover upgrades, security patches, re-testing after platform releases and ongoing support of the evaluation set. Before signing, decide which evidence your supervisors and internal audit will expect, and agree in writing which items DeployOpen delivers, which your team operates and how often each is refreshed.
