Private LLM chat for financial services

Private LLM chat for financial services starts with what staff may paste

Many bank and insurer staff already paste client files, deal memos and underwriting notes into public AI tools, often without a policy that says what is allowed. A self-hosted assistant built on Open WebUI or LibreChat, with models served by Ollama or vLLM, keeps those prompts, uploads and chat histories in the institution's own infrastructure, so client and deal material never reaches a public AI service. The first decision is not the interface. It is the written list of data classes that may be entered, the ones that may be uploaded, and the ones that stay out, agreed by compliance, information security and the business before any user receives access.

Financial services face three pressures that most other industries do not. Under DORA, which has applied since 17 January 2025, an assistant that depends on a model provider is an ICT third-party arrangement that must appear in the register of information, and its contract must state where data is processed and how data is returned on termination. If cardholder data can appear in prompts or files, PCI DSS scope may extend to the assistant and the systems around it, so the team should confirm with its QSA or internal assessor whether the chat environment falls inside the cardholder data environment. Supervisors also expect records that can be reconstructed, which the EU AI Act reflects in its Article 12 and Article 19 log duties where a system is high-risk.

On platforms, Open WebUI and LibreChat are both workable front ends, and Ollama or vLLM can serve the models, with LiteLLM often placed between them for keys, budgets and logging. The right combination depends on the written scope: which data classes are in, which user groups need access, whether a model provider is in the loop, and which logs the supervisor will ask for. Confirm the Open WebUI license terms before rollout, since branding changes above 50 users in a rolling 30-day period require an enterprise license or written permission.

Which financial regulations shape the chat build

This table maps each financial regulation or standard to what it asks of the chat platform, the build decisions it drives and the evidence an assessor will expect. Read it as a starting checklist for scoping, then confirm applicability with your compliance team and assessors, since DeployOpen prepares evidence but does not issue attestations.

Requirement	What it means for the build	Evidence to prepare
DORA Article 28(3) register of information	Every ICT arrangement that supports the assistant, including model providers, hosting parties and the gateway vendor, needs a register entry.	Register entries for each provider, with the services covered and the functions they support
DORA Article 28(8) exit strategy	If the assistant supports a critical or important function, document and test how you move models, gateway configuration and chat history to another provider or runtime.	Written exit plan, transition steps and records of the tests run
DORA Article 30(2) contract terms	Contracts must state where services are provided and data processed, require advance notice of location changes, and cover data return on termination or insolvency.	Contract clauses on data location, change notice and data return, reviewed by legal
PCI DSS v4.0.1 scope rule	If staff may paste cardholder data or sensitive authentication data, or the assistant could affect the cardholder data environment, it is in scope. Keep card data out of prompts and files, and segment the chat stack from payment systems.	Data-flow diagram, scope statement and a documented decision on whether the assistant is in scope
FTC Safeguards Rule, 16 CFR 314.4(c)(5)	For non-bank lenders under FTC jurisdiction, MFA is required for anyone accessing an information system unless the Qualified Individual approves an equivalent control in writing. Enforce MFA at the identity provider for SSO logins and for admin consoles.	MFA configuration from the identity provider, written approval for any exception, and access review records
SEC ITGC areas for ICFR (Release 33-8810)	Where the assistant touches financial reporting data or workflows, program change, access to programs and data, and computer operations become relevant. Tie model and gateway changes to tested change control and restrict admin rights.	Change tickets, approval records, role assignments and operations runbooks
Swift CSCF (v2026) where payment systems are involved	If the assistant sits near Swift connectivity or payment operations, its hosts and admins fall within the scope of the controls, and the annual attestation is made by the Swift user, not by DeployOpen. Confirm which controls apply to the chat stack.	Scoping note on Swift-connected systems and the control mapping the Swift user maintains
EU AI Act Article 12(1) and Article 19(1) logs	If the assistant is a high-risk system, it must technically allow automatic event logging over its lifetime, and providers must keep logs for at least six months unless other law requires longer. Confirm the risk classification and set log retention with legal and records teams.	Logging configuration, retention settings per log type and a documented retention decision

Which client and deal data must stay off public AI services

Client identity and account data is the category most staff will paste without thinking, because it appears in every service request, complaint and reconciliation query. Under a self-hosted design, these prompts stay within the institution's infrastructure, but the chat history, uploaded files and any vector store built from them become new records that need the same access rules as the source systems. The classification should decide which fields may appear in a prompt at all, such as masked account numbers or reference IDs, and which must be resolved inside the core system instead.

Pasting: Permit anonymized or masked identifiers, and prohibit full names combined with account numbers, tax IDs or contact details unless the use case has been approved.
Uploads to knowledge bases: Client-level files should not enter a shared knowledge base. Restrict them to per-user or per-case collections with defined owners.
Chat history: Store prompts in the application database (for Open WebUI, PostgreSQL in multi-instance deployments), and apply the same retention and deletion rules as the core systems.
Vector stores: Embeddings of client documents are still client data. Keep them in a self-hosted vector database under the same access groups as the source.
Logs: Gateway and observability logs can capture prompt text, so confirm whether they redact personal data before storage.

Confirm with the data protection officer which identifiers are permitted in prompts before pilot users are onboarded.

Which open-source platforms fit a regulated chat deployment

A regulated chat deployment usually combines a user interface, a model server, a gateway that controls keys and budgets, and a tracing layer that records what happened. Each component below carries a licensing, identity or evidence question that a bank or insurer should settle before choosing it.

Open WebUI for the chat interface

Open WebUI gives staff a ChatGPT-style interface that connects to Ollama, vLLM or any OpenAI-compatible endpoint, with OIDC or LDAP single sign-on, SCIM 2.0 provisioning and group-based roles. The branding clause matters here: deployments above 50 users in any rolling 30-day period may not alter or remove Open WebUI branding without an enterprise license or written permission. Confirm the user count, the branding position with legal, and whether the enterprise option covers the bank's own look and feel.

LibreChat as an MIT alternative

LibreChat is an MIT-licensed chat interface that supports multiple model providers and is a credible alternative where the license must impose no branding restriction on a large user population. It fits teams that want the simplest license review. Confirm which identity options, role model and audit features the chosen release supports for your SSO and retention requirements, since these differ from Open WebUI and should be tested rather than assumed.

Ollama for local models and vLLM for concurrency

Ollama is the simpler choice for running open-weight models on a small set of servers, while vLLM is an inference and serving engine built for higher concurrent load across many staff. Both keep prompts inside the bank's infrastructure. Ask the implementation partner to load-test each option against your peak concurrency, since throughput depends on GPU type, model size and context length. Confirm the license of each model, not only the server.

LiteLLM as the gateway for keys and budgets

LiteLLM sits between the chat interface and model providers to issue virtual keys, set per-user and per-team budgets, apply rate limits, route requests and record logs. For audit, the logs are the key evidence. Its admin UI SSO is free for up to 5 users from v1.76.0, and Enterprise is required beyond that, along with SCIM, audit logs and organization RBAC. Budget this license early, and confirm which features your supervisory review will need.

Langfuse or Phoenix for tracing

Tracing records prompts, outputs, latency and cost so reviewers can reconstruct how the assistant was used. Langfuse is MIT-licensed outside its ee directories, while Arize Phoenix is under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. An internal deployment run by the bank is typically a different case, but confirm that reading with counsel if a partner or service provider operates it for you. Decide early what traces contain, since they may hold personal data.

How DORA ICT third-party rules change the choice of model provider

Under DORA, any external model API, hosting provider or managed platform that supports the assistant is an ICT third-party arrangement, and the choice of model provider becomes a regulatory decision as well as a technical one. Each such arrangement must appear in the register of information required by Article 28(3), and the contract must state where the service is provided and where data is processed, with advance notice of changes and data return on insolvency or termination, as Article 30(2) requires. If the assistant supports a critical or important function, Article 28(8) also expects a documented and tested exit strategy and transition plan. A hosted model API that cannot meet these terms in writing, or that cannot be exited without disrupting staff workflows, is a poor fit for a regulated bank or insurer regardless of its output quality. Ask legal and third-party risk teams to classify the assistant's function before any model contract is signed.

A self-hosted model changes the question rather than removing it. The assistant still depends on software and hosting providers, so the register and contract terms apply to each one: the open-source projects in the stack (for example Open WebUI or LibreChat, Ollama or vLLM, and LiteLLM if it is used as a gateway), any support subscription, and any cloud or data center operator that runs the hardware. For each open-source vendor, confirm the license terms that govern your use, including any restriction on branding or user counts, and whether a commercial agreement carries service levels, security notices and support for upgrades. For each hosting partner, confirm the processing locations, subcontractors, data return and deletion steps, and who holds the keys. Note that DORA sets no certification scheme, so none of these checks produces a DORA certificate; they produce the contract and register evidence that supervisors ask to see.

The exit strategy is where self-hosting helps most, because the weights, configuration, chat history and vector stores can be moved within your own infrastructure if a partner fails. Document a tested path to a second model server or a second hosting provider, including how conversation exports and uploaded knowledge are returned in a usable format. Also confirm what the register of information records for each component, how the version of each open-source release is tracked, and who approves an upgrade of a critical component. Where a threat-led penetration test is part of the wider program, Article 26(7) is the only area where DORA issues an authority-issued attestation, and the ECB states that TLPT and TIBER-EU core requirements are identical, so scope the assistant into that testing plan where it is relevant.

How a bank or insurer phases a private chat rollout

The sequence moves a private chat assistant from a documented scope and data rules to a production service that is governed, logged and tied to the firm's third-party risk process. Each phase closes with an artifact that a risk, compliance or audit reviewer can inspect.

Scope use cases and classify data
Compliance, risk, business owners and the security team agree which functions will use the assistant, such as drafting, summarizing policy or analyzing deal notes, and which data classes each function may touch. Client identifiers, cardholder data, material non-public information and credentials are classified explicitly, with a rule for each class on whether it may be pasted, uploaded or excluded. The output is an approved data-use rule set and a scoped list of user populations. The step is done when the business owners sign off on the rules in writing.
Map controls to DORA and internal policy
The team records the assistant as an ICT service in the firm's register of information, identifies whether it supports a critical or important function, and maps the model provider and hosting arrangements to the contract terms DORA requires, including location of processing and data return. Internal acceptable use, retention and access policies are updated to cover the chat. Where PCI DSS applies because cardholder data is in scope, the team checks whether the assistant could affect the cardholder data environment. The output is a control map, a register entry and a documented exit strategy for the assistant. The step closes when the third-party risk function accepts the map.
Run a pilot in non-production with one SSO group
A small group, selected through a single identity provider group, uses the assistant in a non-production environment that mirrors production configuration. The pilot tests SSO through OIDC or LDAP, group-based roles, file upload restrictions and the behavior of the chosen model server under realistic prompts. Testers also check that logs capture the events the firm needs for supervisory review, and that the fallback path works if the model endpoint is unavailable. The output is a pilot test report listing defects, accepted residual risks and a go or no-go decision signed by the owner. The step closes when every high-severity finding has a fix or a documented exception.
Harden production with shared state and storage
Before production, the platform team moves the chat to a multi-replica design with PostgreSQL in place of SQLite, Redis for shared session and application state, and shared file storage so uploads and knowledge stay reachable from every replica. Secrets move to a secret manager, administrative access is restricted, and backups and restore tests cover the database and file store. The team confirms the retention decision for chat history and model logs against the approved rules and any applicable record-keeping duties. The output is a hardening record, a tested restore and a log retention decision. The step closes when the restore test succeeds and the retention setting matches the signed decision.
Release access in role and group waves
Access expands in waves defined by identity provider groups rather than by individual request. Each wave adds one business area, with role scoping applied to models, knowledge collections and administrative functions. The team checks access reviews, provisioning and deprovisioning through SCIM where it is in use, and confirms that leavers lose access. Usage, error and policy-violation metrics are reviewed before the next wave begins. The output for each wave is an access list approved by the group owner and a short operating review. The step closes when the wave's metrics stay within the thresholds the firm set in advance.
Manage models and releases under change control
Model versions, serving software and the chat interface are placed under change control with a named owner, a test set of approved prompts, and a rollback path. Security advisories and upstream releases are assessed on a fixed review cadence, and updates are tested in non-production before promotion. Changes that alter data flows, providers or logging trigger a new review under the DORA and internal control map. The output is a change log, a model inventory with licenses and versions, and a periodic review record. The step is ongoing and closes only when the firm retires a model or the assistant.

Which evidence supervisors and auditors ask for from the chat platform

Who can access the chat assistant, and how is that proven?

Access is proven by the identity provider and the chat platform's group mapping, not by a list kept by hand. Open WebUI documents SSO through OIDC and LDAP, SCIM 2.0 provisioning of users and groups, and role-based access control with groups and per-resource permissions, so the reviewer can trace each user to a directory group and each group to a role. The evidence set usually includes the group-to-role mapping, a SCIM sync log, a dated export of active accounts, and the MFA policy enforced at the identity provider. Confirm with your IdP team whether deprovisioning is pushed through SCIM or relies on a nightly job, because a delay there is a finding in its own right.

Which models and versions produced each answer?

The gateway record is the primary source, because it names the provider, the model and the deployment that served each request. In a design with LiteLLM between the chat interface and the model servers, the request log can carry the model identifier, the virtual key and the team, which lets you show that only approved models were callable. Models served by Ollama or vLLM need a separate model register that records the weights, version, license and hash, with a change history. Ask the platform team to show that a given week's answers can be tied to a specific model build.

What do gateway audit logs record, and what do they leave out?

A well-configured gateway records who called which model, when, with which key, the token counts and cost, and the outcome of guardrail checks. LiteLLM's enterprise tier adds audit logs and team-level logging, so confirm the exact fields your edition produces before you promise them to a supervisor. Gaps are common: the chat interface may store full prompt text separately from the gateway, and attachments may sit in file storage that the gateway never sees. Map each record type to its store, then decide which one is the system of record for a complaint or investigation.

How is retention of chats, files and logs configured?

Retention is set in three places: the chat database, the file and knowledge store, and the log sink, and each needs its own period and deletion method. The chat interface holds conversation history in PostgreSQL once it runs on more than one instance, so retention jobs and backups must be designed together. Your records policy should state the period for prompts and outputs, the period for gateway logs, and what happens to backups. Confirm with legal and compliance which periods apply, since the EU AI Act sets a minimum of six months for logs of high-risk systems under Article 19(1), and your own regulator may require longer.

How are model and release updates controlled and documented?

Change management for an LLM platform covers three moving parts: the chat application, the gateway and the model weights. Open WebUI publishes releases frequently, with v0.12.3 released on October 16, 2026, so an unpinned container tag can change behavior without a planned change. Pin versions, test in a staging environment with a fixed evaluation set, and keep the approval, the test results and the rollback plan in the change record. Auditors will look for the same discipline they expect from any production system: who approved the change, what was tested, and whether the previous build can be restored.

How are exit, data return and provider changes documented?

DORA Article 28(8) asks for documented, tested exit strategies for ICT services that support critical or important functions, and Article 30(2) expects contracts to cover data return on termination. Because the platform is self-hosted, much of the exit plan concerns your own components: export of chat history, knowledge bases and user records in an open format, and a tested restore into another environment. Model providers used through the gateway need their own exit steps, including how a model is swapped without retraining staff. The register of information under Article 28(3) should list the components and their contracts. Confirm the exact export formats with each vendor before you rely on them.

How are the logs protected from alteration and misuse?

Logs need integrity, restricted access and a retention lock that stops anyone from quietly editing or deleting records. Store gateway and application logs in a separate account or log platform that the chat administrators cannot modify, and limit read access to compliance and security staff. Keep secrets such as the LITELLM_MASTER_KEY in a secret manager, as LiteLLM's production guidance recommends, and record access to that manager. Ask your security team to test whether a platform administrator could alter a log entry without leaving a trace, and document the answer.

Who makes the supervisory judgment, and what does DeployOpen prepare?

The regulator, the internal audit function and the external auditor make the judgment about whether controls are adequate. DeployOpen prepares the systems, controls and evidence that those reviewers will examine, including access configuration, log design, retention settings, change records and exit documentation. DeployOpen does not issue audit reports, attestations or certifications, and it does not decide whether your deployment meets DORA or any other rule. Ask your compliance team to define the evidence list before the engagement starts, so the scope in the written agreement matches what they need to show.

What logging for supervisory review should capture

Supervisory review needs four different record types, and they should not be merged into one log stream. Chat history is what staff and the assistant said, kept per user and per conversation, and it is the record reviewers read first. Admin audit logs record who changed configuration: model access, group membership, SSO mappings, retention settings and file-upload rules in Open WebUI or LibreChat. Gateway spend and access logs come from the LiteLLM layer and show which virtual key, user or team called which model, when, with what token counts and cost, and whether a request was blocked by a budget or model restriction. Tracing data from a tool such as Langfuse or Phoenix shows the internal steps of a request, including retrieval and prompt templates. Confirm with your compliance team which of these your supervisor expects to see, and in what format.

Where the assistant is a high-risk system under the EU AI Act, Article 12(1) requires the system to technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Six months is a floor, not a ceiling. Banking and insurance record-keeping rules, complaint handling and litigation holds often require longer periods, so set retention per record type with your legal team rather than applying one number. Whether this assistant is high risk depends on its use, so classify it before you set the retention schedule. The EU AI Act Article 19 text is the natural starting point for that discussion.

Access to these logs should be restricted and separated from the people being reviewed. Create a dedicated reviewer role in the identity provider and map it to a read-only group in the assistant, with no ability to edit users, models or retention settings. Keep admin audit logs and gateway logs in storage that the platform administrators cannot delete or alter, for example through write-once object storage or a log forwarder that writes to a separate account. Restrict LiteLLM's LITELLM_MASTER_KEY to a secret manager with its own approval path, and log every access to the review tooling itself. Test this design with a tabletop exercise: a platform administrator should be unable to remove a conversation or alter a budget without that change appearing in a record they cannot touch.

Integration points for a bank's existing estate

A private chat assistant only becomes useful and defensible once it connects to the identity, document, data and logging systems a bank already runs. Each integration point below carries a pattern and a decision the institution must make before rollout, and most of those decisions sit with security, data governance and third-party risk teams rather than the platform team alone.

Identity provider and SCIM

Open WebUI documents SSO through OIDC and LDAP, plus SCIM 2.0 for user and group provisioning, so group membership in the bank's directory can drive who sees which models and knowledge bases. The decision is which directory is the source of truth for joiners, movers and leavers, and how quickly a removed employee loses access to chat history and uploaded files. If LiteLLM sits behind the chat, confirm whether your SSO and SCIM needs fall within its free tier or require an enterprise license.

Document repositories for knowledge

Knowledge features need connectors or sync jobs to the bank's document stores, such as shared drives, policy libraries and research repositories. The pattern is usually a scheduled sync that copies only approved folders into a knowledge base, rather than granting the assistant broad read access. The decision is which repositories are in scope, whether source permissions are mirrored into the assistant, and who approves a new source before it is indexed.

Vector database placement

Open WebUI supports 13 vector databases, including PGVector, Qdrant, Milvus and Elasticsearch, so the choice of store is largely about what the bank already operates and can back up. Embeddings derived from client or deal documents are sensitive data in their own right and need the same retention and deletion rules as the source. Decide where the vector store runs, who administers it, and how a document deletion request removes its vectors.

Data catalog and DLP tooling

Data loss prevention and classification tools decide which content may be pasted or uploaded, and a data catalog records which systems hold regulated data. The pattern is to route uploads and pasted content through an inspection step or classification label before indexing, and to tag knowledge sources with their classification. The decision is which classification levels the assistant may process, and whether existing DLP products can inspect chat traffic or only files.

SIEM for log forwarding

Prompts, responses, administrative actions and model access events should reach the bank's SIEM, so supervisory review and incident response do not depend on the chat application's own logs. Article 12 and Article 19 of the EU AI Act describe automatic event logging and a six-month minimum retention for high-risk systems, which is a useful reference even when the use is not high-risk. The decision is what is logged in full, what is redacted, and which retention period the SIEM enforces.

API gateway for internal services

Internal applications that call models should go through the bank's API gateway or an AI gateway such as LiteLLM, rather than holding provider keys directly. The gateway enforces authentication, rate limits and model allowlists, and it gives third-party risk teams one place to show which services reach which model endpoints. The decision is whether to reuse the existing gateway, such as Apache APISIX or Kong, or to run a separate AI gateway, and who owns virtual key issuance and budget limits.

What a private LLM chat for a bank costs and what it needs from the institution

What drives the cost of a private LLM chat for a bank?

The main cost drivers are GPU capacity for the models, the licenses and support you choose, integration work with identity and existing systems, security testing, and ongoing operations. GPU capacity depends on how many staff chat at the same time and which models they need, so ask the partner to size concurrency explicitly in the quote. Open-source software carries no license fee for the core components, but some enterprise editions do, and the Open WebUI License adds a branding condition above 50 users in any rolling 30-day period. Ask for each cost line to be shown separately, with one-time and recurring items marked.

Can a bank build a private LLM chat in-house?

A bank can build it in-house if it has platform engineers, security engineers and operators who can run GPU servers, identity integration, logging and patching on an ongoing basis. The harder part is usually not the first deployment but the continuing work: model updates, release tracking, access reviews, backups and evidence for supervisors. Where the institution lacks that capacity, a fixed-scope partner engagement can cover design, implementation, hardening and documentation, with operations handled under a retainer. Either way, the institution remains responsible for its own ICT risk management under DORA.

How long does a private LLM chat deployment take at a bank?

The timeline depends on the scope agreed in writing before work starts, not on a generic duration. The main factors are the number of user groups and waves, whether a multi-replica setup with PostgreSQL, Redis and shared file storage is needed, the number of identity and data integrations, the GPU procurement lead time, and how many approval gates sit with risk, security and compliance. Ask the partner for a milestone plan with the approval points named, and confirm which steps depend on hardware delivery or on your own change-control calendar.

What does the partner need from risk, security and business owners?

The partner needs decisions and inputs from three groups before design is finalized. Risk owners should confirm which data classes may be entered or uploaded, what the retention period for chat history is, and how the assistant fits into the register of ICT services. Security owners should provide identity requirements, group mapping for SSO, network segmentation rules, and the approach to secrets and key management. Business owners should name the initial user groups, the acceptable use rules and the success measures for each rollout wave. Missing decisions in any of these areas are the most common cause of scope changes.

Is an enterprise license required for a private LLM chat?

Not for the core open-source components, but some features and terms depend on the license. Open WebUI is distributed under the Open WebUI License, which prohibits altering or removing its branding in deployments above 50 users in any rolling 30-day period unless the operator holds an enterprise license or written permission. Open WebUI's enterprise licenses allow white-labeling and dedicated support, and pricing is quoted after contacting its sales team with an expected seat count. On the gateway side, LiteLLM's SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that for SSO and SCIM, audit logs and organization-level RBAC. Ask the partner to list every license condition that applies to the chosen stack.

What should a bank ask for in a fixed-scope quote?

A fixed-scope quote should itemize the work, the deliverables and the assumptions behind them. Ask for the scope of design and architecture, the number of environments, the SSO and group integrations, the logging and retention configuration, the security testing included, and the documentation and evidence delivered. Request the GPU sizing basis, the assumed concurrency and model list, and which hardware is supplied by the bank versus the partner. Confirm what is one-time, what recurs under a support retainer, what triggers a change request, and how upgrades to fast-moving releases are handled. DeployOpen does not publish prices and issues a quote from the written scope.

Do I need security testing before a bank rolls out a private LLM chat?

Security testing is strongly advisable before wider rollout because the assistant accepts prompts, files and identity tokens, and each of these is an attack surface. The testing scope can cover the chat interface, the APIs behind it, the model endpoints, the gateway keys and budgets, and the controls against prompt injection and data leakage. DeployOpen runs application and API testing, cloud and infrastructure testing, and AI and agent security testing, and it can scope these as part of a fixed-scope engagement. Ask the partner to state which findings are retested and how results are documented for the institution's own assurance file.

Questions to ask an implementation partner before signing

Start with the DORA questions, because they decide whether the build can be contracted at all. Ask the partner how the deployment record will feed your register of information on ICT contractual arrangements under Article 28(3), and how the written scope will state where the chat platform, model endpoints, logs and backups physically run and how data is returned if the engagement ends. Ask for the exit plan and transition steps for the chat service, with a named owner for each, since Article 28(8) expects them to be documented and tested for services that support critical or important functions. Ask who in your organization must approve any change of hosting location or subcontractor, and what advance notice the partner will commit to in the contract.

Next, ask how access is scoped in the build. Request the mapping from your identity provider groups to Open WebUI roles through OIDC or SCIM 2.0 provisioning, and ask how deprovisioning is tested when someone leaves. If a LiteLLM gateway is in the path, ask whether SSO for the admin UI is covered by the free tier or needs an Enterprise license, and who holds the LITELLM_MASTER_KEY. Then ask how gateway and chat logs are retained, where they are stored, who can read them, and how they are protected from alteration. Ask the partner to confirm in writing how these logs support supervisory review and how long they are kept, and to confirm that your own legal and records teams have set those periods.

Finally, ask for the evidence and sign-off model. Ask how model updates are tested, approved and rolled back, and who signs off each change before it reaches staff. Ask how security testing of the chat interface, the gateway, the inference endpoints and the retrieval layer will be scoped, and whether prompt injection and data leakage tests are included. Request the fixed-scope document before work starts, listing deliverables, timelines and acceptance criteria. Ask who approves the data classification that determines what staff may paste or upload, and insist that this approval is recorded in the scope. Confirm that the evidence set, such as configuration exports, access reviews and test reports, is delivered in writing. Independent auditors, not the partner, will judge whether it is sufficient.

How DeployOpen scopes and supports a regulated chat deployment

A fixed-scope sequence that moves a financial services chat assistant from written requirements to a tested, documented and supported production service inside the organization's own infrastructure. DeployOpen plans and builds the platform, and compliance readiness work prepares evidence for the institution's own assessors.

Confirm the use cases and data boundaries
DeployOpen meets the risk, compliance, information security and business owners to define which functions the assistant serves, which data classes staff may paste or upload, and which data is excluded, such as client account records or deal material held under restricted access. The output is a written scope that names the in-scope user groups, the data classes, the model providers under consideration and the open items that need confirmation from the institution, including its DORA register entries for any provider. The step closes when the institution signs off the scope.
Design the target architecture and controls
DeployOpen drafts a reference architecture for the chat interface, model serving, the AI gateway, the database, shared storage, identity integration and the logging path, all placed in the institution's own cloud accounts, data centers or private network. The design maps each control to the institution's identity, logging and third-party requirements, and records where model inference runs and where chat history is stored. The output is an architecture document and a control matrix that the institution's architects and security team review before any build starts.
Build and integrate the platform
DeployOpen installs and configures the chat interface, model serving and gateway components, then connects single sign-on through the institution's identity provider, group-based role scoping and the agreed retrieval or file features. Configuration is kept as code where practical so the environment can be rebuilt and reviewed. The step closes when a test environment runs the agreed user journeys with each group seeing only the models and features its role allows, and the build record lists every component version in use.
Test the AI components and access paths
DeployOpen runs security testing against the assistant, its gateway, its APIs and its retrieval path, covering authentication, authorization between groups, handling of uploaded files, prompt injection attempts and data leakage between sessions. Findings are logged with severity, owner and retest status. The output is a test report and a remediation list, and the step closes when the institution accepts the residual risk in writing or the agreed fixes are retested.
Harden, document and prepare evidence
DeployOpen applies the hardening baseline, documents the operating procedures, access review steps, backup and restore procedures, and the logging and retention settings, and maps the evidence to the control areas the institution's internal audit and assessors will examine. This is compliance readiness work: DeployOpen prepares systems, controls and evidence, but it does not issue audit reports, attestations or certifications, and independent auditors make those determinations. The step closes when the evidence pack is handed over and reviewed with the institution's control owners.
Roll out in controlled waves
DeployOpen supports the institution's go-live by enabling user groups in phases, monitoring usage, performance and access exceptions at each wave, and recording issues and changes. Rollout happens only after the institution approves the previous wave, so the institution keeps decision rights over who gets access. The output is a rollout log and a signed-off production baseline that records the configuration, component versions and open risks at go-live.
Support operations under a retainer
After go-live, ongoing support, upgrades, patching, backup verification and continuous testing can be covered by a retainer agreed in writing. DeployOpen tracks component releases, plans upgrades with the institution's change process, retests after each change and keeps the documentation current. The step closes with a recurring service review that confirms the production baseline still matches the documented controls and that any changes to model providers are reflected in the institution's third-party records.

Private LLM chat for financial services