Why EU data sovereignty changes the private LLM chat decision
Staff already paste EU personal data into hosted assistants, and the DPO or CISO now has to decide whether that can continue. A self-hosted chat assistant in EU-controlled infrastructure removes most processor and third-country transfer questions, because prompts, uploaded files and chat history never leave the organization's own environment. In practice the common stack is Open WebUI or LibreChat as the interface, with models served by Ollama or vLLM on hardware the organization or an EU operator controls. The question is sharper in Europe than in a generic private chat project. Under the EDPB Guidelines 05/2021, remote access from a third country, even only displaying data on a screen for support, counts as a transfer, and so does storage in a cloud outside the EEA. Schrems II invalidated the Privacy Shield, and the Data Privacy Framework that replaced it rests on a decision now under appeal as Case C-703/25 P, so a US-hosted assistant depends on a legal basis that may change.
The reach of US law is the second issue. The CLOUD Act provision at 18 U.S.C. 2713 requires US providers to preserve or disclose data in their possession, custody or control wherever it is stored, so an EU region run by a US provider may not settle the question. GDPR Article 48 limits when a third-country order can be enforced in the EU, but it does not remove the provider's conflict between that order and the GDPR. Sovereignty therefore depends on who operates the platform, who holds administrative and support access, and which legal orders can compel those operators, not only on server location. Before choosing a model host, the DPO should confirm with the vendor and the operator in writing who can access logs, backups and support sessions, and which jurisdiction governs them.
For the decision itself, the reader should map every component of the chat stack to an operator and a jurisdiction: the interface, the inference endpoint, the vector store for uploaded documents, the database holding chat history, and the observability tooling. Each one that a non-EU provider can reach is a question to resolve before rollout. Open WebUI's SSO, role and group controls help confine access to staff who need it, but they do not change who can reach the underlying infrastructure. Keep the mapping as part of the record of processing and the transfer assessment, so the choice of platform can be defended when a supervisory authority asks how the data stays in the EU.
Which EU rules shape the chat architecture
This table maps each EU rule that bears on a private chat assistant to what it means for where the platform is hosted, who can access it and how logs are designed. Read it as a list of design questions to settle before choosing a model, since most of the rules turn on who can reach the data, not only where it sits. The main takeaway is that EU hosting alone does not settle transfer or disclosure exposure, so operator control and access paths need to be documented.
| Rule | What it means for hosting, access and logs | What to confirm before build |
|---|---|---|
| GDPR Article 44 and onward transfers | Any transfer of personal data to a third country, including onward transfers, must meet Chapter V. Prompts, uploads and chat history stored or reachable from outside the EEA count as transfers. | Confirm every location where prompts, embeddings, files and backups are stored or processed, including support tooling. |
| EDPB Guidelines 05/2021 (version 2.0) | Remote access from a third country, even only displaying data on a screen for support, is a transfer. Storage in a cloud outside the EEA is also a transfer. | Map vendor support and administrator access paths, not just storage regions, and restrict them to approved locations. |
| EDPB Example 12 on EU processors | An EU processor subject to third-country law with extraterritorial effect is not itself a transfer, but complying with a third-country authority request would be. | Ask the operator to assess its exposure to third-country orders before contracting. |
| GDPR Article 48 limit on disclosure orders | A third-country court or authority order to disclose personal data is enforceable only under an international agreement such as a mutual legal assistance treaty. | Ask whether the controller can demonstrate that disclosure requests would be handled through such agreements. |
| CLOUD Act reach under 18 U.S.C. 2713 | US providers must preserve or disclose data in their possession, custody or control, wherever it is located. A US-owned operator can be reached even when data sits in an EU region. | Check corporate ownership and control of every operator and subprocessor, and whether they hold keys or data. |
| Data Act Article 32 safeguard (Regulation (EU) 2023/2854) | Data processing service providers must take measures to prevent third-country governmental access to non-personal data held in the Union where it conflicts with EU or national law. | Ask the provider for its documented technical, organizational and legal measures and how they are tested. |
| Data Act switching rights (Articles 25, 29 and 50) | Customers may move to another provider or to on-premises infrastructure. Switching charges are banned from 12 January 2027, so exit paths for models, chat history and vector stores should be designed early. | Confirm export formats for conversations, users, knowledge bases and embeddings in the contract. |
| AI Act Article 12(1) event logging | High-risk systems must technically allow automatic recording of events over their lifetime. Chat platform logs must capture the events the system design needs, stored under access controls. | Determine whether the assistant could fall under high-risk use before finalizing what is logged. |
| AI Act Article 19(1) log retention | Providers must keep those logs for at least six months unless other law provides otherwise. Retention periods for chat history must reconcile this minimum with GDPR storage limits. | Set one retention schedule that satisfies both the AI Act minimum and GDPR data minimization, and document the reasoning. |
Which open-source platforms fit an EU-sovereign chat stack
An EU-sovereign chat stack is only as sovereign as its weakest component. The platforms below cover the interface, the local model runtime, the shared serving layer and the optional gateway, and each carries its own license and operating model. Check the license and the hosting plan together before committing to any one of them.
Open WebUI for the chat interface
Open WebUI provides the ChatGPT-style interface, with SSO through OIDC and LDAP, SCIM 2.0 provisioning, and role-based access control with groups. For multiple replicas, it needs PostgreSQL instead of SQLite, plus Redis and shared file storage. Its license, however, prohibits altering or removing Open WebUI branding in deployments above 50 users in any rolling 30-day period unless you hold an enterprise license or written permission. Confirm the branding clause with counsel before rollout to staff.
LibreChat as an MIT alternative
LibreChat is an AI chat interface licensed under MIT, which removes the branding restriction that applies to Open WebUI above 50 users. It is a practical option where the organization wants a permissive license for its own interface and branding. Confirm the identity integration, group controls and audit logging you need against the current release, since these features differ from Open WebUI and need to be verified in your environment.
Ollama for local model runtime
Ollama runs open-weight models on a single server or workstation under the MIT license. It suits pilot groups and departmental deployments where a small number of concurrent users share a few GPUs. Confirm its concurrency behavior against your expected peak load, because it is not built as a high-throughput shared serving layer. Keep the endpoint on an internal network and never expose it directly to users.
vLLM for shared OpenAI-compatible serving
vLLM is an Apache-2.0 serving engine that exposes OpenAI-compatible endpoints for chat and embeddings and supports tensor and pipeline parallelism across GPUs. Its security documentation says the --api-key option does not cover every endpoint, so place it behind a reverse proxy that allowlists only user-facing paths. Multi-node communication is insecure by default and must sit on an isolated network. Ask any partner how they will enforce both.
LiteLLM as an optional gateway
LiteLLM sits between the chat interface and the model servers to issue keys, track costs, apply guardrails and log requests in one place. It is MIT licensed except for its enterprise directory, so confirm which features you need fall outside that carve-out. Adding a gateway is optional for a single-model deployment, but it helps where several departments need separate budgets and audit trails. Decide where its logs are stored, because they will contain prompt metadata.
Dify and Phoenix license checks
Dify uses a modified Apache License 2.0 that requires a commercial license to operate a multi-tenant service from the source code, among other conditions. Phoenix is licensed under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service. Neither is an automatic exclusion, but either can conflict with a partner-hosted or shared-service operating model. Have legal review the license against your hosting plan before any component reaches production.
Who operates the stack determines whether the data is sovereign
Hosting in an EU region does not make data sovereign on its own. What matters is who can reach the data in the clear. If a parent company outside the EU, a managed service provider or a vendor's support engineers can view prompts, uploaded files or chat history from another country, the EDPB treats that as a transfer. The EDPB's Guidelines 05/2021 state that remote access from a third country counts even when someone only displays the data on a screen for support or administration. For a self-hosted chat stack, the practical test is whether anyone outside the EU can log in to the Open WebUI or LibreChat host, the vector store, the PostgreSQL database, the model servers or the backup target. The same guidelines (Example 12) say that using an EU processor subject to third-country law with extraterritorial effect is not in itself a transfer, but complying with a third-country authority's request would be. Controllers should assess that exposure before they engage such a processor. The EDPB's Recommendations 01/2020 add a hard limit: where a provider needs data in the clear and the destination's public authorities can reach it beyond what is necessary and proportionate, the EDPB cannot envision an effective technical supplementary measure. Encryption at rest will not solve that case, so the decision is about who operates the stack, not only where it sits. Write down, before go-live, the people and systems that hold each control: administrator and root accounts on the chat and inference hosts, the owners of encryption keys and key management, the staff of any support or managed service contract, and where backups are stored and who can restore them. Confirm with counsel whether each operator's jurisdiction, contract terms and access paths create a transfer or a disclosure exposure, and record the answer in the processing records. Those points are contractual and jurisdictional, so counsel should decide them, not the platform team.
How an EU sovereign chat rollout moves from assessment to staff use
The sequence turns an unmanaged use of US-hosted assistants into a controlled, EU-operated chat service with documented transfer analysis, data rules, model licensing, access control, logging and a staged release. Each phase produces a record that legal, security and business owners can review before the next phase starts.
Map current assistant use and transfer exposure
Start by finding out which public or US-hosted assistants staff already use, through approved tools, browser extensions, plug-ins and personal accounts. Legal counsel and the data protection officer then assess each service for third-country transfers, including remote access from outside the EEA, storage outside the EEA, and whether the provider could be compelled by a non-EU authority to disclose data. The output is a register of services, the data types each one receives, and a documented decision on which uses must stop or move. Ask the provider for its transfer mechanism and its subprocessor list, and confirm both in writing.
Classify the data staff may paste and upload
Define which categories of information may enter the assistant, which may enter only with conditions, and which are excluded. Classification should cover personal data under GDPR, client or employee records, contracts, source code and any special categories. The output is a short classification table tied to the acceptable use policy, plus decisions on file upload types and maximum sizes. Compliance and business owners sign off, and the table becomes the reference for access rules and user training in later steps.
Design EU hosting and operator control
Choose the hosting location and the operators who run the stack, then document who can reach the prompts, files, vector stores, databases and backups. This covers the cloud account or data center, the EU-established company that operates it, support access paths and encryption key ownership. Where a foreign-owned operator is involved, record the extraterritorial law exposure and the assessment that supports the engagement. The output is an architecture and operating model diagram with access paths marked, reviewed by security and legal before procurement.
Select open-weight models and review licenses
Shortlist open-weight models that meet the quality, language and context needs of the defined use cases, and check the license of each model and its serving software separately. Confirm whether the license permits internal commercial use, restricts acceptable use, or adds conditions on redistribution. Interfaces such as Open WebUI and LibreChat connect to local model servers, so their license terms, including any branding clause for Open WebUI above 50 users in a rolling 30-day period, are reviewed too. The output is a model register with license notes and a named owner for each model version.
Integrate identity, groups and access roles
Connect the chat interface to the corporate identity provider using the supported single sign-on and provisioning methods, then map directory groups to chat roles, model access and file permissions. Enforce multi-factor authentication for administrators and confirm how API keys are issued, rotated and revoked. Inference endpoints stay behind a reverse proxy that exposes only user-facing routes. The output is a tested access matrix and evidence that a leaver loses access to the assistant and its stored chats in the same cycle as other systems.
Configure logs and retention for AI Act records
Set retention for chat history, uploaded files, retrieval records and model server logs according to the classification table and the processing purpose. Where the assistant may be classed as a high-risk system, configure automatic event recording so that logs are kept over the system lifetime, with records retained for at least six months under Article 19(1) unless other law sets a longer period. Under Article 12(1), the system must technically allow this recording. Whether any use falls into a high-risk category is a legal determination to confirm with counsel. The output is a retention schedule, a log inventory, and a tested deletion process for chat content.
Run a controlled pilot with a defined group
Select a small group of users from business areas that handle the lower-risk categories from the classification table. Train them on the acceptable use policy, the AI literacy expectations and the rules for files, then measure response quality, access problems, incident reports and how often staff attempt to paste excluded data. Test prompt injection and data leakage scenarios against the file and retrieval paths before expanding. The output is a pilot report with findings, fixes applied and a go or hold decision signed by the data protection and security owners.
Release to staff in controlled phases
Expand access in waves by business unit or directory group, adding model options, data categories and file types only after the previous wave meets the agreed measures. Each wave repeats the access review, reviews logs for policy exceptions and confirms that the transfer register and records of processing still match the live service. Model and software updates follow a tested change process with a rollback path. The output is a release record for each wave and a standing review calendar that keeps the assessment, classification and license register current.
What a DPO, auditor or regulator will ask to see
This table lists the evidence a data protection officer, auditor or regulator is likely to request for an EU-hosted staff chat assistant, and the component that produces each item. Read it as a preparation checklist: the owner column shows where the record must be generated and kept, and DeployOpen prepares this evidence but does not issue audit reports or certifications.
| Evidence | What it shows | Component that produces it |
|---|---|---|
| Records of processing | Purposes, categories of data in prompts and files, recipients, retention and the locations where chats are stored | Controller's register of processing, populated from the Open WebUI or LibreChat deployment design and the data inventory |
| Transfer impact assessment inputs | Confirmation that no personal data leaves the EU, and an analysis of any third-country operator or support access | Infrastructure inventory, operator list and the written assessment of any remote support path |
| Operator access register | Named staff and vendor personnel with administrative or database access, and whether each person is located in the EEA | Identity provider group membership, PostgreSQL and host access logs, and the vendor support access log |
| Data residency configuration | Region of compute, object storage, vector database, backups and model weights, plus proof that replicas stay in the EU | Infrastructure-as-code state, cloud or data center account settings, and the storage and backup configuration files |
| SSO and SCIM provisioning records | Which identity provider federates users, how joiners and leavers are deprovisioned, and when accounts were disabled | Open WebUI OIDC or LDAP configuration and SCIM 2.0 provisioning logs from the identity provider |
| RBAC configuration | Groups, roles and per-resource permissions that limit who can use models, upload files or view shared knowledge | Open WebUI group and permission settings, exported and reviewed on a fixed schedule |
| Retention settings and log retention evidence | How long chats, uploaded files and audit logs are kept, and that deletion runs as configured | Open WebUI retention settings, PostgreSQL retention jobs and the log store; confirm each period with the controller's policy |
| Inference endpoint allowlists | Which vLLM or Ollama routes users can reach, and that administrative and metrics routes are blocked from the user network | Reverse proxy allowlist configuration, since the vLLM --api-key option does not cover every endpoint |
| Model and version inventory | Which open-weight models, licenses and software versions run in production and when each was changed | Deployment manifests, container image digests and the change record for each model update |
What do buyers ask about EU sovereign private LLM chat?
What drives the cost of a self-hosted EU private LLM chat?
Cost is driven mainly by GPU capacity, operator staffing and integration work. GPU servers or cloud GPUs in EU regions are paid to the infrastructure provider, and they scale with the models you serve and the number of concurrent users. Inference software such as vLLM has no paid edition, so its license cost is zero, but the accelerators, the surrounding infrastructure and the people who run them are real costs. Integration with your identity provider, document sources and logging adds implementation effort. Ask any partner for a quote that separates one-time work (design, deployment, integration, hardening) from recurring work (operations, upgrades, model updates, support), and state the assumed concurrency and model sizes so the GPU line can be checked.
How long does an EU private LLM chat deployment take?
Duration depends on scope and the number of users. A pilot for a small group on a single model, with SSO and a defined data policy, is much shorter than a rollout to thousands of staff across several replicas, with PostgreSQL, Redis, shared file storage and a vector database. Work that often extends timelines includes GPU procurement or lead times, data residency confirmation, DPIA input, identity group design and phased waves of access. No duration is given here because no researched figure applies to your environment. Ask a partner for a written scope with phases, dependencies and the acceptance criteria for each wave before work starts.
What does our internal team need to provide for the project?
The internal team provides decisions and access that only your organization can give. Data owners should classify which data may be pasted or uploaded into the assistant. Identity administrators need to supply the SSO connection, group structure and SCIM provisioning rules. The data protection officer should review the processing, retention settings for chat history and any transfer assessment, and the security team should confirm logging and access controls. Infrastructure owners must confirm the EU locations, network segments and GPU hosts. Agree in writing who approves each decision, because delays on these items usually move the schedule more than the technical build does.
Can our own team run an EU sovereign private LLM chat without a partner?
Your team can run it alone if it already operates containers, Kubernetes or GPU servers, has identity engineering skills and can staff on-call coverage. Open WebUI supports Docker and Docker Compose, Kubernetes with Helm, and pip installation, and vLLM is Apache-2.0 licensed with no paid edition. The harder parts are sizing GPUs for concurrency, securing inference endpoints (vLLM documents that its --api-key option does not protect every endpoint, so a reverse proxy allowlist is recommended), applying frequent releases and keeping logs for the retention periods your policy sets. Many teams split the work, keeping operations in-house and buying design review, hardening or retainer support from a partner. Decide this against a written skills inventory.
Does the Open WebUI branding license apply above 50 users?
Yes, the branding clause can apply. Since v0.6.6, Open WebUI code is under the Open WebUI License, which forbids altering or removing Open WebUI branding in deployments that exceed 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written contributor permission. Code through v0.6.5 remains BSD-3-Clause. Open WebUI offers enterprise licenses that allow white-labeling and custom branding, with dedicated support options, but pricing is not published. You request a quote from Open WebUI enterprise sales with an expected seat count. Have legal review the license text against your user count before you commit to a version.
Do Data Act switching rights matter for exit planning?
Yes, they give you a documented route out. The EU Data Act, applicable since 12 September 2025, gives customers the right to switch from a data processing service to another provider or to on-premises ICT infrastructure, with a maximum two-month notice period and a 30-day transition under Articles 25, 29 and 50. Switching charges are banned from 12 January 2027. These rights apply to data processing services you buy, so a managed hosting provider may be covered, while your own self-hosted open-source stack is not a service you switch from. Keep exit planning concrete: export chat history, knowledge stores and configuration in open formats, and confirm the contract terms with counsel.
Which open-source components should we confirm before committing to a version?
Confirm the license and release status of each component before you commit. Open WebUI and vLLM are central to most designs, and their licenses differ, so check each one against your user count and any hosting arrangement. LiteLLM is MIT licensed except its enterprise directory, and Langfuse is MIT except its ee directories, so confirm which features you need fall outside those paths. Ask the partner to pin versions, document upgrade cadence and show how they test a new release before it reaches users. Version numbers change frequently, so take the current tag from each project's repository at the time of quoting.
What the AI Act asks of the organization that deploys the assistant
For a staff assistant, the deployer side of the EU AI Act comes down to two questions: whether the people using the assistant have the AI literacy the organization needs, and whether the organization's own build choices make it a provider. The literacy question is practical. Staff who paste files and ask questions need a short, role-based course that covers what may go into the assistant under your data classification rules, how to check model output before acting on it, and who to contact when something looks wrong. Keep dated completion records alongside the rollout plan, because those records are what you would show if asked how the duty was met. Confirm with counsel which staff groups the duty covers and what content the regulator would expect.
The provider question turns on what the organization does beyond running the platform. Installing Open WebUI or LibreChat, pointing it at a self-hosted model, and giving employees accounts is a different position from building and offering an AI system under your own name. Decisions that deserve a legal review before the build starts include whether you fine-tune or otherwise modify a model, whether you wrap the assistant in a product that you offer to customers or partners, and whether you place a system on the market or put it into service under your name. Record each decision and its reasoning in the design file, and ask counsel to confirm the position in writing.
If counsel concludes that a system falls within the high-risk category, the logging duties become concrete. Article 12(1) requires that such systems technically allow automatic recording of events over their lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. For the chat stack, that means checking that prompts, model version, user identity and key events can be logged and exported, and that retention settings in the chat interface and gateway do not delete records before the required period ends. Whether the staff assistant is in scope at all is a question for counsel, and the EU AI Act Service Desk text on Article 19 is the starting point for that discussion.
What to ask an implementation partner before signing
Where does every component run, and who holds administrator and key access?
Every component, including the chat interface, the database, the vector store, the model servers, backups and logs, should run in cloud accounts or data centers located in the EU and operated by EU-controlled teams. Ask the partner to produce a written component inventory with the hosting location and the named administrator groups for each one. Then ask who holds the encryption keys, where they are generated, and whether any vendor support channel can reach the systems. Remote access from a third country, even view-only support, counts as a transfer under the EDPB guidance, so the access model belongs in the scope document.
How do you assess transfers and third-country government access?
A good partner will separate three questions: whether any personal data leaves the EEA, whether any party with a non-EU parent could be compelled to disclose data, and whether the chosen hosting provider is subject to laws such as the US CLOUD Act. Ask for a written transfer assessment that covers each supplier in the stack, including the hosting provider, backup vendor and any support provider. Ask how they treat the Data Privacy Framework, which covers only certified US organizations and is subject to a pending appeal, and whether they will document the reasoning your legal team needs for a Chapter V review. Confirm with your counsel which conclusions depend on the facts of your own deployment.
How does Open WebUI or LibreChat connect to our identity provider?
Open WebUI documents single sign-on through OIDC and LDAP, plus SCIM 2.0 for user and group provisioning, so role mapping can follow your directory groups. Ask the partner which protocol they will use, how group claims map to roles, and how deprovisioning is tested when someone leaves. Ask how role-based access controls are scoped for knowledge bases and shared models, and whether MFA is enforced by the identity provider or by the application. For LibreChat, ask the same questions and request evidence of the exact configuration on your version, since authentication options differ by release and should be confirmed in the current documentation.
How are vLLM endpoints protected given its documented limits?
The vLLM --api-key option does not cover every route, and the project's security documentation says not to rely on it alone. Ask the partner to place a reverse proxy in front of the inference server that allowlists only the endpoints users need, such as /v1/chat/completions, and blocks control and health routes from user networks. Ask how /metrics is exposed to monitoring only, how nodes in a multi-node deployment are isolated because inter-node traffic is insecure by default, and how the gateway's keys are rotated. Request a test report showing that blocked routes return denial from a user network.
How long are logs kept, and who can read them?
The EU AI Act sets a floor for high-risk systems: Article 19(1) requires providers to keep automatically generated logs for at least six months unless other law provides otherwise, and Article 12(1) requires the system to technically allow automatic event recording over its lifetime. Whether a staff chat assistant is high-risk depends on its use, so confirm that with counsel. Ask the partner to set one retention period per log type, covering access logs, chat history, gateway logs and model server logs, and to explain how deletion requests for chat history are carried out in backups. Ask who can query the logs and whether access is itself logged.
How are model and software updates tested before they reach users?
Open WebUI and vLLM both ship releases frequently, and vLLM publishes new minor versions about every two weeks, so an untested update can change behavior for every user at once. Ask the partner for a written change process: a staging environment that mirrors production, a fixed evaluation set for each model before promotion, a rollback path that restores the previous model weights and container image, and a record of which version each environment runs. Ask how security advisories are triaged and how long an emergency patch takes to reach production under the agreed support model.
How are the written scope and the exit path defined?
Most engagements are fixed-scope, so ask for a scope document that lists the components, the hosting locations, the integrations, the acceptance tests and the deliverables, with the timeline agreed in writing before work starts. For the exit path, ask what happens to the code, configuration, documentation and data if you change partners or bring the platform in-house. The EU Data Act gives customers of data processing services a right to switch to another provider or to on-premises infrastructure, with a maximum two-month notice period and a 30-day transition, so ask how your contract reflects those terms. Confirm which recurring support costs apply after handover.
How DeployOpen scopes and operates a sovereign chat deployment
The sequence turns an EU data sovereignty requirement into a written scope, a deployment inside your own EU infrastructure, and evidence your compliance team can hand to an auditor or assessor. DeployOpen prepares the controls and records, while independent auditors and certification bodies make any formal determinations.
Agree data, operator and model decisions in writing
Before any build work, DeployOpen and your team record which data classes may enter the assistant (prompts, uploaded files, chat history, logs), which legal entity and personnel operate the stack, which regions and providers host it, and which open-weight models are approved. The output is a signed scope document with deliverables, exclusions and dates agreed in advance. Your DPO and legal team should confirm the processor and transfer analysis in this document, since DeployOpen does not give legal opinions on GDPR positions.
Confirm the EU hosting and operator chain
Your infrastructure team and DeployOpen map every component that could touch chat data: compute, storage, backups, support access, DNS, container registries and model weight sources. Each element is checked for location and for remote access from outside the EEA, because the EDPB guidelines treat remote viewing from a third country as a transfer. The check closes with a written inventory of locations and access paths that your team signs off.
Build the stack in your EU environment
DeployOpen installs Open WebUI or LibreChat, the chosen inference server such as vLLM or Ollama, and the supporting database and shared storage inside your own cloud accounts or data centers in the EU. Models are loaded from weights you have verified and approved. The step ends with a working staging environment that passes the agreed functional checks for SSO, groups and model routing, and with infrastructure defined as code so the build can be reproduced.
Harden the inference and chat endpoints
DeployOpen restricts the endpoints users and applications can reach. vLLM's own --api-key option does not cover every route, so a reverse proxy is placed in front of the server to allowlist only the endpoints meant for users. Inter-node traffic in multi-node deployments is isolated on a private network, and admin and metrics routes are kept off user networks. The output is a configuration review and a list of exposed routes that your security team accepts.
Test prompt injection, access and data leakage
DeployOpen runs security testing against the assistant, including prompt injection through uploaded files, attempts to reach other users' chat history or knowledge collections, and checks that administrative functions are restricted to the right groups. Findings are ranked, fixed or accepted by your team, and retested. The closing artifact is a test report with the scenarios run, results and remediation status.
Prepare documentation and compliance evidence
DeployOpen assembles the records your assessors will ask for: the data flow and location diagram, processing and retention settings, access control configuration, logging and log retention settings, the model and license register, backup and restore procedures, and the AI literacy and acceptable use materials for staff. The evidence supports your own GDPR, AI Act and other assessments. DeployOpen does not issue audit reports, attestations or certifications; those come from independent auditors and certification bodies that you engage.
Hand over and agree optional retainer support
Your operations team receives a runbook, a trained handover session and the access and configuration records needed to run the platform. If you want ongoing help, DeployOpen can offer a retainer covering upgrades, model updates, continuous testing and support, scoped and priced separately in writing. The handover is complete when your team has performed a restore test and a model update in the staging environment without DeployOpen involvement.
Sources and further reading
- Regulation (EU) 2016/679 (GDPR), EUR-Lex ↗
- EDPB Guidelines 05/2021 on the interplay between Article 3 and Chapter V of the GDPR (version 2.0) ↗
- EDPB Recommendations 01/2020 on measures that supplement transfer tools (version 2.0) ↗
- 18 U.S.C. 2713, Required preservation and disclosure of communications and records, GovInfo ↗
- Open WebUI License ↗
- vLLM online serving (OpenAI-compatible server) ↗
