Why software companies need a private LLM chat before engineers paste code into a public chatbot
A software company needs a private LLM chat before engineers paste code into a public chatbot, because the data that leaves the company in those prompts is often the company's most valuable and most sensitive material. Engineers already use public chatbots for code review, debugging and incident write-ups, and the security team usually cannot see which source files, stack traces, tenant identifiers or customer records were pasted, or which vendor retained them. A self-hosted assistant does not stop engineers from getting help. It moves that help onto infrastructure the company controls, where access, logging and retention can be set and checked. SaaS firms differ from most other industries in four ways. Source code is the product. Customer tenant data appears in support and debugging sessions. Incident details become customer-facing evidence in post-incident reports and status pages, so inaccurate or leaked text can reach a customer's security team. And customers audit subprocessors, so every external AI service that touches their data may appear on a subprocessor list and in a security questionnaire. Before rollout, confirm with your own counsel and customer contracts which data classes engineers may paste, and which vendors your current subprocessor list already names.
The short answer on platforms is to use Open WebUI or LibreChat for the chat interface, Ollama for smaller internal models that a team can run on modest hardware, and vLLM for shared GPU serving when many engineers use the same model at once. vLLM exposes OpenAI-compatible endpoints, so tools that already speak that API can point at it. Ollama and vLLM are both open source and need no paid edition, so the main costs are hardware, integration and operations. Open WebUI carries a licensing condition to plan for. Its license forbids altering or removing Open WebUI branding in deployments that exceed 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written contributor permission. A software company with a large engineering population will likely exceed that threshold, so confirm the license terms with the project before deciding whether to use Open WebUI, LibreChat (MIT licensed) or an enterprise arrangement.
Requirements a software company's assistant must meet and where they come from
This table maps each requirement that a SaaS or software firm's private chat assistant must meet to the framework or license that creates it, and to what it means for the build. Read it as a checklist for the design review and as a list of evidence to collect before a SOC 2 or ISO 27001 audit or a customer questionnaire. The main takeaway is that most of these requirements reach the assistant through the vendors and open-source licenses you choose, not only through the model.
| Requirement | Where it comes from | What it means for the build |
|---|---|---|
| Security common criteria CC1 to CC9 | SOC 2 (AICPA 2017 Trust Services Criteria): apply to every SOC 2 report | Documented access, change and monitoring controls for the assistant; auditors will sample evidence from its logs and admin changes |
| Trust Services Criteria categories chosen for the report | SOC 2 scope: security is required, while availability, processing integrity, confidentiality and privacy are added by choice | Decide early whether confidentiality and privacy apply to chat history and uploads, since that sets the controls you must evidence |
| Access control and SSO for staff | ISO/IEC 27001:2022 Annex A technological controls on access and authentication | Connect Open WebUI or LibreChat to the corporate identity provider, enforce MFA, and use groups to limit who can reach models and knowledge bases |
| Logging and monitoring of assistant use | ISO/IEC 27001:2022 Annex A technological controls on logging and monitoring | Capture sign-ins, admin actions and model endpoint access; confirm the chat interface can export logs to your central log store |
| Retention settings for chats and uploads | Organizational controls on information deletion and retention, plus GDPR storage limitation if customer personal data appears in chats | Set retention periods per group, document them in the policy, and test deletion on backups and vector stores |
| Approved model list and supplier relationships | ISO/IEC 27001:2022 Annex A organizational controls on supplier relationships | Maintain a list of approved models and their licenses; confirm each model source and inference engine is recorded in the supplier register |
| Subprocessor register | GDPR Article 28(3) processor terms for any vendor that handles customer personal data, plus customer questionnaire requests | Record every hosting, support or model-service vendor that touches prompts or files; a self-hosted model keeps this list short but not empty |
| Processor instructions and security of processing | GDPR Article 28(3): processing only on documented instructions and measures required by Article 32 | Where a vendor is involved, sign terms that bind it to documented instructions and security measures before it touches customer data |
| Log keeping if the assistant is ever high-risk | EU AI Act Article 12(1) automatic event recording and Article 19(1) retention of logs for at least six months | Most staff assistants are not high-risk, but confirm the classification with counsel and design log retention so the duty can be met if the use case changes |
| Branding and license limits above 50 users | Open WebUI License clause 4: no removal of branding above 50 users in any rolling 30-day period without an enterprise license or written permission | Plan the interface branding or a license decision before rollout to all staff, and confirm the terms with the project |
| Switching rights for any hosted vendor | EU Data Act (Regulation (EU) 2023/2854) Articles 23, 25 and 50: customers can switch providers or move to on-premises infrastructure | Confirm exit, export and migration terms in any hosted service contract, and keep chat history and configuration in formats you can move |
Which data and systems the assistant touches in a software company
Source code is the most common thing engineers paste into a chat assistant, and it is also where credentials most often hide. Repositories mix proprietary logic, configuration files, infrastructure definitions and occasionally keys or tokens committed by mistake. A self-hosted assistant keeps that code on infrastructure the company controls, but it does not remove the need to decide what is allowed in. Secrets should be blocked before they reach the model, not cleaned up afterward, because a chat history is one more copy of the value.
- Block secrets at the boundary: Run secret detection on uploads and pasted text where the platform allows it, and treat any match as a blocked request, not a warning.
- Allow code by repository tier: Permit public or internal-only repositories for general assistance and restrict proprietary product repositories to named teams.
- Redact configuration values: Strip connection strings, environment files and certificate bodies from examples before they are shared.
- Inherit repository classification: Files uploaded from a restricted repository and any knowledge base indexed from it should carry the same restricted label, and access to that knowledge base should follow the repository's access list.
- Govern chat history: Set retention for conversations that contain code, and confirm whether deleted chats are removed from the database and file storage.
Confirm with your security team which repositories hold regulated or customer-specific code, and test the secret detection with known-fake credentials before relying on it.
Support tickets, escalations and debugging sessions often include customer tenant identifiers, user records, exported reports and screenshots. This is the data class most likely to trigger contractual obligations, because customer agreements and data processing terms usually limit where that data may go and who may process it. Keeping the model self-hosted narrows the question, but the assistant still stores that data in chat history, uploads and any index built on top of it, so customer data needs explicit rules rather than an assumption that internal means permitted.
- Block raw exports: Prohibit uploading full database exports or unredacted customer records, and provide a sanitized sample format for debugging.
- Redact identifiers: Require replacement of names, email addresses, account numbers and tenant IDs with stable placeholders before pasting ticket content.
- Allow tenant-scoped work: Permit assistance on a single tenant only for staff already authorized to access that tenant's data in the ticketing system.
- Mirror ticket permissions: A knowledge base built from ticket history should not expose one customer's records to staff who cannot see that customer in the ticketing tool.
- Set deletion rules: Align chat and upload retention with the customer contract terms for deletion and return of data.
As an illustrative case, a support engineer might paste a stack trace containing an email address. The rule set should define whether that is redacted, blocked or allowed, and the answer should be the same across the team.
Production logs, traces and incident timelines are among the most useful inputs for an assistant during an outage, and among the most sensitive. Logs can carry IP addresses, session tokens, request payloads and internal hostnames that reveal architecture. Incident timelines often name individuals and describe security weaknesses still being fixed. The assistant should help responders reason about the incident without becoming a second, poorly controlled copy of the log store, so scope and filtering matter more than the model choice.
- Filter before ingestion: Strip authorization headers, cookies and request bodies from log excerpts before they reach the assistant.
- Limit time windows: Allow excerpts covering the relevant incident window rather than bulk exports of a log index.
- Classify incident records: Mark timelines that describe unpatched vulnerabilities as restricted and limit them to the incident response group.
- Control indexing of post-incident material: Do not index postmortems into a general knowledge base until the sensitive findings are reviewed.
- Log the assistant itself: Record who asked what about which incident, since the assistant's own logs become evidence in later reviews.
Confirm how your observability platform exports data, because an assistant connected by API may retrieve more than an engineer would copy by hand.
Runbooks, architecture notes and onboarding guides are the material that makes an internal assistant useful for everyday engineering questions. They are usually classed as internal, but some contain hostnames, break-glass procedures, or steps that reveal how privileged access works. A runbook library is also the natural first source for a retrieval knowledge base, so its classification decisions carry directly into the index. Treating every page as equally shareable is a common mistake that the assistant then amplifies.
- Separate public from internal docs: Keep customer-facing documentation and internal runbooks in distinct sources so the index can be scoped.
- Restrict break-glass material: Exclude emergency access procedures and credential recovery steps from the indexed corpus, or limit them to named administrators.
- Keep source permissions: Apply the wiki or document system's access groups to the knowledge base so a page's visibility is preserved in answers and citations.
- Version the corpus: Record which runbook revision an answer came from, so outdated procedures can be identified and removed.
- Plan re-indexing on change: Deleting or restricting a page at the source should remove it from the index on a defined schedule.
Before indexing, confirm which access groups the wiki uses today, because the knowledge base can only inherit permissions that the source system actually models.
Which open-source platforms fit a software company's private assistant and why
A private assistant for engineers and support staff is usually a stack, not one product: an interface for people, a model server behind it, a gateway that controls keys and logs, and a trace store for prompts and outputs. The options below cover each layer, with the fit and the checks that matter for source code, customer data and incident details.
Open WebUI for the chat interface
Open WebUI gives engineers a ChatGPT-style interface that connects to Ollama or vLLM endpoints. It documents SSO through OIDC and LDAP, SCIM 2.0 provisioning, role-based access with groups, API keys and authenticator-app MFA, which map to the access reviews auditors ask about. Multiple replicas need PostgreSQL and Redis, plus shared file storage, so plan that before a company-wide rollout. Check the branding clause: deployments above 50 users in a rolling 30-day period cannot alter or remove branding without an enterprise license or written permission.
LibreChat as an alternative interface
LibreChat is an AI chat interface under the MIT license, so the license itself imposes no branding or user-count limit. It is a reasonable choice when a company wants a permissive license for the interface layer and expects to connect several model providers. Confirm its authentication options, SSO integration, group-level permissions and audit logging against your identity provider before choosing it, because those features determine whether it meets your access review requirements.
Ollama for small teams
Ollama serves open-weight models locally under the MIT license and is the simplest way to stand up a model for a small engineering group on a single server. Its concurrency is limited, so a team with many simultaneous users or long code-context prompts will see queueing. Use it for a pilot or a small internal team, and test latency with your real prompt sizes. Plan a move to vLLM when concurrency grows, since the serving behavior and operating model differ.
vLLM for shared production inference
vLLM is an Apache-2.0 inference engine that exposes OpenAI-compatible endpoints, so the chat interface and internal tools can point at it without code changes. It publishes Prometheus metrics at /metrics for request counts, token throughput and KV cache use, which feed capacity reviews. The --api-key option does not cover every endpoint, including /metrics and /health, so place a reverse proxy in front that allowlists only the endpoints users need. Multi-node communication is insecure by default and must sit on an isolated network.
LiteLLM as the gateway
LiteLLM sits between applications, the chat interface and model endpoints to issue keys, set budgets, route requests and log calls in OpenAI format. For a software company, this is where per-team keys and usage records come from, and where a single point of control exists when a model is swapped. Its MIT license covers the core, but its enterprise directory is under separate terms, so confirm which features you need are in the MIT portion before you rely on them.
Langfuse or Phoenix for trace records
Tracing tools record prompts, outputs, latency and cost so the company can investigate incidents and show what the assistant did. Langfuse is MIT licensed outside its ee directories and can run in your own storage. Phoenix is under the Elastic License 2.0, which prohibits providing the software to third parties as a hosted or managed service, so a company offering a managed assistant to customers must review that term. Decide retention and redaction of source code and customer data before enabling traces.
Deployment patterns for SaaS: single cloud account, Kubernetes or on-premises GPUs
Most software companies should run the assistant in a dedicated cloud account or subscription that sits under the same isolation rules as production, with its own identity boundary, network controls and logging. The alternative is a set of dedicated GPU servers in a colocation facility or on-premises rack, which suits teams that need fixed capacity or already operate hardware. Either way, the deployment boundary is the evidence boundary: the account, cluster or rack that holds the model, the chat database and the uploads is the scope an auditor or customer questionnaire will ask about, so define it before choosing the platform.
If the model is served across more than one node, the vLLM security documentation is the first thing to read. It states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, so the nodes must sit on an isolated network. Ask the platform team to confirm where Ray or the multiprocessing backend traffic flows, and to document that path as its own system in the architecture diagram. For Kubernetes, the vLLM docs list native manifests, Helm, the vLLM production-stack, AIBrix, KServe, KubeRay, LeaderWorkerSet, llm-d and NVIDIA Dynamo. Each option brings different operators, upgrade paths and admission policies, so pick one that your existing cluster standards already cover.
Run a separate staging assistant with its own model endpoint, database and SSO application registration, and do not reuse production chat history for tests. Staging is where new model weights, Open WebUI or LibreChat upgrades and prompt-injection tests are checked before release, and it should never receive production customer data or source code. Keep the endpoint allowlist, the API key handling and the metrics route identical in both environments so that a passing test means something. Ask your partner to show how a model version moves from staging to production, who approves it, and how the rollback is recorded.
Rollout phases for an engineering assistant from pilot to company-wide use
This sequence moves a private engineering assistant from a scoped pilot to company-wide use, with each phase closing on a written output that security, legal and engineering leads can review. The order keeps data classification and subprocessor records ahead of any wider access, so the evidence an auditor or customer questionnaire will ask for exists before users depend on the tool.
Scope the assistant and classify data with security and legal
Security, legal, privacy and engineering leadership agree what the assistant is for, which data classes may be pasted or uploaded (source code, customer records, incident details, credentials), and which are excluded. Legal reviews the acceptable use policy and the terms that apply to the hosting parties. The output is a signed scope document and a data classification table that the pilot must follow. The step is done when every data class has a stated yes, no or conditional answer with an owner.
Run a pilot with one engineering team
One engineering team uses the assistant on a restricted model list, chosen from the models the platform team has tested for code and reasoning quality and approved for the classes of data in scope. Users are added through a manual account process so the pilot population is known. The team reports failure modes, unsafe prompts and missing features in a shared log. The output is a pilot report with usage data, issues and a recommendation on whether to proceed.
Configure and test SSO, group mapping, logging and retention
The platform team connects the interface to the corporate identity provider through OIDC, maps identity groups to chat roles and model access, and confirms that offboarding removes access. Logging captures user, model, time and administrative actions, and a retention period for chat history and uploaded files is set in line with the data classification. Open WebUI documents SSO through OIDC and LDAP, SCIM 2.0 provisioning and role-based access control, so the team confirms each of these in its own configuration. The output is a test record showing each control working as written.
Complete the subprocessor and model register
The team lists every component that touches prompts, files or logs: the interface, the inference server, the vector database if one is used, the reverse proxy, the hosting provider and any support vendor, with its location and the data it handles. Each model in use is recorded with its license, version and the approved data classes. Legal checks whether any party needs a processor agreement under GDPR Article 28 or a customer-facing subprocessor notice. The output is a versioned register that customer security questionnaires and auditors can be given.
Expand in waves with measured concurrency
Access widens in groups of engineering teams rather than all at once. Before each wave, the platform team measures concurrent sessions, response latency, token throughput and error rates from the inference server's metrics endpoint, and compares them with the capacity planned for that wave. Open WebUI needs PostgreSQL, Redis and shared file storage for multiple replicas, so the team confirms those components are in place before the wave that requires them. The output is a wave sign-off record listing the measurements and the decision to proceed.
Plan a standing update cadence for the interface and models
Interface and inference releases move quickly. Open WebUI published v0.12.3 on October 16, 2026, following v0.12.0 on October 10, and vLLM ships new minor releases about every two weeks, so the team sets a recurring review rather than updating ad hoc. Each proposed update goes through a change process: release notes review, a staging test of SSO, logging and model behavior, a rollback plan and approval from the owner of the register. A quarterly review of models and components, with out-of-cycle patches for security fixes, is a reasonable starting point to confirm with the platform owner. The output is a change calendar and an approved change record for each update.
Review evidence and hand over to operations
Before company-wide availability, the team compiles the scope document, data classification, test records, register, wave sign-offs and change records into one evidence folder mapped to the controls auditors and customers ask about. Operations takes ownership of monitoring, access reviews and the update calendar, with named owners for each. Security reviews a sample of logs to confirm they record what the policy says. The output is a handover record and an evidence index that the compliance lead can present to an auditor without rebuilding it.
Evidence SOC 2 and ISO 27001 auditors and customer questionnaires ask for about the assistant
This table lists the evidence an auditor or customer security reviewer is likely to request for a self-hosted staff assistant, the component that produces each item, and what to confirm before the assessment. DeployOpen prepares and organizes this evidence, but the independent auditor issues the SOC 2 or ISO 27001 opinion or certification. A SOC 2 Type 1 report describes controls at a point in time, while a Type 2 report tests operating effectiveness over a period, so plan evidence collection for the window your auditor will examine.
| Evidence item | Component that produces it | What the reviewer expects | Confirm before the assessment |
|---|---|---|---|
| Access control listings and SCIM provisioning records | Identity provider and Open WebUI or LibreChat group and role configuration, with SCIM 2.0 provisioning logs where enabled | Current user and group lists, role assignments, and joiner and leaver records tied to HR events | Whether Open WebUI SCIM 2.0 provisioning is in use, and how deprovisioning is recorded |
| MFA and API key policies | Identity provider MFA settings, Open WebUI authenticator-app MFA, and API key issuance and revocation records | Written policy, enforcement settings, and a sample of key creation and revocation events | Which login paths bypass the identity provider, and who may create API keys |
| Audit and chat logs with retention | Open WebUI database and log pipeline, plus the gateway or observability layer that records prompts and responses | Defined retention periods, restricted access to logs, and evidence that deletion runs on schedule | Whether chat content is retained by default, and whether logs contain customer data that needs its own controls |
| Change records for model and platform upgrades | Version control and change tickets for Open WebUI, Ollama or vLLM, and model weight updates | Approval, test results, and rollback plan for each upgrade, with dates and release versions | Release cadence of each component and who approves model swaps |
| Vendor and subprocessor inventory | Asset register covering the hosting provider, the model source, and any support or implementation firm with access | Named subprocessors, their locations, and the contract terms that bind them to security obligations | Whether any party can reach prompts, files or logs, and where backups are stored |
| Encryption and network segmentation configuration | TLS termination at the reverse proxy, storage encryption settings, and network policies around the inference endpoints | Configuration exports and diagrams showing that model servers are reachable only from the application tier | Whether vLLM nodes sit on an isolated network and whether metrics and health endpoints are restricted |
| Vulnerability and penetration test results for the assistant | Scanner output for container images and hosts, and a penetration test that covers the chat interface, APIs and prompt injection | Dated reports, tracked findings with owners, and proof that critical items were closed or accepted | Whether the test scope includes the inference endpoints and file upload paths, and who retests after fixes |
How to answer customer security questionnaires about a self-hosted assistant
Where are prompts, files and chat history stored in a self-hosted assistant?
In your own infrastructure, in the database, object storage and vector store you choose and configure. A typical Open WebUI deployment keeps chat records in a relational database, uploads in shared file storage and retrieval data in a vector database you operate. Answer with the actual storage map for your environment: which database holds chats, where uploads land, where embeddings are kept, which backups copy them and how long each copy is retained. Confirm the encryption at rest and in transit settings for each store, and name the team that owns them. Do not claim data never leaves a location until your own network controls and logs show it.
Does a third-party model provider receive our prompts?
Not when the models run on your own GPUs or servers and the interface points only at your internal endpoints. The chat interface, the inference engine such as vLLM or Ollama, and the model weights all run in your environment, so no prompt is sent to an outside model service. The answer changes if you also enable external providers in the interface, which Open WebUI supports alongside local models. Document which providers are configured, disable any that are not approved, and show the egress rules or proxy allowlist that enforce this. Keep a record of the configuration as evidence.
Who are the subprocessors for a self-hosted chat assistant?
If you run everything in your own cloud account or data center, the core software is not a subprocessor because it does not process customer data for you. List the parties that can touch the data: your cloud or colocation provider if hosted there, any managed database or backup service, the implementation partner if it has administrative access, and any external model or add-on service that is enabled. Each entry needs a contract that binds it to documented instructions and security measures. Confirm each name, location and role with the vendors involved before the questionnaire is sent, because the list must match what is actually running.
How is access revoked when an employee leaves or changes role?
Access is revoked centrally at the identity provider, and the chat application follows that change. Open WebUI documents SSO through OIDC and LDAP and SCIM 2.0 user and group provisioning, so deprovisioning in the directory can remove the account in the application. Answer with the exact flow: who disables the directory account, how quickly SCIM or the sign-in check takes effect, what happens to active sessions and API keys, and how group-based permissions to knowledge bases are removed. Ask your implementation partner to test this with a sample leaver and keep the test result as evidence.
Is customer data used to train the models?
Not by the assistant itself, because the models are open-weight and are not retrained on chat content during normal use. Fine-tuning happens only if your team starts a separate project, so state that clearly and give the governance step that would be required for it. Check the licence and terms of every model you serve, since they may restrict use but do not change how your prompts are handled. Also confirm that telemetry, usage analytics and any feedback features are disabled or pointed at your own storage, so no content leaves the environment for product improvement.
Which evidence supports our SOC 2 or ISO 27001 answers for the assistant?
Auditors look for controls and records, not a certificate for the software itself. SOC 2 examinations use the 2017 Trust Services Criteria, and the security common criteria CC1 to CC9 apply to every SOC 2. ISO/IEC 27001:2022 Annex A has 93 controls, and only an accredited certification body can certify your organization. Provide the system description, access review records, change logs, backup and restore tests, logging configuration and vendor records. DeployOpen prepares systems, controls and evidence for these assessments but does not issue reports or certifications, and your independent auditor decides what is sufficient.
How long are chat logs kept, and can we delete them on request?
Retention is a setting you choose and document, and it must be enforced in the database and backups, not only in the interface. Define retention for chat records, uploaded files, vector embeddings, application logs and backup copies, then show how a deletion request removes all of them. Confirm the backup rotation period and how deleted data is purged from older backups. If the assistant is used for regulated data, check whether record-keeping duties apply. Ask your partner to demonstrate one full deletion from start to finish and file the output with your evidence.
What should a questionnaire answer say about the vLLM endpoint exposure?
State that the inference endpoint is reachable only from the chat service and approved internal clients, and show how that is enforced. The vLLM security documentation notes that the --api-key option does not protect every endpoint, including /metrics and /health, so do not rely on it alone. Recommend and describe a reverse proxy that allowlists only the endpoints meant for users, plus network isolation for multi-node communication, which is insecure by default. Include the scan or test results that confirm the allowlist works, and be ready to explain which endpoints are blocked and why.
What to ask an implementation partner before signing a fixed-scope engagement
Start by asking the partner to map each SOC 2 common criterion and each ISO/IEC 27001:2022 Annex A control you are in scope for to a concrete setting in the assistant. An auditor will ask which control covers access to the chat, who can see other users' histories, and where transcripts are kept, so the answer should name the systems: the Open WebUI or LibreChat configuration, the reverse proxy, the PostgreSQL database, the log sink and the backup target. Ask for the mapping as a working document you can hand to your auditor, not a summary slide. Also ask how they will show that each control operates over the audit period, because a SOC 2 Type 2 report tests operating effectiveness over time, not design alone.
Ask how SCIM 2.0 provisioning and OIDC or LDAP login will be configured for your identity provider, and how groups map to Open WebUI roles and per-resource permissions. Ask who approves a new group, how a leaver's access is removed, and how the partner tests deprovisioning end to end. For inference endpoints, ask what sits in front of vLLM beyond the --api-key option. The vLLM security documentation says that option does not protect /metrics, /health or /invocations, and recommends a reverse proxy that allowlists only user-facing endpoints. Ask for the allowlist, the network placement of model nodes, and the test that proves a request to an unlisted path fails. Also ask how chat history, uploaded files and trace data are retained and deleted, and how they test that deletion actually happens.
Before you sign, confirm three things with the partner. First, how they will handle the Open WebUI License for your user count: the license forbids altering or removing branding above 50 end users in any rolling 30-day period unless you hold an enterprise license or written permission, so ask for a written position on your expected seat count and on any custom branding. Second, how they rehearse upgrades, since Open WebUI and vLLM ship releases often, and ask for the staging environment, the test suite and the rollback plan. Third, what evidence package you receive at close, such as configuration exports, access reviews, backup restore records and the subprocessor register. Be ready to supply your data classification decisions, read-only IdP access, an approved model list, and a named owner for the subprocessor register, because the partner cannot finish those items without you.
Common questions from software companies about private LLM chat
What drives the cost of a private LLM chat for a software company?
Cost is driven by four components: GPU or accelerator capacity for the models, the chat interface and any enterprise license, integration and security work, and the people who operate the service. GPU capacity usually dominates and scales with concurrent users, model size and context length, so your expected peak concurrency is the first number to fix. Open WebUI and LibreChat are open source, but Open WebUI's branding clause can require an enterprise license above 50 users in any rolling 30-day period. Request a quote that separates one-time costs (design, SSO and logging integration, hardening, initial model rollout) from recurring costs (hardware or cloud GPU hours, support, upgrades, model refresh). Ask for each line to name who pays for it: your team, your cloud provider or DeployOpen.
How long does a private LLM chat pilot take to deliver?
The duration depends on the scope agreed in writing before work starts, not on a fixed template. The main variables are whether GPU capacity already exists or must be procured, how many identity groups and data sources are connected, whether logging must reach your existing SIEM, and how many models you want to evaluate. A pilot limited to one model, SSO login and basic chat history is far smaller than one that adds file uploads, a knowledge base and audit exports. Ask the partner for a written timeline with milestones, the assumptions behind it, and the decisions you must make before the clock starts, such as which models are approved and who signs off on the acceptable use policy.
Can our internal platform team run a private LLM chat without a partner?
Many engineering teams can deploy Open WebUI or LibreChat with Docker Compose for a small group, but production use needs more than a container. Multi-replica deployments need PostgreSQL instead of SQLite, Redis for shared state and shared file storage, according to the Open WebUI scaling guide. Inference endpoints also need protection: vLLM's --api-key option does not cover every endpoint, so a reverse proxy that allowlists only user-facing routes is required. Assess your team against four needs: GPU operations, identity integration, security testing and evidence collection for audits. If one of these is thin, a fixed-scope partner engagement can cover that gap while your team keeps operational ownership.
Does self-hosting an LLM remove the need for a subprocessor list?
No. Self-hosting removes the model provider as a subprocessor for prompts and files, but it does not remove every vendor. Your chat platform, GPU or cloud host, object storage, identity provider, log tooling and any support partner may still access customer or source data. Maintain a subprocessor list that names each of these, the data each one can reach, and where it is hosted. Customer security questionnaires typically ask for this list and for the contractual terms that bind each party, so confirm the data processing terms with every vendor you keep. Where you process personal data, GDPR Article 28(3) requires processor contracts to bind the processor to documented instructions and security measures.
Does a private LLM chat change our SOC 2 or ISO 27001 scope?
It can, because the assistant becomes a system that handles customer data and must be described and controlled. For SOC 2, the security common criteria CC1 to CC9 apply to every report, and the chat platform, its access controls, logging and change management will appear in the system description and be tested by the auditor. For ISO/IEC 27001:2022, the assistant falls within your information security management system, and the relevant Annex A controls cover access, logging, supplier relationships and development. Ask your auditor early whether the assistant is in scope for the next examination or certification cycle, and what evidence they will sample, such as access reviews, configuration baselines and model change records. DeployOpen prepares systems and evidence for these assessments but does not issue audit reports or certifications.
How do we handle the Open WebUI branding clause?
Open WebUI code is distributed under the Open WebUI License, which prohibits altering or removing Open WebUI branding in deployments with more than 50 users in any rolling 30-day period, unless you hold an enterprise license or written contributor permission. Code through v0.6.5 remains BSD-3-Clause, and the clause applies from v0.6.6 onward. Engineering teams should decide early whether the company will keep the default branding, obtain an enterprise license that allows white-labeling, or choose a different interface such as LibreChat, which is MIT licensed. Ask the vendor for the current license text and enterprise terms in writing, and have your legal team review them before rollout. Enterprise pricing is not published, so request a quote based on your expected seat count.
What should engineers check before connecting the assistant to source code?
Engineers should confirm how prompts, uploads and chat history are stored, who can read them, and how long they persist. Check that the model endpoint is reachable only from the chat application, not from user networks, and that multi-node vLLM traffic runs on an isolated network because its inter-node communication is insecure by default. Verify that logs capture who asked what and when, while avoiding secrets being written to logs in plain text. Test prompt injection paths for any tool or retrieval feature, because a document or code file can carry instructions. Document these checks as part of your acceptance criteria so they can be reused in security reviews and audits.
Which logs does the EU AI Act require from a staff chat assistant?
Whether the EU AI Act's logging duties apply depends on your role and the system's risk classification, so confirm this with counsel before assuming either way. If the assistant were a high-risk AI system, Article 12(1) requires that it technically allows automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Even where those duties do not apply, you should retain interaction logs that support incident investigation and customer questionnaire answers. Define the retention period, the access policy and the deletion process in writing, and ask any implementation partner how logs are stored, protected and exported.
How DeployOpen scopes, builds and supports a private assistant for a software company
DeployOpen is a Chainscore Labs company that plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or networks. For a private assistant, that means the Open WebUI or LibreChat front end, the model servers, the PostgreSQL and Redis tier, and the identity and logging connections all run where your engineers' prompts already stay. Most engagements are fixed-scope: the written agreement names the components, the user groups, the models in scope, the integrations and the acceptance tests before any build work starts. Before signing, ask the partner to list which Open WebUI license terms apply at your expected user count, since the branding clause changes above 50 users in any rolling 30-day period, and to confirm whether the quote includes an enterprise license or only the self-hosted deployment.
The evidence package is where software companies usually feel the gap between building and proving. DeployOpen prepares the system description, the configuration baselines for SSO and group-based roles, the settings for ENABLE_SIGNUP and model access, retention and export procedures for chat history, the inventory of model weights and container images, and the logs that show who used the assistant and when. It does not issue SOC 2 reports, ISO 27001 certificates or any other attestation. Your independent auditor or certification body decides what is sufficient, so agree the evidence format with them during scoping, and confirm which Trust Services Criteria or Annex A controls the assistant will be cited against in your own control matrix.
Security testing covers the assistant as a system and as an AI application. That includes the web interface and its API keys, the vLLM endpoints (a reverse proxy should expose only the routes users need), the retrieval path for uploaded files, and prompt injection attempts against tools or connected knowledge. Ask whether the test plan includes AI-specific cases such as data exfiltration through uploaded documents and cross-user leakage in shared knowledge bases. After go-live, retainers can cover upgrades to the fast-moving Open WebUI and vLLM releases, patch review and recurring testing, which keeps the evidence current for each audit period. Confirm the retainer scope, response expectations and the change process in writing, because none of these terms are published as standard.
