Private LLM chat requirements checklist for enterprises

How to use a private LLM chat requirements checklist before you choose a platform

A CISO or platform lead deciding what a staff chat assistant must do should settle the requirements before comparing Open WebUI, LibreChat or a SaaS tool. This checklist is written for a self-hosted build, where the organization runs the interface, the models, the storage and the logs inside its own cloud accounts, data centers or air-gapped network. Each section below is meant to be used in three ways: as a list of requirements the business has to state in writing, as a set of verification steps that a pilot or proof of concept must pass, and as a set of separating questions that quickly eliminate options that cannot meet the stated needs.

Three answers should be settled first because they shape the architecture early. The license terms decide whether the interface can be rebranded and whether a given user count triggers a commercial agreement, so the expected number of end users and the branding expectations belong in the first meeting. The concurrency and data-handling answers decide the serving layer, the database and the storage design: how many people will chat at the same time, which data classes may be pasted or uploaded, and how long chats are retained. A team that answers these late will often discover that a single-node setup it has already piloted cannot support multiple replicas or the logging the organization requires.

Use the sections that follow to record an owner and a decision for each requirement. Where a figure is needed, such as expected peak concurrency, GPU memory per model or retention periods, ask the business owner and the security team to supply it and confirm it against the vendor documentation for the versions being evaluated, because release cadence in this category is fast and defaults change between versions.

Functional requirements every private LLM chat must meet

This table lists the functional requirements a private chat assistant has to satisfy, with why each one matters and how to verify it before rollout. Read each row as a question to put to a vendor, a partner or your own test team. The main takeaway is that concurrency, endpoint choice and license terms behave differently between Ollama and vLLM, so the same requirement needs different proof depending on the stack.

Requirement	Why it matters	How to verify
Model licenses and weights	Open-weight models carry their own license terms on use, redistribution and commercial scale, separate from the chat interface license.	Read each model's license, record the version and weight source, and have legal confirm permitted internal use before adding it to the catalog.
Interface license and branding	Open WebUI's License forbids altering or removing its branding above 50 users in any rolling 30-day period unless an enterprise license or written permission applies. LibreChat is MIT licensed.	Count peak users per rolling 30 days, read the Open WebUI License clause 4, and get a written decision on branding before the pilot widens.
Users and concurrency	Peak simultaneous requests drive latency and queueing. Ollama defaults to one parallel request per model, while vLLM batches requests and is tuned through its own engine settings.	Run a load test with your expected peak prompt and response lengths, and record time to first token and queue depth for each serving engine.
Serving engine (Ollama versus vLLM)	Ollama suits smaller teams and single-GPU hosts with settings such as OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE. vLLM exposes OpenAI-compatible endpoints and tensor and pipeline parallelism for larger shared services.	Benchmark both candidates on your hardware and model, and confirm which settings your team will tune and monitor in production.
GPU capacity	Model size, context length and concurrency determine memory and GPU count. Undersized hardware causes queueing, slow responses or failed loads.	Ask the partner for a sizing worksheet tied to your concurrency test results, and confirm headroom for model updates and fallback capacity.
Model endpoint security	Ollama's local API has no authentication by default and binds to 127.0.0.1 port 11434. vLLM's --api-key does not cover every route, so unprotected admin or metrics endpoints can remain reachable.	Scan the network for open ports, confirm the bind address, and place a reverse proxy in front that allowlists only the endpoints users need.
Supported providers	Open WebUI connects to Ollama, OpenAI, Anthropic or any OpenAI-compatible provider. Any external provider adds a processor and a data transfer question.	Confirm which providers are enabled, disable the ones not approved, and test that blocked providers cannot be selected by users or API keys.
File upload and knowledge features	Uploads and knowledge collections hold document content. Open WebUI's retrieval uses vector databases and needs shared storage when running multiple replicas.	Upload test files of each approved type, confirm where they are stored and indexed, and verify deletion removes both the file and its vectors.

Choosing the chat interface: Open WebUI, LibreChat or AnythingLLM

Open WebUI is the most widely deployed option in this group and is the interface most often paired with Ollama or vLLM. Since v0.6.6 (April 19, 2025), its code is distributed under the Open WebUI License, a BSD-style license with a branding clause. Clause 4 prohibits altering or removing Open WebUI branding except in deployments with no more than 50 end users in any rolling 30-day period, with written permission, or under other listed exceptions. Code through v0.6.5 remains BSD-3-Clause, which matters if you are evaluating older releases.

User count threshold: The branding restriction applies once you exceed 50 users in any rolling 30-day period, so a pilot can stay under it while a company-wide rollout cannot.
Enterprise license: Open WebUI offers enterprise licenses that allow white-labeling and custom branding. Pricing is not published and is quoted after contacting enterprise sales with an expected seat count.
Identity and access: The project documents OIDC and LDAP single sign-on, SCIM 2.0 provisioning, groups with role-based access, and authenticator-app MFA.
Scale requirements: Multi-instance deployments need PostgreSQL, Redis and shared file storage, so plan for more components than a single-node install.
Retrieval features: Built-in retrieval supports 13 vector databases and hybrid search with reranking.

Confirm current features against the project documentation, and read the license against your expected user count and any plan to change the interface branding before you commit.

Security requirements for the chat platform and its endpoints

The chat interface, the model servers and the data that flows between them each need their own controls. Identity decides who can reach the assistant, endpoint protection decides who can reach the models behind it, and data and prompt rules decide what users may send and what the assistant may do with it. Check each of these before the pilot, not after the first incident.

Single sign-on and provisioning

Require that staff sign in through the organization's identity provider over OIDC or LDAP, so disabling an account removes chat access at the same time. Open WebUI documents SCIM 2.0 user and group provisioning, which keeps group membership in step with the directory. Confirm with your identity team which group claims map to chat roles and who approves new groups.

Role-based groups and permissions

Map directory groups to chat roles, model access and knowledge collections, so finance staff do not see models or documents reserved for another team. Open WebUI supports role-based access control with groups and per-resource permissions. Ask the vendor or partner to demonstrate that a removed group member loses access to shared chats and uploaded files.

Multi-factor authentication

Decide whether local accounts are allowed at all, and if they are, require an authenticator-app second factor for administrators and privileged users. Open WebUI documents authenticator-app MFA and API keys, and API keys need their own owner, scope and rotation policy. Treat every API key as a credential that can bypass the interface.

Model endpoint exposure

Ollama binds to 127.0.0.1 on port 11434 by default and its local API does not require authentication. Changing the bind address with OLLAMA_HOST exposes that unauthenticated API to the network, so put a reverse proxy with authentication in front of it and block direct access. Confirm that no team has pointed a script at the port from another host.

vLLM endpoint allowlisting

The vLLM --api-key option protects only the /v1, /v2, /inference and /cohere paths, leaving endpoints such as /metrics, /health and /pause open. Its security documentation warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints users need. Verify with a scan from the user network that administrative and metrics paths are unreachable.

Data pasting and upload rules

Write a classification table that states which data classes may be pasted into the chat, which may be uploaded as files and which must never leave the source system. Enforce the rules in the interface through group-level settings and in the gateway through content checks where your data classes are machine-recognizable. Ask the partner which controls are technical and which depend on user behavior.

Prompt injection controls at the gateway

Documents and web content that the assistant reads can carry instructions that override the system prompt or request data from other users. Place input and output checks at the AI gateway, such as LiteLLM or Apache APISIX, so that tool calls and retrieved content are filtered before they reach the model or the user. Test these controls with known injection patterns before any knowledge base is connected.

Which data may be pasted, uploaded and retained in chat history

Classify data before the assistant reaches staff, because the classification decides what may be pasted, what may be uploaded and how long chats are kept. Start with a short, written matrix of data classes (public, internal, confidential, regulated personal data and any sector-specific category) and map each class to a permitted action: paste, upload into a knowledge collection, or never enter the assistant. Owners of the data (legal, HR, clinical or finance leads) should sign off on their rows, not only the platform team. Put the rules into the acceptable use policy so that the technical controls and the written rule say the same thing.

Ollama states that when it runs locally it does not see your prompts or data, and cloud features can be turned off with OLLAMA_NO_CLOUD=1 or the server configuration file. That protects the model runtime, but it does not protect everything around it. The chat database in Open WebUI or LibreChat stores conversations, user records and uploaded files, and multi-replica Open WebUI deployments move that state into shared storage. Ask the implementation partner to document where each of these lives, who can read it, how backups are encrypted and how deletion requests are carried out across the database, file storage and any vector store. Confirm the retention period for chats, deleted chats and uploads with the data owners, and confirm whether users can delete their own history.

Logging is a separate decision from retention. Chat and access logs help with incident response, but they also hold personal data. If any use case may be high-risk under the EU AI Act, Article 12 on record-keeping requires such systems to allow automatic event logging over their lifetime, and Article 19 requires providers to keep those logs for at least six months unless other law provides otherwise. Whether either article applies to your deployment, and who the provider is in your case, must be confirmed with counsel. Decide the log content, the retention period and the access roles before rollout, and do not let log retention quietly default to longer or shorter than the policy.

Integration requirements for identity, data stores and gateways

A staff chat assistant is only as controlled as the systems around it. Identity, the state stores behind multiple replicas, retrieval storage, the gateway that brokers model calls and the observability layer each hold part of the prompt and access story. Confirm each dependency early, because several of them carry their own licenses and scaling rules.

Identity provider and group mapping

Open WebUI documents SSO through OIDC and LDAP, plus SCIM 2.0 provisioning for users and groups. Decide which directory is the source of truth, which groups map to model access and which map to admin rights. Confirm whether your identity provider's group claims are passed through in the form your chosen version expects, and test removal of a leaver's access end to end, including active sessions and API keys.

PostgreSQL for multiple replicas

The Open WebUI scaling guide requires PostgreSQL instead of SQLite once more than one instance runs, because SQLite does not handle concurrent writes from multiple processes well. Plan the database backup, restore testing and retention of chat records as part of the requirements, not as an afterthought. Ask who operates the database and where its backups are stored.

Redis and shared file storage

For multiple replicas, Open WebUI needs Redis as a shared store for sessions, websocket connections and application state. Uploaded files and knowledge collections must also sit on shared storage so every replica can reach them. Confirm the storage class, encryption at rest, and whether file deletion propagates to backups, since uploaded documents are often the most sensitive data in the system.

External vector database

Open WebUI's retrieval features support 13 vector databases, including ChromaDB, PGVector, Qdrant, Milvus and Elasticsearch, with hybrid search and reranking. Choose one that your team can back up and monitor, and keep it inside the same trust boundary as the documents it indexes. Confirm which vector store your access model requires for per-group document permissions, because not every option handles them the same way.

AI gateway between chat and models

An AI gateway such as LiteLLM or Apache APISIX can hold model keys, enforce budgets and rate limits, and write request logs in one place. LiteLLM is MIT licensed except its enterprise directory, and APISIX is Apache-2.0. Confirm which gateway features you need, such as guardrails or per-user budgets, and whether those features sit in the licensed part of the code before you depend on them.

Observability with Langfuse or Phoenix

Langfuse is MIT licensed except its ee directories, and Phoenix is under the Elastic License 2.0, which prohibits offering the software to third parties as a hosted or managed service. Both can keep traces of prompts, outputs and costs in your own storage, but the license terms differ by component. Have counsel confirm the license terms for your planned use, especially if a partner operates the tool for you.

Operational requirements and how to verify them

This table lists the operating requirements a private chat assistant must meet after launch, with a concrete verification step for each. Read it as a test plan: if a requirement has no named check and owner, it is not yet covered. The main takeaway is that backups, upgrades and monitoring need to be proven in a staging environment before staff depend on the assistant.

Requirement	What it asks for	How to verify it
Backups and restore	Chat history, user and group data, uploads and knowledge stores backed up on a schedule, with PostgreSQL and shared file storage included when Open WebUI runs on multiple replicas.	Restore the database and file store into an isolated environment, log in as a test user and confirm a past conversation and an uploaded file are readable.
Model weight backups	Model files stored under the path set by OLLAMA_MODELS or the vLLM model cache, kept so a model can be re-pulled or restored without internet access.	Delete a model in a test host, restore it from the backup copy and confirm the model answers the same test prompts.
Upgrade cadence for fast-moving releases	A written policy for how often Open WebUI, Ollama and vLLM are updated, given that Open WebUI v0.12.3 (October 16, 2026), Ollama v0.40.5 (October 14, 2026) and vLLM v0.31.0 (October 5, 2026) are recent releases and vLLM ships minor releases about every two weeks.	Upgrade a staging copy first, run a fixed set of regression prompts and SSO login tests, then promote to production only after a recorded sign-off.
Monitoring with vLLM metrics	Scrape the Prometheus-format data at /metrics for request counts, token throughput, latency histograms and KV cache use, and alert on sustained queueing or latency.	Generate a controlled load in staging, confirm the metrics move as expected in the dashboard and confirm the alert fires at the agreed threshold.
GPU health	Monitoring of GPU temperature, memory errors, utilization and driver state, with a defined response when a card degrades or a host drops out.	Simulate a host failure in staging and confirm the failover path, the alert and the time to restore service are recorded.
Incident response	A runbook covering model outages, data exposure through a chat or upload, and a compromised API key, with named roles and a contact path for the vendor or partner.	Run a tabletop exercise with a sample exposure scenario and confirm the team can revoke keys, disable a user group and preserve logs within the runbook steps.
Acceptable use policy enforcement	A written acceptable use policy that defines which data classes may be pasted or uploaded, mapped to technical controls such as group restrictions, disabled features and user notices.	Test with seeded sample data in a staging tenant to confirm restricted groups cannot use the restricted features, and review the policy against the controls each quarter.
Endpoint exposure	Only user-facing endpoints reachable from staff networks, with inference endpoints behind a reverse proxy that allowlists the paths meant for users.	Scan the inference host from a user network segment and confirm that endpoints not meant for users, such as /metrics and /health, are unreachable from there.

Steps to accept the chat platform before the first user group

The sequence turns a configured chat platform into a signed-off service by testing licenses, capacity, identity, data handling, logging and policy in a fixed order. Each step produces a written result that the next step relies on.

Confirm licenses and branding obligations
Legal or procurement staff, with the platform engineer, check the license of every component: the chat interface, the inference engine, the gateway and any retrieval or workflow add-ons. For Open WebUI, confirm whether the expected user count keeps the deployment under the 50-user threshold in any rolling 30-day period, or whether branding must stay untouched or be changed under an enterprise license. Done means a written license register lists each component, its license, the branding position and the owner who approved it.
Run a concurrency test at expected peak
The platform engineer loads the model endpoint with a scripted workload that matches the expected peak concurrency, including long prompts and file-based requests. Record queue depth, time to first token, throughput and GPU memory use, and compare them with the Ollama or vLLM settings in place, such as parallel request limits and queue size. Done means the results show whether the agreed peak holds with headroom, and the GPU sizing decision is recorded or revised before any user is onboarded.
Verify SSO group mapping and role access
The identity engineer signs in with test accounts from each intended group, using the OIDC or LDAP connection and the SCIM provisioning in place. Confirm that users land in the correct groups, that role-based permissions limit model access and knowledge collections as designed, and that a removed or disabled account loses access. Done means a test matrix of groups, roles and expected outcomes has been executed, and every mismatch has a fix and a retest result.
Test data-handling rules with a red team
The security team defines the data classes that staff may paste or upload, then a red team tries to break those rules through the interface, uploaded files, retrieved documents and API keys. Include prompt injection attempts that try to override instructions or pull content from uploaded files, and confirm that model endpoints are reachable only through the intended path. Done means every finding is logged with severity, owner and retest status, and the rules that remain open are accepted in writing by the risk owner.
Confirm chat retention and log settings
The compliance lead and platform engineer confirm where chat history, uploaded files and inference logs are stored, how long each is kept and who can read them. Check that the retention settings in the chat interface, database and metrics stack match the policy, and that inference logs needed for review are captured without storing content that the policy forbids. Where the system may fall under the EU AI Act, confirm the record-keeping duties with counsel. Done means a retention table is signed off and a test deletion and an audit log sample have been reviewed.
Sign off the acceptable use policy
The policy owner finalizes the acceptable use policy with the approved data classes, prohibited uses, reporting channel and consequences, then publishes it at first login. Each pilot user confirms they have read it, and the owner records the approval with the names of the approvers and the version number. Done means the policy is linked from the sign-in screen, acknowledgement is tracked, and the first user group is cleared to start.

Questions buyers ask about private LLM chat requirements and effort

What drives the cost of a private LLM chat assistant?

Cost comes from four places: GPU or accelerator capacity to serve models, the chat interface and any enterprise license, integration and security work, and the people who operate the service. Hardware and cloud GPU spend recur every month, while design, SSO setup, hardening and testing are mostly one-time. Capacity is the largest variable, and it rises with the number of concurrent users, the model size and the response length. For comparison, Ollama's pricing page lists its Team plan at $500 per month for cloud models, and running models on your own hardware is listed as always unlimited. vLLM has no paid edition, so its cost sits in hardware, infrastructure and operations. Ask any quote to split one-time and recurring items and to state the assumed concurrency.

Can our internal team build a private LLM chat service alone?

Many teams can run a pilot with in-house staff, especially if they already operate containers, identity systems and GPU servers. The harder parts are production readiness: PostgreSQL and Redis for multiple replicas, shared file storage, SSO group mapping, reverse proxy allowlisting for inference endpoints, model update cadence and monitoring. vLLM's security documentation says its --api-key option does not protect every endpoint, so a proxy that exposes only user-facing routes is needed. Ollama's local API requires no authentication by default, so binding and access control must be designed deliberately. Assess whether your team can staff an on-call rotation and track fast-moving releases. A partner can take over a defined scope under a written agreement, and you can keep operations in-house afterward.

What does our team need to provide for the work?

Plan to provide a named business owner, an identity administrator for SSO and SCIM provisioning, a security reviewer for the data classification decision, and an infrastructure lead who controls the GPU hosts, networks and storage. You will also need to decide the user population, expected concurrency, which document types may be uploaded and how long chat history is kept. Confirm access to your model sources, whether models are pulled from the internet or moved across a controlled transfer path, and who approves changes. Ask any implementation partner to list these inputs in the written scope so that dependencies and sign-off points are visible before work starts.

How long does a private LLM chat rollout take?

No supplied duration applies to every organization, so treat any single number with caution. The schedule depends on the number of users, whether you need SSO and SCIM, GPU procurement lead times, the retention and logging policy, the security review cycle and whether you roll out in waves. A pilot for a small group is usually faster than a multi-replica deployment for thousands of staff, which needs PostgreSQL, Redis and shared storage. Ask a partner for a phased plan with explicit gates, such as a security sign-off before user access and a measured concurrency test before each wave. Fixed-scope engagements have their timelines agreed in writing before work starts.

Do the Open WebUI branding license limits apply to us?

They may apply, and you should confirm the position with counsel before rollout. Since v0.6.6, Open WebUI code is distributed under the Open WebUI License, which forbids altering or removing branding in deployments that exceed 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written contributor permission. Code through v0.6.5 remains BSD-3-Clause. Count active users across the rolling window, not only named accounts, and check whether white-labeling is needed. Enterprise licenses are quoted after contacting sales with an expected seat count, and pricing is not published. If you prefer LibreChat, its MIT license does not carry this clause, but you should still confirm the terms of the version you deploy.

Which questions should we ask a partner about this platform?

Ask how they size GPUs for your stated concurrency, which models they recommend and under which licenses, and how Open WebUI or LibreChat is connected to your identity provider through OIDC, LDAP or SCIM. Ask how inference endpoints are protected, including whether the vLLM endpoints are fronted by a reverse proxy with an allowlist and whether Ollama is bound only to the internal network. Ask how chat history is retained, logged and deleted, and how logs are kept for any obligations you carry. Finally, ask how model and platform updates are tested and rolled out, what is in the fixed scope, and what a retainer covers after go-live.

How should we handle chat logs and retention for compliance?

Start by defining which events you need to record, who may read them and how long they are kept, because these decisions shape storage and access design. The EU AI Act sets log duties for high-risk AI systems: Article 12(1) requires automatic event recording over the system's lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Whether a staff chat assistant is high-risk depends on its use, so confirm that with your legal team. Retention settings in the chat interface, backup policy and deletion requests should all be written down. DeployOpen prepares systems and evidence for compliance assessments, but auditors and certification bodies issue the actual conclusions.

Questions that separate private LLM chat platforms and partners

Start with concurrency, because it decides the hardware and the serving engine before anyone discusses features. Ask the partner to show how many simultaneous generations they sized for, at what context length, and on which GPUs. Ask how the design changes between Ollama and vLLM: Ollama's defaults allow one parallel request per model unless OLLAMA_NUM_PARALLEL is raised, while vLLM is built for batched serving on GPUs. Ask whether the chat interface is pointed at a single inference node or at a shared endpoint, and what happens to the PostgreSQL and Redis tier when the user count grows. A partner who cannot answer with numbers from your own peak usage is not yet sizing the system.

Next, ask how every inference endpoint is protected. The chat interface should be the only caller. Ask whether the partner will place a reverse proxy in front of vLLM that allowlists only the paths users need, since the vLLM --api-key option does not cover endpoints such as /metrics or /health. Ask how Ollama is kept off the network by default, how its bind address is set, and whether cloud features are disabled with OLLAMA_NO_CLOUD=1. Then ask how SSO groups map to model access: which Open WebUI groups receive which models, who may create knowledge bases, and how SCIM deprovisioning removes access when someone leaves. Request a test plan that shows a user outside a group cannot select the restricted model.

Finally, ask how chats are retained, logged and updated. Get the retention setting, the place where chat history and uploads are stored, and who can read them. Ask whether prompt and response logs are kept for the period your policy and any applicable law requires, and whether the partner has a plan for the record-keeping the EU AI Act sets out for high-risk systems, if that applies to you. For model updates, ask who tests a new model release, how a rollback works, and how the Open WebUI, Ollama or vLLM upgrade cadence is handled. The separate Private LLM chat RFP questions page carries the full question set.

Private LLM chat requirements checklist