Private LLM chat RFP questions

Concurrent users do not map one to one onto GPU load. Each active chat turn produces a request, and the load depends on how many of those requests run at the same moment, how long each prompt is, how long each answer runs, and whether users upload files that get embedded for retrieval. A good partner converts your headcount into peak simultaneous requests using your usage pattern, then tests that number against the chosen model. Ask for the assumptions in writing: peak hour share, average prompt and response length, context window, and the share of users who use retrieval or long document uploads. Red flag: a partner who quotes GPU counts before asking how your staff actually work.

Ollama and vLLM solve concurrency differently. Ollama controls it with OLLAMA_NUM_PARALLEL (default 1 parallel request per model), OLLAMA_MAX_LOADED_MODELS (default 3 times the number of GPUs, or 3 on CPU) and OLLAMA_MAX_QUEUE (default 512 queued requests), which suits smaller teams and mixed model use. vLLM scales a model across GPUs with tensor parallelism within a node and pipeline parallelism across nodes, and is built for high-throughput serving of one model to many users. Ask the partner which engine they would run for your peak load, why, and what throughput and queue depth they measured. Red flag: a partner who treats the two as interchangeable or cannot explain the defaults they changed.

A partner should review each model's license before recommending it, because open-weight licenses differ on use restrictions, attribution, acceptable use terms and commercial scale limits. Ask for a list of every model proposed, its license text, the obligations it creates for your organization, and whether it permits your intended internal use at your user count. Also confirm the license of the serving and interface software: Ollama is MIT licensed, vLLM is Apache-2.0, and Open WebUI now ships under the Open WebUI License, whose branding clause applies above 50 users in any rolling 30-day period. Red flag: a partner who names a model without mentioning its license or who assumes all open weights are free for any use.

Test on a set of prompts drawn from your real work, not on public benchmarks alone. A sound method builds a representative set of tasks with expected outcomes, runs each candidate model with the same system prompt and settings, scores answers with reviewers from the business, and records latency and tokens per second at your target concurrency. Ask the partner to describe the test set, who approves it, how sensitive data is handled during testing, and how results will be stored as evidence for the decision. Red flag: a recommendation based only on leaderboard rankings or a demo the partner prepared.

A partner needs enough information to model peak load and the data path, which typically includes:

• Named user count and the count of users active at peak
• Expected prompt and response length, context window and file upload volume
• Required response time and tolerance for queueing
• Which models staff must use and any model you want excluded
• Where the hardware will run (your data center, cloud account or air-gapped network) and any GPU types you already own
• Retention, logging and availability requirements that change the storage and redundancy needs

Ask the partner to list these inputs in the written scope before quoting hardware. Red flag: a sizing proposal that arrives before these inputs have been collected.

Yes, an engineering team can size GPUs using published serving documentation and its own load tests, and some organizations do. A partner adds value by building the load test, running candidate models on your hardware, documenting the configuration and handing over runbooks your team can repeat when usage grows. Costs come from the hardware or cloud GPUs themselves, the engineering time to measure and tune, and the recurring work of re-testing when models and engine versions change. Ask the partner for the deliverables, which should name the tested configuration, the measured concurrency limits and the upgrade path. Red flag: a sizing answer that cannot be reproduced by your own team.

Plan capacity with a margin and a retest step, because serving engines release often. vLLM shipped new minor releases about every two weeks in the period it was reviewed (for example, v0.30.0 on September 22, 2026 and v0.31.0 on October 5, 2026), and Ollama published v0.40.5 on October 14, 2026. A partner should state how a new engine or model version is load tested against your target concurrency before it reaches users, and how a rollback works. Ask whether the sizing is rechecked after each model change and who owns that retest in the retainer. Red flag: a sizing plan that is fixed for the life of the hardware with no retest step.

No. Ollama binds to 127.0.0.1 on port 11434 by default, so only processes on the same host can reach it. Moving it to a shared address requires setting the OLLAMA_HOST environment variable, which is a deliberate change that your partner should document and justify. The local API at localhost:11434 does not require authentication, so anything that can reach the port can use the models. A good answer places Ollama on a private subnet, puts a reverse proxy such as Nginx in front of it that handles authentication and TLS, and blocks direct access from user networks. A red flag is an answer that sets OLLAMA_HOST=0.0.0.0 and treats the network perimeter as the only control.

The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere. It does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and vLLM's security documentation warns against relying on it alone. A strong answer treats the key as one layer: the proxy enforces user authentication, the key is stored in a secrets manager and rotated, and the team tests each unprotected route from outside the network. A red flag is a partner who says the API key is the security control and cannot name the routes it leaves open. Ask them to show the test results for each route.

The proxy should use an allowlist that passes only the endpoints meant for users, which is the approach the vLLM security documentation recommends. For chat use, that typically means the OpenAI-compatible paths under /v1, such as /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, and only if your applications use them. Everything else returns a denied response by default. Ask the partner to provide the allowlist as a versioned configuration file, with a test that confirms a blocked path returns an error. A red flag is a proxy that forwards all paths and relies on the application to behave.

No. The /metrics endpoint exposes Prometheus data on request counts, token throughput, latency and KV cache use, which reveals how the service is used and how loaded it is. The /health endpoint can reveal service state to anyone who can reach it. Both should be reachable only from the monitoring network, with the Prometheus scraper and the health checks on that network segment. Ask the partner to show where the scraper connects, how the proxy blocks these paths for user traffic, and whether they verified that the block works from a user network. A red flag is a plan that publishes these paths through the same ingress as chat.

It is not encrypted by default. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, and must be protected by an isolated network. A good answer places all GPU nodes on a dedicated, non-routed network segment, restricts that segment to the cluster members, and adds transport encryption or a service mesh where the design requires it. Ask for the network diagram and the firewall rules that show which hosts can reach the node ports. A red flag is a multi-node design that shares a general purpose network with user traffic or workstations.

Set OLLAMA_NO_CLOUD=1 in the environment or in the server configuration file to disable cloud features. Ollama states that when it runs locally it does not see your prompts or data, and direct cloud inference at ollama.com requires an API key sent as a Bearer token. A good answer sets the variable in the service definition, confirms it in the running process configuration, and checks egress firewall rules so that the server cannot reach ollama.com even if the setting changes. Ask the partner to show the test that proves no outbound call is made. A red flag is a partner who has not checked this setting in the deployed environment.

Yes, and you should. Each control can be tested from a machine on a user network: attempt a direct connection to the Ollama and vLLM ports, call each vLLM path without a key, request /metrics and /health through the public ingress, and check outbound connections from the model host. Keep the results as evidence for your security review. DeployOpen can run security testing on AI systems and infrastructure as a separate engagement, and it prepares the controls and evidence for compliance assessments, but independent auditors issue any audit report. Ask your partner for a written test plan with expected results for each check.

The cost comes from design, configuration and testing rather than from software licenses, because Ollama and vLLM are open source and their security features are configuration choices. The work includes the network segmentation, the reverse proxy and its allowlist, the secrets handling for keys, the monitoring path, and the test evidence. Costs recur when the proxy rules, keys or model servers change with upgrades. Ask for a fixed-scope quote that lists each control, the environments covered, and the tests included. DeployOpen does not publish prices, and the quote follows from that written scope.

The quote follows from the written scope, which is agreed in writing before work starts. DeployOpen does not publish a price list, so the scope is the input that determines the price. Expect the scope to cover the number of users and concurrent sessions, the models and inference software to be deployed, the GPU or server capacity in your environment, the identity integration, the integrations with internal systems, security testing, backup and documentation, and any ongoing support. Ask the partner to itemize each of these so you can compare quotes on the same basis and see which assumptions drive the total.

One-time costs usually include GPU or server hardware (or the setup of cloud GPU capacity), implementation, identity and integration work, security testing and the initial documentation. Recurring costs usually include power, data center or cloud fees for GPU capacity, support and upgrades, operations staff time, and any vendor license or subscription. Ask the partner to separate these lines in the quote, and to state which items renew annually and which are billed per engagement phase.

Open WebUI code is distributed under the Open WebUI License, which prohibits altering or removing Open WebUI branding in deployments that exceed 50 users in any rolling 30-day period, unless the operator holds an enterprise license or written contributor permission. Enterprise licenses allow white-labeling and custom branding, and pricing is not published, so it is quoted after contacting Open WebUI enterprise sales with an expected seat count. Confirm with your legal team which exception applies to your deployment, and ask the partner to include the license line in the quote as a separate item if you need branding changes.

Ownership after go-live is a decision you should make before signing. Options include your own platform team running the stack, DeployOpen operating it under a support retainer, or a split where your team handles user administration and the partner handles upgrades and incident response. Ask the partner which tasks they would perform under each model, what response expectations they would commit to in writing, and what staff skills your team would need for GPU, container and identity operations. Ongoing support, upgrades and continuous testing are available as retainers.

Ask for runbooks, architecture records, configuration documentation and recovery procedures to be listed as named deliverables in the written scope, so they are part of what you pay for and accept. Useful runbooks cover adding a model, rotating API keys, restoring PostgreSQL and file storage, scaling replicas, and rolling back a model or interface upgrade. Confirm the format, the review process and who on your side signs off on each document, and check that the documentation covers your environment, not a generic template.

GPU capacity is usually the largest recurring infrastructure line, and it scales with the models you choose, the number of concurrent requests and the context length your users need. Ollama controls concurrency through OLLAMA_NUM_PARALLEL, which defaults to one parallel request per model, while vLLM is a serving engine with no paid edition, so its cost comes from accelerator capacity, the surrounding infrastructure and the people who operate it. Ask the partner to show the sizing assumptions behind the hardware line, and to state whether capacity is bought, leased or rented from a cloud provider.

Changes to scope, such as more users, extra integrations or additional models, should go through a written change request that states the effect on deliverables, timeline and cost before work begins. Ask the partner how change requests are priced, who approves them, and how they affect the go-live date. Keeping the original scope list with each agreed change makes it easier to audit what was delivered against what was paid for.