Which runtime holds up better under concurrent load for your team
Picture a team sizing a private AI service that many staff will call at the same time. The short answer is that Ollama's defaults suit light or single-user load, while vLLM is built for concurrent serving. Ollama's OLLAMA_NUM_PARALLEL defaults to one parallel request per model, OLLAMA_MAX_LOADED_MODELS caps how many models stay in memory, and OLLAMA_MAX_QUEUE holds up to 512 queued requests by default. Those controls let you handle a small group of users, but they do not describe a scheduler designed around many simultaneous streams. vLLM's engine batches requests continuously, exposes KV cache use through its Prometheus-format /metrics endpoint, and scales one model across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes.
What the right choice depends on is more specific than the headline. Four factors drive the result: the number of concurrent users and their typical prompt and output lengths, the GPU or CPU hardware you can dedicate, the size and quantization of the model, and the latency target your users will accept. A model that fits comfortably on one workstation for a few people may saturate its KV cache well before it saturates the GPU, so the bottleneck can differ from the one you expect. No throughput or latency figures for either runtime are supplied for this comparison, so you should not treat any published number as a planning value for your environment.
Measure both runtimes on your own hardware before you commit. Run the same model, quantization, prompt set and concurrency steps against each server, and record time to first token, tokens per second per request, error and timeout rates, and queue depth. On the vLLM side, watch the KV cache and request metrics at /metrics. Ask your hardware vendor and both projects to confirm which settings they recommend for your model size. Keep the endpoint behind a reverse proxy during testing, because vLLM's --api-key option does not cover every route, including /metrics.
How Ollama and vLLM handle concurrency, scaling and observability
This table compares how each runtime handles parallel requests, queues, model loading, GPU scaling, multi-node runs, metrics and licensing. Read it as a checklist: where Ollama's defaults stop fitting a multi-user workload, vLLM asks you to build more of the operating layer yourself.
| Area | Ollama | vLLM |
|---|---|---|
| Default parallelism | One parallel request per model by default, set with OLLAMA_NUM_PARALLEL | Batches concurrent requests in its scheduler; confirm the batching and concurrency settings for your model and GPU |
| Queue behavior | Up to 512 queued requests by default, controlled with OLLAMA_MAX_QUEUE | Queue and admission behavior depend on configuration; confirm limits and overload responses in the serving docs for your version |
| Model loading limits | OLLAMA_MAX_LOADED_MODELS defaults to 3 times the GPU count, or 3 on CPU | One model is served per server process; run separate processes or deployments for each model |
| Batching and KV cache | Parallel slots share the loaded model; no KV cache metrics are exposed in the supplied docs | Batches requests and exposes KV cache use through Prometheus metrics at /metrics |
| GPU and multi-GPU scaling | Runs on the available hardware; confirm multi-GPU behavior for your model size and driver stack | Tensor parallelism within a node and pipeline parallelism across nodes; data parallel for mixture-of-experts models |
| Multi-node support | Single machine deployment is the documented pattern; confirm any multi-node options before planning | Multi-node runs use Ray or a built-in multiprocessing backend; inter-node traffic is unauthenticated and unencrypted by default |
| Metrics exposure | No Prometheus endpoint in the supplied facts; confirm logging and metrics options with the project | Prometheus-format metrics at /metrics covering request counts, token throughput, latency histograms and KV cache use |
| API surface | REST API on port 11434 with OpenAI compatibility, plus Python and JavaScript SDKs | OpenAI-compatible endpoints including chat, completions, responses and embeddings; protect /metrics and /health with a reverse proxy |
| License and cost of software | MIT licensed; local runs are unlimited, and paid plans apply only to cloud models | Apache-2.0 with no paid edition; cost is GPU capacity, infrastructure and the staff who operate it |
What changes the result under real load
Throughput numbers from one benchmark rarely transfer to another setup. The factors below decide which engine wins for a given workload, so check each one against your own users, models and hardware before choosing.
Concurrent users versus concurrent requests
Ten users sending one request each is a different load from one user sending ten requests in parallel. Ollama defaults to one parallel request per model through OLLAMA_NUM_PARALLEL, so a busy team can queue behind a single generation. vLLM batches many requests together on the GPU, which is where its throughput advantage appears. Measure requests per second at the concurrency your users actually generate, not at a single-user benchmark.
Model size and quantization
The model's weights and KV cache must fit in GPU memory for either engine to batch well. Quantized models in Ollama reduce memory use and can keep a model on a single consumer GPU, but they change output quality and speed profiles. Run the same model and quantization on both engines before comparing numbers, and record which quantization each result used.
GPU count and interconnect
vLLM splits a model across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes, so a larger model can serve more concurrent sessions when GPUs are linked well. Ollama runs a model on the hardware it sees and does not offer the same multi-GPU serving model. Check how many GPUs a model needs, how they connect within a node, and whether multi-node traffic stays on an isolated network, since vLLM's security docs say node-to-node communication is insecure by default.
Prompt and output length
Short prompts with short answers finish quickly and hide the cost of memory use. Long context windows and long generations hold KV cache space for longer, which limits how many requests vLLM can batch and how many Ollama can keep resident. Build a benchmark set that mirrors your real prompt and output length distributions, and watch KV cache use through vLLM's metrics endpoint during the test.
Latency targets versus throughput targets
A chat assistant for a handful of engineers may only need a fast first token for each user, which a single-request runtime can deliver well. A batch summarization job or a shared service for hundreds of staff needs high aggregate tokens per second, which favors batching. Decide whether you are optimizing time to first token, p95 latency or total throughput, because the two engines trade these differently.
Tuning Ollama and vLLM settings
Ollama's OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE control parallel slots, resident models and queue depth, and raising parallelism also raises memory use. vLLM exposes its own batching and memory settings, and its metrics at /metrics report request counts, token throughput and latency histograms. Tune each engine to its best reasonable configuration and document every setting, so the comparison reflects the tuned result on both sides.
How vLLM scales across GPUs and nodes
vLLM scales one model by splitting it across hardware in layers, and each layer adds operating work. Within a node, tensor parallelism splits the model's weight matrices across GPUs so they act together on each token, which is the first lever to pull when one model does not fit on one card or when a single-node server needs more throughput. Across nodes, pipeline parallelism places consecutive groups of layers on different machines. Mixture-of-experts models can instead use data parallel deployment, where whole replicas of the engine serve traffic side by side. Ask your team which of these you actually need before you buy nodes, because a model that fits on one multi-GPU server rarely justifies cross-node pipeline stages.
Multi-node runs depend on a coordination backend. vLLM uses Ray or its own multiprocessing backend to start and connect workers, and the choice changes how you deploy, monitor and debug the cluster. Confirm which backend your chosen version and hardware support, and test a failed worker on a staging cluster before production traffic depends on it. Kubernetes users have several deployment options to compare, including native manifests, Helm and KubeRay, and the choice should follow your platform team's existing tooling rather than the benchmark chart.
The security docs state that communication between nodes is insecure by default, with no authorization or encryption, so inter-node traffic must sit on an isolated network. Treat the cluster interconnect as a private control plane: restrict it to the GPU nodes, keep it off user-facing segments, and verify the network policy with your security team before go-live. Scaling also raises the operating load: more nodes mean more driver and version alignment, more monitoring of the /metrics endpoint, more capacity planning and more incident paths. Get a written estimate of that ongoing work from any vendor or integrator, and compare it with the single-node Ollama setup you would be replacing.
How to benchmark both runtimes on your own hardware
This sequence produces a like-for-like load test of Ollama and vLLM on the same hardware and workload, so you compare measured throughput, queueing and tail latency instead of vendor claims. Each step ends with an artifact you can review before moving on.
Fix the model and quantization on both sides
The benchmark owner picks one open-weight model and one precision or quantization level, then confirms that both runtimes load the same weights. On Ollama, this means pulling a specific model tag and recording its quantization from the model details. On vLLM, it means pointing the server at the matching Hugging Face checkpoint. Record the exact model name, tag and checkpoint revision in the test log. The step is done when both runtimes answer a test prompt from the same model build, and the log shows identical identifiers. Results from mismatched weights are not comparable, so treat any difference as a blocker.
Define the prompt and output length distribution
The performance lead builds a request set that reflects your real traffic, including the spread of input prompt lengths and the expected output lengths, rather than one fixed prompt repeated. Use logs from an existing chat or application if you have them. If you do not, label the mix as illustrative and document it. Store the dataset and the random seed used to sample it. The step is complete when the dataset file, its length histogram and the seed are committed next to the test scripts, so anyone can rerun the same requests against both runtimes.
Set the concurrency levels to test
The benchmark owner chooses a ladder of concurrent request counts, from a single user up to the number of simultaneous users you expect at peak. Apply the same ladder to both runtimes. For Ollama, the server-side limit is set by OLLAMA_NUM_PARALLEL, which defaults to 1 parallel request per model, so record the value used for each run in the configuration log. For vLLM, record the scheduler settings that govern batching for the same run. The step is done when a written test matrix lists every concurrency level, the Ollama and vLLM settings for each, and the hardware it runs on.
Run warm-up requests before measuring
The test engineer sends a warm-up batch to each runtime before any measured run, so model loading, cache population and first-request overhead do not distort the numbers. For Ollama, confirm that the model is loaded and remains resident for the duration of the test. For vLLM, let the server finish startup and compilation before starting the clock. Discard warm-up results. The step is complete when the log shows the warm-up completed, the model was loaded before the measured window began, and no server restarts occurred during the measured runs.
Scrape vLLM metrics and record Ollama queue behavior
During each measured run, the test engineer scrapes the Prometheus-format endpoint at /metrics on the vLLM API server, which exposes request counts, token throughput, latency histograms and KV cache usage. Configure the scrape interval and keep the raw output. Confirm that the /metrics path is reachable only from the monitoring host, because the vLLM security guidance says that path is not covered by --api-key. For Ollama, record queue depth and how requests wait behind OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE, which defaults to 512 queued requests. Store both sets of data with timestamps. The step is done when each run has a matched metrics file for vLLM and a queue log for Ollama.
Measure time to first token and tail latency
The load generator records time to first token, time per output token and end-to-end latency for every request, then reports percentiles rather than averages alone. Tail values, such as the 95th and 99th percentile, show how each runtime behaves when requests queue. Record failed or rejected requests separately, since a queue limit being reached counts as a result. Compare the client-side numbers with the server-side histograms from the previous step to check for agreement. The step is complete when each concurrency level has a table of first-token and tail latency for both runtimes, with error counts shown beside them.
Repeat the matrix across GPU counts
The infrastructure owner reruns the full matrix with different GPU counts on the vLLM side, using tensor parallelism within a node as the documented scaling method, and records how throughput changes with each added GPU. Ollama is run on the same hardware configurations it can use, so the comparison stays fair. Note that scaling behavior depends on the model, interconnect and workload, so a single GPU count does not predict another. The step is done when a table exists for each GPU count, with the same workload and concurrency ladder applied to every configuration.
Document the limits of the results
The benchmark owner writes a short report that states the exact hardware, driver and software versions, model and quantization, workload dataset, concurrency settings and scrape configuration for every run. Include the conclusion that applies only to this hardware and workload, and list what was not tested, such as other models, longer contexts or multi-node runs. Recommend a re-run before any capacity decision that changes the hardware or user load. The step is complete when a reviewer can reproduce each run from the report and the stored artifacts, and the report carries a clear statement that the results hold only for the tested configuration.
Questions buyers ask before choosing or hiring help
How much effort does a fair Ollama versus vLLM benchmark take?
A fair benchmark takes a dedicated test rig, a fixed set of models and prompt lengths, a load generator that holds concurrency steady, and enough time to run each configuration more than once. The effort depends on how many concurrency levels and models you test and whether you measure time to first token, tokens per second and tail latency. Your team needs someone who can run containers or Python services on the target GPUs, read Prometheus-format metrics and explain why results differ between runs. vLLM exposes request counts, token throughput, latency histograms and KV cache use at /metrics, which makes the vLLM side easier to instrument. Ollama has no equivalent metrics endpoint in the supplied facts, so confirm what it exposes for your version before you plan the measurement.
Can Ollama be tuned for more concurrent users?
Yes, within limits. Ollama's concurrency is controlled by OLLAMA_NUM_PARALLEL, which defaults to 1 parallel request per model, OLLAMA_MAX_LOADED_MODELS, and OLLAMA_MAX_QUEUE, which defaults to 512 queued requests. Raising the parallel setting lets one model serve more requests at once, but each slot consumes memory for context, so the gain depends on the model size, context length and available VRAM. Ollama does not expose the batching and KV cache controls that vLLM provides, so the ceiling is set by the runtime design as well as the hardware. Test the tuned settings under your real request mix, and confirm the behavior for your Ollama version against its FAQ before you assume a throughput gain.
What drives the cost of running vLLM when there is no license fee?
vLLM is Apache-2.0 licensed and has no paid edition, so the cost comes from four places: GPU or other accelerator capacity, the surrounding infrastructure, implementation work and the people who operate it. GPU capacity is usually the largest recurring item and depends on model size, precision, context length and the concurrency you need to serve. Infrastructure includes networking, storage, load balancers, a reverse proxy for access control, and the Kubernetes or VM layer you choose. Implementation and operations cover setup, upgrades (vLLM ships minor releases about every two weeks), monitoring and security hardening. Ask any partner to split the quote into one-time setup and recurring operations, and to state which hardware assumptions the estimate uses.
Can our own team run this benchmark without outside help?
Often yes, if you have engineers who already operate GPU workloads and can write a repeatable load test. The work is less about installing either runtime and more about designing the test: matching models and quantization, fixing prompt and output lengths, controlling warm-up, and recording metrics the same way for both sides. Teams without that experience tend to produce results that reflect setup mistakes rather than the runtimes. A fixed-scope engagement can cover the test design, execution and a written report, while your team keeps ownership of the hardware. Decide early who will repeat the test when models or hardware change, because a one-off benchmark goes stale quickly.
How long does a benchmark and rollout take?
No duration is supplied for this work, so treat any timeline as unknown until a written scope exists. The main variables are the number of hardware configurations, the models and concurrency levels under test, how quickly GPU capacity is available in your environment, and whether the benchmark feeds a production rollout. Access approvals, network isolation for multi-node vLLM deployments and security review of the serving endpoints often take longer than the load tests themselves. Ask a partner to show the phases, the dependencies each phase waits on and the points where your team must provide access or sign-off.
What should we ask a partner to include in a fixed-scope quote?
Ask for the scope in writing before work starts, because most DeployOpen engagements are fixed-scope and agreed in writing before work begins. The quote should name the runtimes and versions under test, the hardware and models covered, the concurrency levels and metrics measured, and the report format. It should also list what is excluded, such as GPU purchases, cloud accounts, production rollout or ongoing benchmarking. For vLLM, ask whether the scope covers securing internal node traffic, placing a reverse proxy that allowlists user endpoints, and protecting /metrics. Separate one-time work from retainers for ongoing support or repeat testing. DeployOpen does not publish a price list, so the quote follows from the written scope.
Does a faster benchmark result mean the runtime is safe for production?
No. A throughput result shows how a configuration behaves under one load pattern, not whether the endpoint is secure or operationally ready. The vLLM security documentation says the --api-key option does not protect every endpoint, including /metrics, /health and /invocations, and recommends a reverse proxy that allowlists only the endpoints meant for users. Ollama binds to 127.0.0.1 by default and its local API requires no authentication, so exposing it through a proxy is a separate design task. Include access control, logging and network isolation in the evaluation alongside performance, and confirm each control against the current documentation for your version.
Why load testing must include the security perimeter
A load test that only measures tokens per second tells you how the model performs, not whether the endpoint is safe to expose. Both servers need a perimeter decision before the benchmark numbers matter, because the performance result assumes traffic reaches the model through a path you control. Treat the security configuration as part of the test setup, not a follow-up task.
For vLLM, the --api-key option is not a complete control. The vLLM security docs state that it protects the endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The docs warn against relying on the key alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Confirm the exact endpoint list for your installed version, then check that /metrics is unreachable from user networks, because it exposes request counts, token throughput and KV cache use that help an attacker profile your capacity. Also confirm that multi-node traffic stays on an isolated network, since the same docs say inter-node communication is insecure by default.
Ollama takes the opposite default. Its local API at port 11434 does not require authentication, and it binds to 127.0.0.1 unless you change OLLAMA_HOST. If you widen that bind address for a load test or for users, put an authenticating proxy such as Nginx in front of it before any client connects. Run your load tests against the proxied address, so the measured latency includes the proxy and the queue behavior you will actually ship, and record the OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE values alongside the results.
How DeployOpen helps you test and implement this
DeployOpen plans, implements, tests and documents self-hosted open-source platforms inside your own infrastructure, including the load testing needed to compare Ollama and vLLM on your hardware. We start from your user counts, models and concurrency targets, then build and measure both candidates against the same workload so the results reflect your environment.
How we scope
- Agree the user count, concurrency targets, model list and hardware profile in writing before any testing begins
- Define the benchmark workload, prompt and output lengths, and the latency and throughput measures that count as success
- Set the deliverables, such as a benchmark report, sizing notes and a handover runbook, in a fixed-scope agreement
How we build
- Install Ollama and vLLM in your cloud accounts, data centers or air-gapped network, with the same models and parallelism settings for each run
- Run load tests at increasing concurrency, capturing request queueing, token throughput and latency from Ollama's settings and vLLM's /metrics endpoint
- Test access controls, placing a reverse proxy in front of vLLM so only user-facing endpoints are exposed
How we hand over
- Document the test scripts, raw results, configuration values and the hardware each figure came from so your team can repeat them
- Deliver a written comparison of throughput, concurrency limits and operating effort, with the conditions under which each result holds
- Walk your platform and security engineers through the deployment, monitoring and scaling steps before handover
How we support
- Offer ongoing support, upgrades and re-testing as a retainer when Ollama or vLLM release new versions and your load grows
- Re-run the benchmark suite after hardware changes, model swaps or major version upgrades so sizing stays current
- Keep the documentation aligned with the running system so changes are recorded and easy to review
