Which of the three fits your situation
If your team needs to serve open-weight models on hardware you control, the deciding factors are how many people will call the service at once, what hardware you have, and how much security hardening you need to add yourself. Ollama fits small, low-concurrency use, while vLLM fits high-throughput GPU serving, and llama.cpp sits between them when you want fine control over the runtime. Work through those three questions before you look at features.
Choose Ollama when the goal is a packaged runtime that a developer, a small team or an edge device can start quickly. It is MIT licensed, runs on macOS, Windows, Linux and Docker, and exposes a REST API on port 11434 with OpenAI compatibility. Its defaults serve one parallel request per model, so confirm your concurrency needs against OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE before you promise a shared service. The local API has no authentication by default and binds to 127.0.0.1, so any wider exposure needs a proxy you run yourself.
Choose llama.cpp when you need direct control over quantization (its README lists 1.5-bit to 8-bit integer quantization), hardware backends such as Metal, AVX-family CPU instructions, CUDA, HIP, Vulkan or SYCL, or specific llama-server features. llama-server offers continuous batching and multi-user parallel decoding, and it supports API keys, TLS and an optional metrics endpoint. The trade-off is more configuration work, which your platform team should size against its own staff. Choose vLLM when you run NVIDIA, AMD or Intel GPUs at volume and want OpenAI-compatible endpoints, tensor and pipeline parallelism and Prometheus metrics. vLLM does not ship an authentication model you can rely on alone, so place a reverse proxy that allowlists only user-facing endpoints in front of it. Confirm the hardware, GPU count and security design in a pilot before you commit.
Which platform fits which workload
This table maps each serving job to the platform that fits it best and the condition under which that choice goes wrong. Read each row as a decision: match your hardware and user count to the first column, then check the wrong-choice column before you commit.
| Workload | Best fit | Reason | Wrong choice when |
|---|---|---|---|
| Laptop and personal use | Ollama | Single binary with MIT license, local REST API on port 11434 and model management from the CLI; local use does not send prompts to Ollama, per its FAQ. | Several people need concurrent access, because the default is one parallel request per model (OLLAMA_NUM_PARALLEL). |
| Edge device or small workstation | llama.cpp | Runs GGUF models on a wide range of hardware, with 1.5-bit to 8-bit quantization to reduce memory use. | You want managed model pulls and a simple daemon, since llama.cpp is run directly and model handling is more manual. |
| Single server, CPU or mixed hardware | llama.cpp (llama-server) | Continuous batching and parallel decoding with multi-user support, plus API key, TLS and metrics options on the server. | You need multi-GPU tensor or pipeline parallelism across nodes, which vLLM is built for. |
| Small team on one server with CPU or modest GPU | Ollama or llama-server | Ollama is simpler to operate; llama-server adds TLS, API key files and Prometheus metrics that Ollama's local API does not expose by default. | You need to expose the endpoint to many users without a reverse proxy, since Ollama's local API has no authentication. |
| High-throughput GPU serving | vLLM | OpenAI-compatible endpoints, tensor parallelism within a node, pipeline parallelism across nodes and Prometheus metrics that include KV cache use. | You have no GPU capacity or a single user, because the GPU and operating cost is paid whether or not traffic justifies it. |
| Multi-node serving | vLLM | Supports tensor and pipeline parallelism across nodes, with Ray or its own multiprocessing backend. | Nodes cannot sit on an isolated network, because inter-node traffic is insecure by default and must be protected by network isolation. |
| Public or shared endpoint | vLLM behind a reverse proxy, or llama-server with TLS | vLLM's --api-key does not cover /metrics, /health or /invocations, so the docs recommend a proxy that allowlists user endpoints. | You rely on Ollama's local API as the only control, because it does not require authentication on localhost:11434. |
| Cost of running | Ollama and llama.cpp (no paid edition for vLLM's engine) | Running Ollama models on your own hardware is listed as unlimited on its pricing page; vLLM has no paid edition, so its cost is accelerator capacity and operating staff. | You want to pay for hosted cloud models through Ollama, since those plans are separate from local use; confirm current plan terms on the pricing page. |
How each runtime handles concurrent requests
Ollama queues work by default, so one request per model runs at a time. OLLAMA_NUM_PARALLEL sets how many requests a model serves in parallel, and its default is 1. OLLAMA_MAX_LOADED_MODELS controls how many models stay resident in memory, with a default of 3 times the number of GPUs, or 3 on CPU. OLLAMA_MAX_QUEUE caps waiting requests and defaults to 512. Raising parallelism gives each extra slot its own share of memory, so a small team on one workstation should raise OLLAMA_NUM_PARALLEL modestly, then watch memory use and response times under its own load. Confirm current defaults in the Ollama FAQ for the release you deploy.
llama-server handles concurrency inside a single loaded model. Its README lists parallel decoding with multi-user support and continuous batching, so several users can share one model without each request waiting for the previous one to finish. More concurrent users generally means less context available to each of them, and that trade-off is the decision to plan first. llama-server also provides API key authentication, TLS and an optional metrics endpoint enabled with --metrics, which matter once several people share the same server. Check the llama-server README for the slot and context settings in your build, then size the deployment against the model, quantization and hardware you intend to use.
vLLM is built around batching for GPU serving. It groups active requests together and depends on the KV cache, the memory that holds attention state for each sequence in progress, so KV cache use is the number to watch. The server exposes Prometheus-format metrics at /metrics, including KV cache use, token throughput and latency histograms, so you can see whether the cache is filling before users notice queueing. Capacity planning therefore starts with a load test on your own GPUs, read through those metrics. Treat every throughput figure, including figures from vendor pages and from this page, as unverified until your own benchmark confirms it. The sibling comparison, Ollama vs vLLM performance, explains how to run that benchmark for both runtimes on your hardware. The vLLM metrics documentation lists the available series.
What each license allows and what is paid
Ollama's runtime is licensed under the MIT license, and its pricing page states that running models on your own hardware is always unlimited. The paid plans apply to Ollama's cloud-hosted models, which are a separate service from the local runtime. A team that keeps inference on its own machines pays no license fee for the runtime, but any cloud tier is a commercial decision that needs its own review.
- MIT license: The runtime's code is under MIT. Model weights carry their own licenses from their publishers, so check those terms model by model.
- Local use: The pricing page lists running models on your own hardware as unlimited, with no per-token charge for local inference.
- Cloud plans: The pricing page lists Pro at $20 per month ($200 per year), Max at $100 per month, Team at $500 per month and Enterprise at custom pricing. These plans cover cloud models only.
- Enterprise additions: The Enterprise plan adds model access controls, cost budgets per user or API key, a dedicated support channel and custom security options on top of the Team plan's features.
- Local-only lock: Setting
OLLAMA_NO_CLOUD=1turns cloud features off, which helps teams that must keep inference inside their own network.
Before you rely on a cloud tier, ask Ollama in writing which terms cover your prompts and usage data, what limits apply and whether Enterprise terms fit your deployment.
vLLM is licensed under Apache-2.0, and the project has no paid edition. The cost of running it comes from the accelerators that serve the model, the infrastructure around them, the work to implement and harden the deployment, and the people who operate it. Most of that cost sits with your organization, so the license is the simplest part of the decision.
- Apache-2.0 license: The license permits use, modification and redistribution. Confirm any attribution and notice duties with your legal team before you ship a modified build.
- No paid edition: There is no commercial tier to buy. Any paid support comes from a third party or from your own staff, so ask any provider for a written scope.
- Accelerator capacity: GPU or other accelerator spend is usually the largest line. Purchased hardware is a one-time cost, while rented capacity recurs every month.
- Operating effort: The project ships new minor releases about every two weeks, so plan regular testing and rollouts of new versions.
- Hardening work: The security docs state that multi-node traffic is insecure by default and that
--api-keydoes not cover every endpoint, so reverse proxies, network isolation and allowlists are engineering work your team owns.
When you request a quote from any implementation partner, ask for the hardware sizing, the support scope in writing, the hardening tasks included and which operational tasks stay with your staff.
llama.cpp is listed under the MIT license in its repository metadata. The public materials this comparison relies on do not describe a paid edition, so the costs you should plan for are the hardware you run it on, the staff who build and maintain it and any hosting you add around it. Confirm commercial terms with the maintainers before you depend on them in a contract.
- MIT license: The repository metadata lists MIT for the project. Model files come from many publishers and carry their own licenses, so review each model you deploy.
- No paid edition: The project's public materials do not describe a commercial tier. If you need a contractual commitment, ask the maintainers or an authorized provider in writing.
- Hardware cost: The engine runs on Apple silicon, x86 CPUs, NVIDIA and AMD GPUs and Vulkan or SYCL backends. Cost follows the hardware you already own or buy, and 1.5-bit to 8-bit quantization can reduce the memory a model needs.
- Build management: Numbered builds publish continuously, so pin a build and test it before each upgrade.
- Server controls:
llama-serversupports API keys, TLS and an optional metrics endpoint, but your team configures, monitors and renews them.
Before you commit, confirm the license of each model you plan to run, decide how your team pins and tests builds and name the person who owns security patches for the server.
What drives the running cost of each platform
Running cost for these three runtimes comes from different places: accelerator capacity, host memory, the people who operate the stack and, for Ollama, optional cloud plans. The cards below separate what is paid once from what recurs, and list what to request in a written quote before comparing totals.
Accelerator capacity for vLLM
vLLM has no paid edition, so its license cost is zero and its main recurring expense is GPU or other accelerator capacity, whether purchased or rented. Capacity needed depends on model size, context length, concurrency and whether the model is split across GPUs with tensor or pipeline parallelism. Ask for the accelerator count, memory per device, interconnect assumptions and the throughput target a quote is sized against. Idle reserved GPUs are the cost most often overlooked.
Host hardware and memory for llama.cpp
llama.cpp is MIT licensed, so software cost is zero and spending goes to the host: CPU, system RAM, and GPU memory if a supported backend is used. Quantization level changes how much memory a model needs, so the same model can fit on modest hardware at lower precision. Ask the integrator to benchmark the chosen GGUF quantization on the target machine and to record which backend is compiled in. Mixed hardware fleets add build and testing time.
Host hardware for Ollama
Ollama is MIT licensed and runs models on your own hardware at no software charge, so its running cost is the same kind of host spend as llama.cpp: CPU, RAM and any GPU. Its default concurrency of one parallel request per model means teams that need more simultaneous users may add hosts or replicas, which raises hardware cost. Confirm how many instances, models and users the quote assumes, and what happens when the queue limit is reached.
Cloud plan fees if Ollama is used
Running models on your own hardware is listed as always unlimited on Ollama's pricing page. Cloud models are a separate recurring charge: the Pro plan is listed at $20 per month ($200 per year), Max at $100 per month, Team at $500 per month and Enterprise at custom pricing, per the Ollama pricing page. Before any cloud use, confirm which plan applies, whether usage credits or limits apply, and whether OLLAMA_NO_CLOUD=1 is set to keep the install local.
People who operate each stack
Staff time is often the largest recurring cost and it differs by platform. vLLM at scale usually needs people who manage GPU drivers, multi-node networking, isolation between nodes and a reverse proxy that allowlists user endpoints. llama.cpp and Ollama need less platform work for a single host but still need patching, model updates and monitoring. Ask any implementer for the named roles, the hours they expect for upgrades and the on-call coverage included in the fixed scope.
One-time setup and recurring upgrades
Setup work such as hardening, proxy configuration, TLS, metrics wiring and benchmarking is usually a one-time cost, while upgrades recur. vLLM ships new minor releases about every two weeks, and Ollama ships frequent releases, so the upgrade cadence is a recurring line item. Ask for the upgrade and testing process, whether retainers include upgrades, and which items are fixed-scope deliverables with written acceptance criteria.
How much hardening each one needs before users touch it
Out of the box, the three runtimes ask for very different amounts of hardening, and the gap matters most once more than one person touches the server. Ollama binds to 127.0.0.1 on port 11434 by default and its local API does not require authentication. Anything that makes it reachable from other machines, including changing OLLAMA_HOST or placing it behind a reverse proxy such as Nginx, means the proxy must supply access control, TLS and logging. Confirm this with the Ollama FAQ and test from a second host on your network before you open the port to users.
llama-server gives you more of the controls in the binary itself. It supports API keys through --api-key or an API key file, TLS through PEM certificate and key files, and an optional Prometheus-compatible metrics endpoint enabled with --metrics. Those flags are still settings you must choose and verify. Check that the key is required on every route your users call, that the certificate chain is correct for your clients, and that the metrics endpoint is not exposed to people who should not see request volumes.
vLLM needs the most deliberate boundary. The --api-key option protects routes under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security documentation warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints meant for users. For multi-node runs, inter-node communication is insecure by default, without authorization or encryption, so those nodes must sit on an isolated network. Ask your platform team to confirm the allowlist and network segmentation in writing before go-live.
What implementing each platform involves
This sequence walks through the work that turns each runtime into a service your applications can call, from where model files live to how the service is hardened, watched and upgraded. Read it to see where the effort sits for a laptop-scale Ollama install, a llama.cpp server or a vLLM GPU deployment.
Place model files and storage
The platform team decides where model weights live before anything else, because the three runtimes store them differently. Ollama keeps models in its own directory, which is ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and the .ollama\models folder on Windows, and the OLLAMA_MODELS variable moves that directory. llama.cpp loads GGUF files that you download and place yourself, for example from Hugging Face. vLLM loads model weights onto GPUs at startup, so the storage and GPU memory plan must fit the model size and parallelism you choose. Done means each host has a documented model path, a known disk budget and a test model that loads successfully.
Set concurrency and hardware parameters
Next, the team tunes the settings that decide how many users one server can serve. For Ollama, the relevant variables are OLLAMA_NUM_PARALLEL, which defaults to one parallel request per model, OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE, which defaults to 512 queued requests. For llama-server, parallel slots and continuous batching are configured on the command line. For vLLM, the team chooses the GPU backend, such as NVIDIA CUDA, AMD ROCm or Intel XPU, and sets tensor parallelism within a node. Done means each runtime starts on the target hardware and a short load test shows the concurrency limits you intend to run with.
Wire the API into applications
Applications connect through each runtime's HTTP API. Ollama listens on 127.0.0.1 port 11434 by default and exposes an OpenAI-compatible API, so the bind address must be changed with OLLAMA_HOST before other machines can reach it. llama-server offers OpenAI-compatible chat completions, responses and embeddings routes, plus Anthropic Messages compatibility. The vLLM server started with vllm serve exposes /v1/chat/completions, /v1/responses and /v1/embeddings, so clients written for the OpenAI API can point at it. Done means one test client from each application succeeds through the gateway or proxy you have chosen, and the base URL and model names are recorded in your configuration standard.
Harden the endpoint and network
Hardening differs sharply between the three. The local Ollama API at localhost:11434 requires no authentication, so exposing it means placing a reverse proxy such as Nginx in front and enforcing access there, and setting OLLAMA_NO_CLOUD=1 if cloud features are not wanted. llama-server supports API keys through --api-key or a key file, TLS through PEM certificate and key files, and metrics through --metrics. vLLM's --api-key does not cover every route, including /metrics, /health and /invocations, and its security guidance recommends a reverse proxy that allowlists only user-facing endpoints. Multi-node vLLM traffic is insecure by default, so nodes need an isolated network. Done means an external scan or a manual request test confirms that only intended routes are reachable and that inter-node traffic is isolated.
Stand up monitoring and logging
The team then decides how it will see load, errors and resource use. vLLM serves Prometheus-format metrics at /metrics covering request counts, token throughput, latency histograms and KV cache use, so it fits a standard scrape setup. llama-server exposes a Prometheus-compatible endpoint when started with --metrics. Ollama's documentation does not describe a comparable built-in metrics endpoint in the supplied facts, so confirm what is available in your version and plan to collect host and process metrics separately. Done means dashboards show request rate, latency and GPU or CPU memory for each runtime, and an alert fires when the queue or memory approaches its limit.
Choose a Kubernetes platform for vLLM
If the GPU service will run on Kubernetes, this is the step that needs a platform decision before work starts. The vLLM Kubernetes documentation lists several deployment options, including native manifests, Helm, the vLLM production-stack, AIBrix, KServe, KubeRay, LeaderWorkerSet, llm-d and NVIDIA Dynamo, and each one brings different operators, scaling behavior and maintenance. Ollama and llama.cpp are normally run as ordinary services or containers on a host, so they do not require this choice. Done means the team has written down which option it adopts, who owns its upgrades, and how it will be tested before production.
Plan upgrades around each release cadence
Upgrades follow the release patterns of each project. Ollama ships frequent releases, including v0.40.5 on 14 October 2026 and v0.40.6 as a pre-release on 16 October 2026. vLLM ships new minor releases about every two weeks, with v0.31.0 on October 5, 2026. llama.cpp publishes numbered builds continuously, with b11717 on October 18, 2026. Because of this pace, pin the version you run, test a candidate build against your own models and prompts, and keep a rollback path. Done means a documented upgrade runbook exists, a staging host runs the candidate version first, and the team records which version is in production.
Document ownership and recurring effort
The last step writes down who runs each service after launch. Recurring work includes model updates, runtime upgrades, security patching of the proxy and host, capacity review and monitoring tuning. Costs that recur are staff time and, for vLLM, GPU capacity, while Ollama and llama.cpp carry no license fee for local use under their MIT licenses. Ask any implementation partner for a written scope that names the runtimes, models, hosts, endpoints and handover documents so the cost of operating each option is visible before you choose. Done means an owner, an on-call path and a runbook are assigned for every component in the stack.
Questions buyers ask before choosing a runtime
What does it cost to implement a self-hosted model runtime?
Implementation cost is driven mainly by people and hardware, not by the runtime license, because Ollama, vLLM and llama.cpp are all open source and carry no license fee for self-hosting. The cost components are:
- Hardware or cloud accelerator capacity, which recurs monthly or is a one-time purchase.
- Platform engineering time to install, secure, monitor and upgrade the server.
- Implementation services, if you hire a partner, which are priced from a written, fixed scope.
GPU-heavy vLLM deployments usually carry the largest hardware line. Ask any provider for a quote that separates one-time work from recurring upkeep and lists the hardware assumptions behind it.
How long does a pilot take to run?
A pilot's duration depends on the scope you agree in writing, the number of models, the hardware you already own and how many users or applications will call the endpoint. A single-user Ollama evaluation on a workstation can be set up quickly, while a vLLM pilot on GPU servers also needs driver, network and access-control work. No timeline figure is given here. Ask for a schedule tied to named deliverables, and confirm what access, hardware and approvals your team must provide before work starts.
Can our own team run a model server without outside help?
Yes, if you have people who can operate Linux services, manage model files, rotate credentials and read metrics. Ollama is the lightest to operate, because it exposes a local REST API on port 11434 that binds to 127.0.0.1 by default. vLLM and llama.cpp need more deliberate configuration for network exposure, authentication and monitoring. Outside help is most useful for the hardening, reverse proxy design and upgrade process, which recur for as long as the server runs. DeployOpen implements and supports self-hosted platforms inside your own infrastructure, and ongoing support is available as a retainer.
Which runtime suits a regulated environment?
None of the three runtimes is certified for any regime by its license, and DeployOpen does not issue audit reports or certifications. The choice depends on your controls. Ollama states that local runs do not expose prompts or data to Ollama, and cloud features can be disabled with OLLAMA_NO_CLOUD=1. vLLM's security documentation says inter-node traffic is insecure by default and recommends a reverse proxy that allowlists user endpoints. llama-server adds API key, TLS and metrics options. Map these to your control set with your assessor, and confirm each setting in your own environment.
Can llama.cpp replace Ollama for our team?
Often yes, but it is a different kind of tool. Ollama is a packaged runner with model management, and llama.cpp is the C/C++ engine that many local stacks sit on, with llama-server as its HTTP server. llama-server adds API key authentication, TLS, a Prometheus-compatible metrics endpoint and continuous batching with multi-user support. You take on model file handling and command-line configuration that Ollama would otherwise manage for you. Test your own model and hardware before switching, because the operating effort changes.
Does vLLM need GPUs to run?
vLLM is built for accelerator serving, and its installation documentation covers NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, along with x86 and ARM CPUs and Apple silicon. Running on CPU is possible on supported platforms, but the throughput advantage of vLLM comes from batching and GPU parallelism. For a small team on mixed hardware, Ollama or llama.cpp may fit better. Confirm the exact backend support for your hardware and driver versions in the installation docs before you buy.
What should we confirm about licensing for commercial use?
All three runtimes are open source, with Ollama and llama.cpp under MIT and vLLM under Apache-2.0, according to their repositories. The license of the runtime does not cover the model weights you load. Each model has its own terms, so check the license attached to every model you plan to serve, including any restrictions on commercial use or redistribution. Ollama's paid plans apply to its hosted cloud models only, and running models on your own hardware is listed as unlimited. Have legal counsel review the terms for each model before production use.
What switching between runtimes costs later
Ollama and llama.cpp both load GGUF-based models, so a move between them is mostly a runtime change rather than a model re-packaging exercise, but you should still confirm which model formats, tags and quantizations each project supports for your exact models before you commit. vLLM is different: it uses its own loading path and GPU setup, including CUDA, ROCm or other supported accelerator stacks, so expect the model preparation and hardware configuration to be redone rather than carried over. The one thing that eases the change is the OpenAI-compatible API. Ollama exposes an OpenAI-compatible endpoint on port 11434, llama-server offers OpenAI-style chat and embeddings routes, and vLLM serves /v1/chat/completions and /v1/embeddings, so application code often needs only a base URL and model name change.
Compatible routes do not mean identical behavior. After any move, retest client code for differences in streaming, tool calling, token limits, error formats and model naming, and rerun your evaluation set against the new runtime. Monitoring is a separate cost: vLLM exposes Prometheus metrics at /metrics by default, llama-server exposes them only when started with --metrics, and Ollama's monitoring surface should be checked against what your dashboards expect. Security controls also move with the runtime. vLLM's --api-key does not cover every endpoint, so its reverse proxy allowlist must be rebuilt, while llama-server's --api-key and TLS settings and Ollama's bind address and proxy setup each need their own review. Treat the proxy and gateway layer as part of the migration, not an afterthought.
Plan the move as a short test cycle before cutover. Confirm model format and quantization support with each project's current documentation, then run a side-by-side test on your own hardware with the same prompts, load pattern and logging. Record which configuration keys, such as OLLAMA_HOST or the vLLM serve options, your deployment scripts depend on, and keep them in version control so the next switch is a documented change. Ask your implementation partner to include these retests in the written scope, since they are the costs most often missed when the first runtime was chosen for convenience.
How DeployOpen helps choose and implement a model runtime
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, including cloud accounts, data centers and air-gapped networks. For Ollama, vLLM and llama.cpp, that means matching the runtime to the job, the hardware you own and the access controls your security team requires.
How we scope
- Confirm the user count, concurrency target and hardware (laptop, CPU server or GPU cluster) before choosing a runtime.
- Agree in writing which API clients, model formats and quantization levels the deployment must support.
- Record the authentication, TLS and network isolation requirements that the written scope must meet.
How we build
- Install the chosen runtime, configure bind address, model storage location and concurrency settings in your environment.
- Place a reverse proxy in front of endpoints meant for users, allowlisting only the routes you intend to expose.
- Connect metrics endpoints to your monitoring stack and test request throughput against the agreed targets.
How we hand over
- Deliver configuration files, runbooks and a record of the tested endpoints, limits and known gaps.
- Document the upgrade path for each runtime, including how your team tracks its release cadence.
- Walk your operators through model updates, key rotation and troubleshooting before the handover is closed.
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
- Ongoing support, upgrades and continuous security testing are available as a retainer.
- Confirm with us which upgrades, runtime versions and hardware changes fall inside a retainer before you sign.
