Self-hosted LLM inference servers compared

vLLM, Ollama and llama.cpp compared for serving open-weight models in your own infrastructure, with SGLang as the other high-throughput GPU engine. Find the fit by team size, hardware and operating model.

Book a callTalk to us
On this page

Self-hosted LLM inference servers compared: which engine fits which team

If your team must serve open-weight models inside its own network, choose vLLM for multi-user GPU serving, Ollama for simple local or small-team use, and llama.cpp for lean, hardware-flexible or edge deployments. All three are open source with no paid edition for self-hosting, so the real cost sits in hardware, implementation and day-to-day operations. SGLang is the other high-throughput GPU engine covered on this page, and it belongs on the shortlist whenever GPU throughput under concurrent load is the deciding factor.

The fit depends on who calls the server. vLLM exposes OpenAI-compatible routes such as /v1/chat/completions and /v1/embeddings, supports tensor parallelism within a node and pipeline parallelism across nodes, and runs on NVIDIA CUDA, AMD ROCm and Intel XPU GPUs. Ollama listens on 127.0.0.1 port 11434 by default and has no authentication on that local API, so a shared deployment needs OLLAMA_HOST changed and a proxy in front of it. llama.cpp's llama-server adds continuous batching, TLS through PEM files and an optional --metrics endpoint, and it runs on Apple silicon, x86 CPUs and a range of GPU backends, which suits hardware that is not a datacenter GPU.

Before you pick, confirm three things in your own environment: the accelerators you can actually provision, the number of users who will call the server at the same time, and who will patch and monitor it. Ask any vendor or integrator for a written scope that names the hardware, the expected concurrency, the security controls and the operating runbook, because a quote cannot be compared without them. Plan the switching cost too. Engines use different model formats and tuning settings, so an early choice made for a pilot can turn into a migration later.

Which inference server fits which scenario

This table matches common deployment scenarios to the inference server most likely to fit, with the reason and the points to confirm before committing. Read each row as a starting point: the right choice depends on your hardware, user count and whether you need built-in access controls. SGLang is noted only where it is a candidate, since its facts were not supplied here.

ScenarioBest fitWhyWhat to confirm
Many concurrent users on NVIDIA GPUsvLLMServes OpenAI-compatible endpoints with multi-GPU tensor parallelism and Prometheus metrics, built for shared serving load.Your GPU model and count, expected concurrency, and the throughput you measure on your own model and prompts.
Developer or small team on laptopsOllamaRuns locally on macOS, Windows, Linux and Docker with a simple REST API, and local use does not send prompts to a service.Model sizes that fit each laptop's memory, and whether one shared server is needed for the team.
CPU or mixed hardware with limited GPUsllama.cpp (llama-server)Supports GGUF models with 1.5-bit to 8-bit quantization and several CPU and GPU backends, including Vulkan and SYCL.Quantization level that keeps answer quality acceptable, and throughput under your expected parallel load.
Apple silicon or edge devicesllama.cpp or Ollamallama.cpp lists Apple silicon via Metal and quantized models for lower memory use; Ollama runs on macOS and Linux with llama.cpp underneath.Memory headroom on the target device and whether the device must work fully offline.
Multi-node Kubernetes servingvLLMSupports pipeline parallelism across nodes and several Kubernetes deployment options, including Helm and KubeRay.Whether inter-node traffic is isolated, since vLLM docs state it is insecure by default, and your Ray or multiprocessing choice.
Teams that need built-in API keys and TLSllama.cpp (llama-server) or vLLM behind a proxyllama-server offers --api-key, PEM-based TLS and optional metrics. vLLM's --api-key does not cover every endpoint, so a reverse proxy is advised.Which endpoints each client needs, the proxy you will run, and certificate management in your environment.
Alternative to vLLM for high-throughput GPU servingSGLang (candidate only)Listed here only as another high-throughput GPU engine. SGLang facts were not supplied for this page.Check the SGLang documentation for hardware support, API surface, licensing and operating model before comparing it with vLLM.

How vLLM, Ollama and llama.cpp differ in design

The core difference is what each engine was built to optimize. vLLM is a GPU serving engine built for many concurrent requests against a model that stays resident on accelerators. It scales one model across GPUs with tensor parallelism inside a node and pipeline parallelism across nodes, and it supports data parallel deployment for mixture-of-experts models. Multi-node runs use either Ray or vLLM's own multiprocessing backend. The node-to-node traffic is insecure by default, so the vLLM security docs require an isolated network for those nodes. For monitoring, the API server exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. Confirm that the /metrics path is reachable only from your monitoring network, because the vLLM security docs note that the --api-key option does not protect it.

Ollama is a local model runtime written in Go with a llama.cpp backend. Its design centers on a single host and a simple REST API on port 11434, which also offers OpenAI compatibility. It binds to 127.0.0.1 by default, so exposing it to other machines means setting OLLAMA_HOST and placing a proxy such as Nginx in front, and the local API has no authentication built in. Concurrency is governed by three settings: OLLAMA_NUM_PARALLEL (default 1 parallel request per model), OLLAMA_MAX_LOADED_MODELS (default 3 times the number of GPUs, or 3 on CPU) and OLLAMA_MAX_QUEUE (default 512 queued requests). Before you size a shared deployment, test these values against your peak concurrency, because the defaults favor a single user at a time.

llama.cpp is a C/C++ inference project that runs models in the GGUF format and targets a wide range of hardware, including Apple silicon via Metal, x86 AVX variants, NVIDIA through custom CUDA kernels, AMD through HIP, and Vulkan and SYCL backends. Its server, llama-server, adds continuous batching and parallel decoding with multi-user support, along with an optional Prometheus-compatible metrics endpoint enabled with --metrics. It also supports API key authentication through --api-key or a key file, and TLS through PEM certificate and key files. Those controls are closer to what a shared service needs than Ollama's defaults, but confirm the batching and slot settings against your own load tests, since the README describes the features without giving throughput figures.

Capability comparison across vLLM, Ollama and llama.cpp

This table compares license, hardware, API, batching, scaling, security and operations facts for the three engines, with SGLang noted separately. Where a supplied fact is missing, the cell tells you what to confirm. Read the rows for hardware and multi-node scaling first, since they usually decide the shortlist.

CapabilityvLLMOllamallama.cppSGLang
LicenseApache-2.0MITMITNot covered in supplied facts. Confirm the license in its repository before shortlisting.
Hardware supportNVIDIA CUDA, AMD ROCm and Intel XPU GPUs; x86, ARM AArch64, Apple silicon and IBM Z CPUsmacOS, Windows, Linux and Docker; runs on the llama.cpp backendApple silicon via Metal, x86 AVX to AMX, NVIDIA CUDA, AMD via HIP, plus Vulkan and SYCLNot covered in supplied facts. Confirm GPU and CPU support with the project.
OpenAI-compatible endpointsYes: completions, chat completions, responses, embeddings and audio transcription and translationYes, REST API on port 11434 with OpenAI compatibilityYes: chat completions, responses and embeddings routesNot covered in supplied facts. Confirm endpoint coverage.
Other API routesNot covered in supplied facts beyond OpenAI-style endpoints. Confirm before relying on any other route.Python and JavaScript SDKsAnthropic Messages API compatibility and a reranking endpointNot covered in supplied facts. Confirm.
Batching and parallel decodingIncluded in the serving engine. Confirm batching behavior for your model and hardware.OLLAMA_NUM_PARALLEL defaults to 1 per model; queue depth set by OLLAMA_MAX_QUEUE (default 512)Parallel decoding with multi-user support and continuous batchingNot covered in supplied facts. Confirm throughput features.
Multi-GPU and multi-node scalingTensor parallelism within a node, pipeline parallelism across nodes, data parallel for mixture-of-experts; Ray or own multiprocessing for multi-nodeNot covered in supplied facts. Confirm multi-GPU behavior with the project.Not covered in supplied facts. Confirm multi-GPU behavior in the llama-server documentation.Not covered in supplied facts. Confirm.
QuantizationNot covered in supplied facts. Confirm supported formats for your model.Models run on the llama.cpp backend. Confirm the formats offered for each model.1.5-bit to 8-bit integer quantization in GGUF formatNot covered in supplied facts. Confirm.
Built-in API keyYes via --api-key, but it covers only some /v1 routes; the docs advise a reverse proxy allowlistNo authentication on the local API; cloud inference uses a Bearer API keyYes via --api-key or an API key fileNot covered in supplied facts. Confirm.
TLSNot covered in supplied facts. Place endpoints behind a TLS-terminating reverse proxy and confirm.Exposed through proxies such as Nginx per the docs. Confirm TLS setup.Yes through PEM certificate and key filesNot covered in supplied facts. Confirm.
Metrics endpointPrometheus format at /metrics on the API serverNot covered in supplied facts. Confirm.Optional Prometheus-compatible endpoint enabled with --metricsNot covered in supplied facts. Confirm.
Kubernetes optionsNative manifests, Helm, production-stack, AIBrix, KServe, KubeRay, LeaderWorkerSet, llm-d and NVIDIA DynamoNot covered in supplied facts. Confirm container deployment patterns.Documented Docker usage; confirm Kubernetes patternsNot covered in supplied facts. Confirm.
Default bind addressNot covered in supplied facts. Confirm in the serve options for your version.127.0.0.1 port 11434, changed with OLLAMA_HOSTNot covered in supplied facts. Confirm in the server README.Not covered in supplied facts. Confirm.
Release cadenceMinor releases about every two weeks (v0.31.0 on October 5, 2026)Frequent releases (v0.40.5 on October 14, 2026)Continuous numbered builds (b11717 on October 18, 2026)Not covered in supplied facts. Confirm.

Each engine in brief, with its license and security defaults

vLLM is an Apache-2.0 licensed inference and serving engine for large language models. It has no paid edition, so the license cost is zero and the real spend comes from accelerator capacity, the surrounding infrastructure and the people who run it. Its server, started with vllm serve, exposes OpenAI-compatible routes, which makes it a common choice when existing applications already call the OpenAI API.

  • License: Apache-2.0, with no paid edition listed in the supplied material.
  • Authentication scope: The --api-key option protects endpoints under /v1, /v2, /inference and /cohere. It does not cover /invocations, /pause, /resume, /abort_requests, /metrics or /health, and the docs warn against relying on it alone.
  • Multi-node traffic: Communication between nodes is insecure by default, with no authorization or encryption. Nodes must sit on an isolated network.
  • Metrics: Prometheus-format metrics are served at /metrics on the API server, so they need to be blocked at the edge if they should not be public.
  • Recommended pattern: Put a reverse proxy in front that allowlists only the endpoints meant for users.

Before exposing a vLLM server, confirm which routes your proxy allows and test that /metrics, /health and /invocations are not reachable from untrusted networks.

What the license and paid tiers mean for a self-hosted deployment

The engine licenses are permissive, and none of the three engines has a paid edition you must buy to run it. vLLM is licensed under Apache-2.0, Ollama under MIT and llama.cpp under MIT, according to their repositories. vLLM has no paid edition, so its cost sits in accelerator capacity, the surrounding infrastructure and the people who operate it. Ollama's pricing page says that running models on your own hardware is always unlimited. Its paid plans (Pro at $20 per month or $200 per year, Max at $100 per month, Team at $500 per month and Enterprise at custom pricing) apply to cloud models, where Ollama's Enterprise plan adds model access controls, cost budgets per user or API key and a dedicated support channel. If your design routes some traffic to ollama.com/api, budget for those plans separately from your hardware. If it does not, confirm with Ollama that the plan terms match your usage before you rely on the free local path.

The engine license does not cover the model weights. Each open-weight model ships under its own terms, which can restrict commercial use, cap the number of users or require attribution, and those terms are separate from whether the engine is Apache-2.0 or MIT. Check the license file on each model you plan to serve, record the version you approved, and have legal review any model whose terms include acceptable-use rules or user thresholds. The same check applies when you change models later, because a new release may carry different terms. Keep this inventory alongside the engine version so an auditor or a procurement reviewer can trace what runs in production.

Label the non-OSI parts of your stack clearly, because the OSI status of an engine says nothing about its surroundings. The licenses above are the ones the repositories state, and this page does not extend OSI status to model weights, hardware drivers such as CUDA, or third-party hardware plugins that vLLM names as outside its main repository. Record each of those components with its own license and source, and confirm the terms with the vendor or the project when they matter to your decision. Where a component is proprietary, treat it as a separate line item in your architecture review.

How to choose and pilot an inference server

This sequence moves from sizing the user load to a tested engine, a hardened endpoint and a clear operating model, so the choice rests on your own hardware, prompts and traffic rather than on feature lists alone.

  1. Define user count and concurrency target

    The platform owner and the teams who will use the service agree on how many people will call the model, how many requests may run at the same time and what response time is acceptable under load. Write these numbers down as the pilot's pass criteria. The step is done when the target is signed off by the business owner and the security lead, since it sets how much batching and parallelism each engine must handle.

  2. Inventory hardware and accelerator vendor

    The infrastructure team lists every GPU, CPU and accelerator available, including the vendor, memory per device, node count and the network between nodes. This inventory decides which engines are even candidates, because vLLM supports NVIDIA CUDA, AMD ROCm and Intel XPU GPUs, while llama.cpp covers Apple silicon, x86 and several GPU backends. The step is done when the inventory is recorded and checked against each engine's installation documentation.

  3. Shortlist engines by hardware and API needs

    The engineering lead compares the inventory with the client applications that must be supported, including whether they expect the OpenAI API, Ollama's REST API or Anthropic Messages compatibility. Confirm each candidate's current endpoints, function calling and batching behavior in its own documentation rather than assuming them. The step is done when two or three engines remain, each with a written reason for staying on the list.

  4. Run a pilot with your own models and load

    The engineering team deploys the shortlisted engines on the target hardware with the models you plan to serve, then replays representative prompts and a load pattern close to the concurrency target. Record throughput, latency, memory use and failure behavior for each engine, and note the configuration that produced each result. The step is done when the measurements are compared against the pass criteria from the first step and the results are documented.

  5. Check the security posture before exposure

    The security team reviews how the chosen engine is reached: a reverse proxy that allowlists only user-facing endpoints, network isolation for any multi-node traffic, TLS on the external path and metrics endpoints kept off public networks. Use the vendor's security documentation to confirm which endpoints the API key protects and which it does not. The step is done when a test from outside the allowed network confirms that administrative and metrics routes are unreachable.

  6. Decide on the operating model

    The platform owner decides who runs upgrades, monitoring, capacity changes and incident response, and whether that work stays in-house, goes to a managed partner or follows a retainer. Ask any implementation partner for a fixed written scope that names deliverables and timelines before work starts. The step is done when a named owner, an upgrade cadence and a support route are written into the runbook for the chosen engine.

Questions buyers ask before choosing an inference server

How much does self-hosted LLM inference actually cost?

The engine license is not the cost. vLLM is Apache-2.0 licensed and has no paid edition, Ollama's local runtime is MIT licensed and its docs say running models on your own hardware is unlimited, and llama.cpp is MIT licensed. The recurring costs come from accelerator capacity or servers, the surrounding infrastructure (storage, networking, monitoring), and the people who operate the service. One-time costs include implementation, hardening and integration. Ask for a quote built from a written scope that lists the hardware class, the number of models and concurrent users, the deployment target, and whether you need support or upgrades after go-live. Ollama's paid cloud plans are a separate product and do not change the cost of running local models.

How long does a self-hosted inference pilot take?

No researched timeline is available for this, so treat any number you hear as unverified. The duration depends on your hardware lead time, whether the GPUs are already racked and drivers are validated, which model sizes you need to test, how many users join the pilot, and how your identity and network controls are configured. Ask for a fixed-scope plan that names the environment, the models to benchmark, the acceptance tests and the deliverables, and agree the timeline in writing before work starts. DeployOpen plans and implements most engagements on a fixed-scope basis, so the timeline follows from that agreed scope.

Can our team run an inference server without outside help?

Yes, if you already have the skills to match the engine you pick. Ollama is the lightest to start with on a single workstation or server. vLLM needs GPU driver and CUDA or ROCm know-how, and on Kubernetes it needs experience with the deployment options the vLLM docs list, such as Helm, KServe or KubeRay. Multi-node vLLM also needs network isolation between nodes, because the security docs say inter-node traffic is insecure by default. llama.cpp is a compiled tool with many hardware backends, so build and quantization choices matter. Check your skills in GPU operations, container orchestration, reverse proxies, TLS and monitoring before you commit, and confirm the on-call coverage you need.

When is it worth hiring a consultant for inference infrastructure?

A consultant adds the most value when the work crosses several disciplines at once. Common triggers include a multi-node vLLM deployment that must be isolated and tested, a shared service that serves many internal teams with authentication and per-user limits, a regulated environment where you must produce evidence for an assessment, and a need for hardening and documented backups and upgrades. DeployOpen plans, implements, hardens, backs up, upgrades and documents self-hosted platforms inside your own infrastructure, and prepares systems and evidence for compliance assessments. It does not issue audit reports or certifications; independent auditors and certification bodies do that. Ask any provider to name the deliverables, the environment they will work in and who keeps the system running after handover.

Which inference server is safest by default?

None of the three is safe to expose to users with its defaults, but the default risks differ. Ollama binds to 127.0.0.1 on port 11434 and its local API has no authentication, so the main risk is someone changing OLLAMA_HOST without adding a proxy with access control. vLLM's --api-key option does not cover every endpoint, including /metrics, /health and /invocations, so the vLLM docs recommend a reverse proxy that allowlists only user-facing routes. llama-server offers --api-key or an API key file, TLS through PEM files and an optional --metrics endpoint, but these are off until you set them. Whichever you choose, put it behind a proxy that authenticates users, restrict the metrics and health routes, and test the exposed surface from outside the network.

Does SGLang replace vLLM for high-throughput serving?

Not on the basis of anything in this comparison. SGLang is covered here as the other high-throughput GPU engine, but the researched facts for this page do not include its license, release history, batching features or hardware matrix, so this page does not rank it against vLLM. Confirm its current license, supported models, hardware backends, API surface and security documentation directly with the SGLang project before deciding. Run a benchmark on your own models and hardware with the same prompts and concurrency for both engines, because throughput depends heavily on the model, quantization and GPU type.

Should we pick the engine by hardware or by user count?

Start with the hardware you can actually run and the number of simultaneous users. Ollama defaults to one parallel request per model (OLLAMA_NUM_PARALLEL) and a queue of 512 requests, which suits small teams and developer machines. llama.cpp's llama-server offers continuous batching and parallel decoding for multiple users, and it runs on Apple silicon, x86 CPUs and several GPU backends. vLLM is built for GPU serving at scale and runs on NVIDIA CUDA, AMD ROCm and Intel XPU, plus several CPU types. If your hardware is mixed or CPU-heavy, test llama.cpp. If you have GPU capacity and many concurrent users, test vLLM. Benchmark both with your own model and traffic before deciding.

Risks to check before committing to one engine

Each of the three engines can be deployed safely or unsafely, and the difference usually lies in configuration, network placement and update discipline rather than in the engine itself. The cards below cover the defaults and risks that most often surprise teams moving from a prototype to a shared internal service.

Unauthenticated default APIs

Ollama's local API at localhost:11434 requires no authentication, and it binds to 127.0.0.1 by default. Moving it to a shared address with OLLAMA_HOST exposes that API directly, so anyone who can reach the port can run prompts. vLLM and llama-server both offer an API key option, but teams must enable it deliberately. Check which bind address, key and proxy layer each engine sits behind in your environment before any shared rollout.

Multi-node traffic without encryption

The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Tensor and pipeline parallel traffic crosses the network between GPU hosts, so this matters as soon as you scale beyond one machine. Plan an isolated network for inter-node traffic and confirm with the vLLM project how your chosen backend, Ray or its multiprocessing option, handles protection in your setup.

Metrics and health endpoints outside key coverage

The vLLM --api-key option covers /v1 routes but not /metrics, /health, /pause, /resume or /abort_requests. Those endpoints can expose operational detail or allow disruptive actions if reachable from users' networks. The vLLM documentation recommends a reverse proxy that allowlists only the endpoints meant for users. Confirm that your proxy blocks these paths for clients and that Prometheus scrapes them over a separate, internal path.

Model license terms apart from engine license

All three engines use permissive licenses: vLLM is Apache-2.0, Ollama is MIT and llama.cpp is MIT. The model weights you serve carry their own license, which can restrict commercial use, redistribution or the number of users. Running an Apache-2.0 engine does not make the model Apache-2.0. Before rollout, record each model's license, its acceptable use terms and any attribution duties, and have legal review them alongside the engine license.

Fast release cadence needs update discipline

vLLM publishes new minor releases about every two weeks, and Ollama and llama.cpp ship even more often, with llama.cpp publishing numbered builds continuously. Fast movement brings fixes but also behavior and flag changes that can affect production. Pin versions or container digests, read release notes before upgrading, and test each candidate build against your models and workloads in a staging environment before it reaches users.

Accelerator support gaps to confirm

vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU, and on other accelerators through third-party plugins outside the main repository. llama.cpp lists Metal, AVX, CUDA, HIP, Vulkan and SYCL backends. Support for your exact GPU model, driver version, quantization format and parallelism mode is not the same as support for the platform family. Request a written test matrix from the vendor or project, and benchmark your own model on your own hardware before committing.

How DeployOpen scopes and delivers inference server selection and implementation

DeployOpen helps teams choose and run a self-hosted inference server inside their own cloud accounts, data centers or air-gapped networks. Work starts with a written scope that fixes deliverables and timelines before any build begins, so the choice between vLLM, Ollama, llama.cpp or SGLang is tied to the hardware, users and controls you actually have.

How we scope

  • Inventory the GPU or CPU hardware, memory, network isolation and user counts the serving stack must support.
  • Agree which API surfaces applications will call, such as OpenAI-compatible chat, completions or embeddings routes.
  • Record licensing, support expectations and the operating team in the written scope before deployment work starts.

How we build

  • Deploy the chosen server in your environment, with the authentication, reverse proxy and endpoint allowlisting it needs.
  • Configure parallelism, concurrency limits, queueing and model storage locations to match the agreed workload.
  • Test the endpoints, metrics and failure behavior against the scope, and document the results for your review.

How we hand over

  • Deliver runbooks covering startup, model updates, metrics collection and access control for the serving layer.
  • Document the configuration, network placement and version pins so your team can reproduce the deployment.
  • Walk your operators through upgrade steps and the checks to run before moving to a newer release line.

How we support

  • Offer ongoing support, upgrades and continuous testing as a retainer after handover, scoped in writing.
  • Re-test authentication and exposed endpoints when a server release changes its API or security behavior.
  • Ask any provider quote to itemize hardware, implementation, support and retainer effort separately so costs stay visible.

Sources and further reading

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com