Ollama vs llama.cpp: which to choose

Ollama or llama.cpp: when each fits and what the decision turns on

Choose Ollama when the goal is a packaged runner with simple model pulls and a local REST API on port 11434. Choose llama.cpp when your team needs direct control over the build, hardware backends, quantization and server hardening. Both are MIT licensed, so the decision rests on operations and controls. Start with who will run the service. Ollama suits a small platform team or one engineer who wants models running quickly, with the model store movable through OLLAMA_MODELS. llama.cpp suits an engineer who is comfortable compiling for a specific backend, such as Metal, CUDA, HIP, Vulkan or SYCL, and who will tune GGUF quantization to fit the memory available.

Next, check the hardware mix and the security controls your team must own. Ollama binds to 127.0.0.1 on port 11434 by default, and its local API does not require authentication, so exposing it means placing a proxy such as Nginx in front and deciding who may call it. llama-server supports API keys through --api-key or a key file, TLS through PEM certificate and key files, and an optional Prometheus-compatible metrics endpoint enabled with --metrics. If your security standard requires your team to configure and evidence those controls, llama.cpp keeps them in your hands. Confirm with your security team whether a reverse proxy with its own authentication and TLS already meets the requirement, because that changes the effort on the Ollama side. The llama-server README lists the flags to check against your standard.

Concurrency is the sizing fact that most often settles the choice. Ollama defaults to one parallel request per model through OLLAMA_NUM_PARALLEL, with a queue of up to 512 requests through OLLAMA_MAX_QUEUE. If several users will hit the same model at once, raising the parallel setting increases memory use, so test it on your own hardware before go-live. llama-server lists parallel decoding with multi-user support and continuous batching among its features. Before deciding, measure your expected peak concurrency and latency targets against both runtimes. Then ask who will patch and upgrade the runtime, since Ollama ships frequent releases and llama.cpp publishes numbered builds continuously.

Which platform fits which scenario

This table maps common deployment scenarios to the platform that fits each one better, with the condition that drives the choice. Read each row for the decisive requirement, then check the condition against your own hardware, users and security controls.

Scenario	Better fit	Condition that drives the choice
Developer laptop	Ollama	Ollama installs as a packaged runner on macOS, Windows, Linux or Docker and manages model downloads, so one engineer can start quickly without building anything.
Shared internal server for a small team	Ollama, with a reverse proxy in front	Ollama binds to 127.0.0.1 by default and its local API has no authentication, so the team must expose it through a proxy it controls. Confirm how many requests run at once, since OLLAMA_NUM_PARALLEL defaults to 1 per model.
Shared server with several concurrent users	llama.cpp (llama-server)	llama-server supports parallel decoding with multi-user support and continuous batching, which suits concurrent use better than Ollama's default single parallel request per model. Benchmark both on your own hardware before deciding.
Mixed CPU and GPU host	llama.cpp	llama.cpp lists Apple silicon Metal, x86 AVX variants, CUDA, HIP, Vulkan and SYCL backends and 1.5-bit to 8-bit quantization, giving more direct control over how a model is split and sized on mixed hardware.
Air-gapped build pipeline	llama.cpp	A single engine and GGUF model files can be copied into a closed network and run without a package manager. Confirm the Ollama cloud features are disabled with OLLAMA_NO_CLOUD=1 if you choose Ollama.
Team must enforce TLS and API keys at the server	llama.cpp (llama-server)	llama-server supports API keys through --api-key or a key file and TLS through PEM certificate and key files. Ollama's local API has no authentication, so you would need to add these controls in a proxy layer.
Team that needs metrics for monitoring	llama.cpp (llama-server)	llama-server offers an optional Prometheus-compatible endpoint enabled with --metrics. Confirm what your monitoring stack requires from Ollama before relying on it.
Model management with minimal operator work	Ollama	Ollama handles model storage paths, pulls and an OpenAI-compatible REST API on port 11434, which reduces the hands-on setup compared with building and configuring llama-server flags yourself.

How Ollama and llama.cpp are built and what that means for operations

Ollama wraps a llama.cpp backend in its own service, model store and API, while llama-server is the HTTP server inside llama.cpp with its own routes and security flags. The cards below cover what that layering means for upgrades, moving parts, network exposure, model files, concurrency and container packaging.

Two layers, two release trains

Ollama is written in Go and uses a llama.cpp backend, so a fix or new model format in llama.cpp reaches you only after Ollama packages it. llama-server ships as numbered llama.cpp builds, with build b11717 published on October 18, 2026. Ask which llama.cpp build your Ollama version bundles before you promise a model or hardware feature to users, and pin both versions in your change records.

Moving parts and what you operate

Ollama is one service that owns model download, storage, the REST API and the SDK surface, so there is one process to monitor and patch. llama-server is a single binary that serves models you place yourself, which leaves model management, routing and any process supervision to your own tooling. Count the components you must own for each option, including the unit file or container definition, log collection and the model sync job.

Bind address and proxying

Ollama binds to 127.0.0.1 on port 11434 by default, and OLLAMA_HOST changes that bind address. The local API has no authentication, so exposing it means placing a reverse proxy such as Nginx in front and handling access control there. llama-server offers --api-key or an API key file, plus TLS through PEM certificate and key files, so authentication and encryption can sit in the server itself. Confirm which controls your security team requires at the edge.

Model storage paths

Ollama keeps models in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and the .ollama\models folder on Windows. Setting OLLAMA_MODELS moves that store, which matters for disk sizing, backup scope and what your retention policy covers. llama-server reads GGUF files from paths you choose, so model files are ordinary files under your own control. Decide where weights live, who may write to that location and how checksums are recorded.

Concurrency settings

Ollama defaults to one parallel request per model through OLLAMA_NUM_PARALLEL, keeps up to three loaded models per GPU or three on CPU through OLLAMA_MAX_LOADED_MODELS, and queues up to 512 requests through OLLAMA_MAX_QUEUE. llama-server provides parallel decoding and continuous batching for multiple users, so its tuning focuses on slots and context size. Test both under your expected peak load, because the default queue behavior shapes what users see during bursts.

Docker packaging

Ollama is available for Docker alongside macOS, Windows and Linux installs, which gives a consistent deployment unit for a single host. The llama.cpp README also documents running with Docker, so both options can run as containers. Check how each image handles GPU passthrough, model volumes and version pinning, since those settings decide whether an upgrade is a tag change or a rebuild.

Setup, formats, backends and security compared

This table compares licensing, model formats, quantization, hardware backends, local API authentication, TLS, metrics and concurrency controls for Ollama and llama-server. Read each row as a question to answer in your own environment, because the default security posture differs sharply between the two.

Capability	Ollama	llama.cpp (llama-server)
License	MIT licensed. Local use is covered by the license, and cloud features are a separate paid service.	MIT licensed according to the repository metadata. Confirm the license text in the version you deploy.
Model format	Models are packaged and pulled through Ollama's own model management, stored under the OLLAMA_MODELS directory when you move them.	Runs GGUF models, for example downloaded from Hugging Face. You manage model files yourself.
Quantization	Quantized variants are available through the model library. Confirm which quantization levels each model offers before standardizing.	Supports 1.5-bit to 8-bit integer quantization for faster inference and lower memory use. Choose the quant per GGUF file.
Hardware backends	Runs on macOS, Windows, Linux and Docker, with a llama.cpp backend underneath. Confirm GPU support for your specific cards and drivers.	Apple silicon via Metal, x86 AVX, AVX2, AVX512 and AMX, CUDA, HIP for AMD, plus Vulkan and SYCL. Build flags select the backend.
Authentication on the local API	The local API at localhost:11434 does not require authentication by default. The server binds to 127.0.0.1 unless OLLAMA_HOST changes it, so exposure needs a reverse proxy with its own controls.	API key authentication through --api-key or an API key file. Enable it before any non-local binding.
TLS	The documentation describes exposing Ollama through a proxy such as Nginx. Confirm where TLS terminates and who manages certificates.	TLS through PEM certificate and key files set on the server. Confirm certificate rotation in your own process.
Metrics	No built-in metrics endpoint is described in the supplied facts. Confirm monitoring options with your team before relying on them.	Optional Prometheus-compatible metrics endpoint enabled with --metrics. Decide whether to expose it only on an internal network.
Concurrency controls	OLLAMA_NUM_PARALLEL defaults to 1 request per model, with OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE controlling loads and queue depth.	Parallel decoding with multi-user support and continuous batching. Confirm slot and context sizing against your expected load.

What each platform costs in license terms

Ollama is released under the MIT license, and its pricing page states that running models on your own hardware is always unlimited. The license cost of the local runtime is therefore zero, and the paid products are optional cloud services that you choose to buy. Those plans are Pro at $20 per month ($200 per year), Max at $100 per month, Team at $500 per month and Enterprise at custom pricing, according to Ollama's pricing page. Enterprise adds model access controls, cost budgets per user or API key, a dedicated support channel and custom security options on top of the Team features.

Local use: Running models on your own hardware carries no license fee under the stated terms, so the recurring cost is the hardware, power and the staff time to operate it.
Cloud plans: Pro, Max, Team and Enterprise pay for hosted model inference, and they are separate from the local runtime.
Enterprise controls: Per-user and per-key budgets and model access controls appear only at the Enterprise tier, so teams that need them should price that tier specifically.
Cloud switch: Setting OLLAMA_NO_CLOUD=1 or the equivalent server configuration turns cloud features off, which keeps an installation on local models.
Custom pricing: Enterprise has no published price, so the cost comes from a written quote.

Confirm current plan prices, included usage and terms directly with Ollama before budgeting, because pricing pages change.

How the request path changes when you add access control

Ollama does not enforce authentication on its local API. By default it binds to 127.0.0.1 on port 11434, and the local API at localhost:11434 requires no key. Any client that can reach that socket can list and run models. If a team needs users or services on other machines to reach the runtime, the bind address is changed with the OLLAMA_HOST environment variable, and the Ollama FAQ describes placing a proxy such as Nginx in front of it. In that design the request path is client, then Nginx, then Ollama, and the proxy is the only place where a credential check can happen. Ollama itself will still accept anything the proxy forwards, so the team must write the proxy rules, the key or identity check, and the TLS termination, then test that a request without credentials is refused before exposing the port.

llama-server carries the access controls itself. Started with --api-key or an API key file, it requires the key on API requests. Started with PEM certificate and key files, it serves TLS directly, so a reverse proxy is optional rather than required. Its --metrics flag enables a Prometheus-compatible endpoint, which the team should also place behind network controls, because a metrics route is a separate path that an API key may not cover in the way the team expects. Confirm in the llama-server README, and in a test against your build, exactly which routes the key protects before you rely on it. A reverse proxy can still sit in front for rate limiting, centralized logging or per-user identity, but those are features the team adds and operates.

The decision is about who owns each control. With Ollama on a private network, the team owns the entire access layer: the proxy configuration, the credential scheme, certificate handling and any user-level logging, and the runtime offers no per-user accounts to fall back on. With llama-server, the key and TLS are part of the runtime, but user management, key rotation and audit still sit outside it. Write down, for each option, the component that rejects an unauthenticated call, the component that encrypts traffic, and the component that records who called. If any of those answers is 'none,' the gap is the team's work item, and it should be scoped before the service is exposed to more than one client.

What drives the running cost of each option

Running either option costs more than the software license, which is free in both cases. The cost comes from hardware sized for the models you choose, the people who install, patch and secure the runtime, and the controls you add around it. Some costs are paid once and others recur every time you upgrade a model or a release.

Hardware and memory sizing

Hardware is usually the largest running cost for both options, because model size and quantization decide how much RAM or VRAM you need and how many requests a machine can serve. Ollama's default of one parallel request per model (set by OLLAMA_NUM_PARALLEL) means you may buy more capacity to match concurrency than llama-server would need with its continuous batching. Ask your team to benchmark both on the same models and hardware before you size anything. This cost is largely one-time for the server and recurs as usage grows.

Build and patch effort for llama.cpp

Running llama.cpp directly means your team compiles the engine for your backends (Metal, CUDA, HIP, Vulkan or SYCL), chooses a build, and rebuilds whenever you move to a newer numbered build. Ollama packages the runtime for macOS, Windows, Linux and Docker, so the build work shifts to its release process. The staff time to build, test and roll forward llama.cpp is one-time at setup and recurs with every upgrade cycle. Plan that time explicitly in your estimate.

Proxy, authentication and TLS

llama-server offers API keys through --api-key or an API key file and TLS through PEM certificate and key files, while the local Ollama API has no authentication by default. Either way, exposing the service to users usually means a reverse proxy, certificate management and access rules, which you build and maintain yourselves. Count the one-time setup and the recurring work of rotating keys and certificates. Ask whether the proxy, certificate tooling and key rotation are in scope of any services quote.

Monitoring and metrics collection

llama-server can expose a Prometheus-compatible metrics endpoint when started with --metrics, which means you run and pay for a Prometheus or equivalent stack, dashboards and alert rules. Ollama's monitoring depends on what you add around the server, so budget for the same collection work in either case. Monitoring is a one-time build followed by recurring work on alert tuning, storage growth and on-call response. Confirm who owns each alert before go-live.

Optional Ollama cloud plans

Running models on your own hardware through Ollama is listed as unlimited, but cloud models are billed separately. Ollama's pricing page lists Pro at $20 per month ($200 per year), Max at $100 per month, Team at $500 per month and Enterprise at custom pricing. These are recurring subscription costs that apply only if you send requests to Ollama's hosted models, and you can disable cloud features with OLLAMA_NO_CLOUD=1. Ask for the exact plan terms and usage limits in writing before budgeting.

Quote items to request

A fixed-scope quote should separate one-time work (installation, configuration, hardening, documentation) from recurring work (upgrades, model updates, monitoring, support). Ask for the hardware assumptions, the number of models and concurrent users in scope, and who performs each patch cycle. Request the assumptions behind any retainer, including the release frequency it covers. Compare both options on the same written scope so the differences reflect the tooling and not the scope.

Moving from Ollama to llama.cpp, or back, without losing work

This sequence takes a team from an Ollama install to a llama-server deployment (or the reverse) with the model weights, client prompts and access rules carried across and the service layer rebuilt around them.

Inventory models, formats and client calls
The platform owner lists every model pulled into Ollama, its quantization and size, and every application that calls port 11434. Ollama stores models under its own models directory (OLLAMA_MODELS can relocate it), so record the directory and the model tags in use. Also record which clients use the native Ollama API and which use its OpenAI-compatible routes, because that decides the client work later. Done when the team has a signed-off table of model, source, format, consumer and owner.
Source or convert GGUF files for each model
Engineering confirms that each model is available as a GGUF file that llama.cpp can load. llama.cpp runs models in GGUF format, so a model that was only available to Ollama in its internal form needs a GGUF source, such as a Hugging Face repository, or a conversion step using the llama.cpp tooling for the model architecture. Verify each file by loading it with llama-server on a test host and sending one prompt. Confirm licence terms for each model file before copying it. Done when every inventoried model answers a test prompt from llama-server.
Map Modelfile settings to llama-server flags
Engineering translates the settings in each Ollama Modelfile, such as the context length, sampling parameters, system prompt and template, into llama-server options. Ollama's concurrency settings (OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_MAX_QUEUE) do not map one for one, so size parallel slots and queue behavior by testing under realistic load. Record each mapping in a version-controlled file. Done when a reviewer can trace every Modelfile setting to a flag or a documented reason it was dropped.
Point clients at the OpenAI-compatible routes
Application owners change each client from the Ollama REST API on port 11434 to the chat completions, responses or embeddings routes that llama-server exposes. Clients that use the Ollama Python or JavaScript SDKs need code changes, while clients that already speak the OpenAI format usually need only a base URL and key change. Test each client against a staging llama-server and compare output for the same prompts. Done when each client completes its standard requests against the new endpoint and the owners sign off.
Add API keys and TLS before exposure
Because the local Ollama API has no authentication, the security owner must add access controls as part of the move. llama-server supports API key authentication through --api-key or an API key file, and TLS through PEM certificate and key files. Store keys in the team's secrets system, not in unit files, and rotate them on a schedule the security team sets. Confirm with a request without a key that the server rejects it, and with a valid key that it succeeds over HTTPS. Done when both checks pass and the results are filed as evidence.
Rebuild service units, proxy rules and monitoring
Model weights and client prompts carry over, but the operating layer does not. Platform engineers write new service units or container definitions for llama-server, recreate reverse proxy rules (the Ollama FAQ describes exposing the service through proxies such as Nginx), and enable the metrics endpoint with --metrics for Prometheus-compatible scraping. Dashboards and alerts built for Ollama must be rewritten against the new metric names. Done when the service restarts cleanly after a host reboot and the metrics appear in the monitoring system.
Cut over and keep the rollback path
The team switches traffic in a planned window, with the Ollama service kept installed but stopped, so a rollback means restarting it and reverting the client configuration. Watch error rates, latency and queue depth for the first working period the owners agree on. Going back to Ollama follows the same sequence in reverse, and Ollama's cloud features can be disabled with OLLAMA_NO_CLOUD=1 if the team wants a local-only install. Done when the new endpoint carries all production traffic, the rollback has been rehearsed, and the decommissioning decision is recorded.

What implementing each involves and where projects run long

Implementing Ollama is mostly configuration. The work centers on the bind address, which defaults to 127.0.0.1 on port 11434 and changes through the OLLAMA_HOST variable, the model directory set by OLLAMA_MODELS, and the concurrency controls OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE. Because the local API has no built-in authentication, any exposure beyond localhost means putting a reverse proxy such as Nginx in front of it, which the Ollama FAQ describes, and that proxy becomes the place where access control, TLS and logging are added. Confirm in your own environment whether OLLAMA_NO_CLOUD=1 is set on every host, since that setting determines whether cloud features can be reached at all.

Implementing llama.cpp involves more decisions before the first request. The team chooses and builds the backend for its hardware (Metal, CUDA, HIP, Vulkan, SYCL or x86 CPU instruction sets), picks a quantization level from the 1.5-bit to 8-bit range that the README describes, and pins a GGUF model file. Authentication, TLS and metrics are also the team's job: --api-key or an API key file, PEM certificate and key files, and --metrics all need to be set, and the team must place its own ingress, rotation process and scrape configuration around them. Treat the llama-server flags as the interface to review, and confirm them against the release you deploy, since builds such as b11717 ship continuously. Projects usually run long in three places. The first is GPU driver alignment, where the CUDA, ROCm or Vulkan driver version on the host must match the backend build, and mismatches tend to appear only after a host update. The second is capacity testing under concurrent load: Ollama's default of one parallel request per model and its queue size set the behavior users see, and llama-server's parallel slots and continuous batching need the same measurement on your own hardware before you set a user target. The third is operating the chosen hardware and quantization over time. Ask for a written list covering driver pins, build pins, auth and TLS design, metrics ownership and a load test with your own models before committing to either path.

Common questions about running Ollama or llama.cpp in-house

Which is easier to run for a small team, Ollama or llama.cpp?

For a small team that wants a model running quickly on a laptop or one server, Ollama is usually the easier starting point. It installs on macOS, Windows, Linux and Docker, manages models through one command line tool and serves a REST API on port 11434 with OpenAI compatibility. llama.cpp gives you more control, but your team then owns the build, the hardware backend, the model file choices and the service setup. The gap narrows if your engineers already build C++ projects and manage GPU drivers. Ask who will patch the server, choose quantization levels and debug hardware problems during the first year, because that person's time is the largest recurring cost. Both projects are MIT licensed, so the license does not separate them.

Does Ollama need an API key when running on localhost?

No. The local Ollama API at localhost:11434 does not require authentication, and Ollama binds to 127.0.0.1 by default, so only the local machine can reach it. Authentication applies to direct cloud inference at ollama.com/api and ollama.com/v1, which requires an API key sent as a Bearer token. The risk appears when you change the bind address with OLLAMA_HOST or publish the port through a reverse proxy such as Nginx. At that point, anyone who can reach the port can use the API unless the proxy enforces access control. Set OLLAMA_NO_CLOUD=1 if the installation must stay local, and confirm in your own environment that the proxy checks credentials before the service is exposed on any network.

Does llama-server support TLS and Prometheus metrics?

Yes, llama-server has built-in options for API keys, TLS and metrics. According to its README, --api-key or an API key file enables API key authentication, PEM certificate and key files enable TLS, and --metrics turns on an optional Prometheus-compatible metrics endpoint. Confirm the exact TLS flag names against the build you deploy, because options change between releases. Two points matter for planning. An API key protects the endpoint but does not replace network segmentation, logging or certificate rotation. Metrics reveal operational data about load and model use, so keep the metrics endpoint on your monitoring network only and scrape it with the same controls you apply to other internal services.

Can our team switch models between Ollama and llama.cpp?

Yes, but plan for a verification step. llama.cpp runs models in GGUF format, for example GGUF files downloaded from Hugging Face, while Ollama keeps models in its own store. By default that store sits in a per-operating-system directory, and the OLLAMA_MODELS variable moves it. Before you point users at a model in either runner, confirm that the same model file, quantization level and context length produce acceptable answers in both. Keep an inventory with file checksums, quantization and context settings, since these change output quality and memory use. Keeping model names, endpoints and base URLs in one configuration layer lets your applications switch runners with a configuration change.

What does running llama.cpp need from our engineers?

Running llama.cpp takes more ongoing engineering time than Ollama, mainly for tuning and maintenance. With Ollama, the team manages models, OLLAMA_NUM_PARALLEL (default 1 parallel request per model), the queue set by OLLAMA_MAX_QUEUE (default 512 queued requests) and server upgrades. With llama.cpp, the team also selects a build for its hardware, which the README covers across Apple silicon via Metal, NVIDIA via CUDA, AMD via HIP, plus Vulkan, SYCL and CPU instruction sets. Engineers choose the quantization level, tune parallel decoding and batching, write service definitions and track numbered builds such as b11717 from the release page. Name an owner for upgrades, GPU driver changes, authentication settings and monitoring before you go live.

When should we bring in outside help for a self-hosted model server?

Outside help makes sense when your team lacks someone to own hardening, authentication, monitoring and upgrades, or when a model server must meet a documented production standard within a set schedule. Cost comes from hardware, engineering time and any support you buy. The software is free to run for both projects, and Ollama's pricing page states that running models on your own hardware is always unlimited, while its paid plans, from Pro at $20 per month to Enterprise at custom pricing, apply to cloud models. DeployOpen plans, implements, hardens and supports self-hosted open-source platforms inside your own infrastructure. Most engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer. DeployOpen does not publish a price list, so ask for a quote based on a written scope that covers your hardware, security requirements and handover.

Risks to check before choosing either platform

Both Ollama and llama.cpp are easy to start and easy to expose by accident. The four cards below cover the checks that most often change the decision: network exposure, concurrency under load, hardware backends and the upkeep of each release line.

Unauthenticated default exposure

The local Ollama API at localhost:11434 requires no authentication, and Ollama binds to 127.0.0.1 by default. Moving it with OLLAMA_HOST or placing it behind a proxy such as Nginx changes who can reach it, and the API has no built-in key layer to fall back on. llama-server, by contrast, offers API key authentication through --api-key or a key file and TLS through PEM files, but both are off until you configure them. Check with a port scan from another host that neither service answers on a network interface you did not intend to open, and confirm which controls sit in front of each one.

Concurrency limits under real load

Ollama defaults to one parallel request per model, with a queue of up to 512 requests, so several users on the same model will wait on each other unless you tune OLLAMA_NUM_PARALLEL. llama-server documents parallel decoding with multi-user support and continuous batching, which changes how many slots and how much context memory a team needs. Run a load test with your expected number of concurrent users, your model sizes and your prompt lengths on the target hardware, and record queue wait time and latency rather than relying on single-user speed.

Hardware backend mismatch

Ollama uses a llama.cpp backend but manages its own builds and hardware detection, so the GPU path you test with Ollama may not match a llama.cpp build compiled for the same card. llama.cpp lists Metal, AVX and AVX2 and AVX512, AMX, CUDA, HIP, Vulkan and SYCL backends, and each one must be enabled at build time and supported by your driver stack. Confirm which backend is actually loaded on each machine type in your fleet, measure tokens per second on it, and check that the quantization you plan to run is supported there before you standardize.

Upkeep of fast-moving release lines

llama.cpp publishes numbered builds continuously, so the build your team pins today may be far behind next month, and each update can change flags, endpoints or model compatibility. Ollama ships frequent tagged releases, including pre-releases, and its CLI and API surface have changed across versions. Decide who owns the upgrade cycle, whether you pin a build or tag, and how you will test a new release against your models and clients before it reaches users. Ask for a written upgrade and rollback procedure if DeployOpen will run it for you.

How DeployOpen helps teams choose and run private model inference

DeployOpen starts with the question of whether a packaged runner or a lower-level inference server fits your users, hardware and security controls. We then plan, implement and document the chosen stack inside your own cloud accounts, data centers or air-gapped networks, so the decision is tested against your environment rather than a generic setup.

How we scope

Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
The written scope names the hardware, operating systems, client tools and user groups that the Ollama or llama.cpp deployment must serve.
We record which API consumers need OpenAI-compatible routes and which need authentication, TLS or metrics before any build begins.

How we build

We install and configure the chosen runtime, set the bind address and place model storage on the directories your team approves.
For llama-server, we configure API key files, PEM certificate and key files, and the metrics endpoint in line with your security policy.
We test concurrency, model loading and failure behavior on your hardware so the results reflect your workload, not a default configuration.

How we hand over

You receive written documentation of the configuration, model inventory, network exposure and the reverse proxy design used.
We walk your platform and security staff through upgrade steps, model updates and how to roll back a failed change.
Handover includes a record of the evidence your compliance assessors may ask for, which independent auditors and certification bodies then evaluate.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainers can cover new llama.cpp builds and Ollama releases, with regression checks run against your own models before rollout.
Ask any provider to confirm in writing how upgrades, security patches and out-of-hours response are handled before you sign.