Self-hosted LLM inference on CPU-only and edge servers

Drift is controlled by treating every node as a copy of one reviewed definition, not as a hand-tuned server. The engine flags, model files, environment variables such as OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE, the bind address and the TLS settings should live in version-controlled configuration and be applied by automation. The drivers are the number of sites, how often staff log in locally to fix something, and whether any node can be changed outside the pipeline. Standardize one baseline image or playbook per hardware class, keep a short list of approved site-specific overrides, and run a periodic check that compares each node's running configuration with the baseline. Ask any partner how they detect a node that was changed by hand and how that change gets back into the baseline.

Patching cadence is set by the release rhythm of the engine and by how much testing each release needs. llama.cpp publishes numbered builds continuously, with build b11717 dated October 18, 2026 as one example, and Ollama shipped v0.40.5 on October 14, 2026 with a pre-release v0.40.6 two days later. Frequent releases mean you need a defined policy: which channel is allowed in production, who approves an upgrade, how a build is tested against your models and prompts, and how a failed node is rolled back. Keep the binary and its checksum in an internal artifact store so every node runs the same verified build. Confirm with your partner how many releases per quarter they expect to test and what happens when a security fix arrives between scheduled windows.

Pin each model by file identity, not by name. A GGUF file or an Ollama model tag can change under the same label, so record the exact file hash, the quantization level and the context length in a model manifest that every node checks at startup. Use one approved model list for the fleet, and let sites differ only where a documented reason exists, such as local language needs. When you upgrade a model, roll it out to a canary node first, run your regression prompts, and only then promote it. Ask suppliers how they tell you a model has changed and whether they publish checksums for the artifacts you download.

Collect metrics centrally from every node through a Prometheus-compatible scraper or an equivalent agent. llama-server can expose a Prometheus-compatible metrics endpoint when started with --metrics, so you can enable it in the baseline configuration and scrape it over a management network. Ollama does not document a comparable built-in metrics endpoint in the sources reviewed here, so confirm how you will gather request counts, latency, queue depth and loaded models, perhaps through a proxy or host-level exporters. Standardize the metric names, the labels (site, node, model, build) and the alert thresholds before the first site goes live. Plan for a central time series store and for what happens to metrics when a site's WAN link is down.

A node that returns after a long outage can be running an old engine, an old model, an expired certificate or a stale key, so it should rejoin the fleet through the same process as a new node. Design for this case from the start: keep the node stateless where possible, store models and configuration on the device in a known state, and have it refuse to serve until its version and checksums match the current manifest. Decide in advance whether a node that was offline longer than your tolerance is reimaged instead of patched. Also consider the data it may hold, such as cached prompts or logs, and how those are collected or wiped. Ask your partner what their reconnection procedure looks like and how they test it.

Rotation at fleet scale depends on a secrets store and automation, because manual copying of keys to each node does not scale and leaves stale credentials behind. llama-server supports API key authentication through --api-key or an API key file, and TLS through PEM certificate and key files, so the key file path and certificate renewal can be handled by your configuration management. Ollama's local API at port 11434 does not require authentication by default and binds to 127.0.0.1, so any network exposure must be added in front of it with a proxy that enforces authentication. Define a rotation interval, a process for revoking a lost node's credentials, and an alert for certificates that are close to expiry. Confirm how your partner will store and rotate these secrets and who holds the keys.

Ownership after rollout is the decision that determines whether the fleet stays healthy, and it should be written down before the first site goes live. Your team, a partner or a mix can own the baseline, the patch pipeline, monitoring, on-call response and model promotion. Costs recur in each of these areas: engineering time for upgrades and testing, hardware replacement, and on-call coverage across time zones. Ask a partner for a handover plan that names runbooks, the process for adding a site, and the support hours they commit to in writing. Ongoing support and upgrades can be arranged as a retainer, and the scope of that retainer should be agreed in writing before work begins.

Yes, a small team can run a fleet if the work is standardized, but the effort grows with the number of sites, the number of model and engine versions in use, and how many people can make changes. Ollama states that running models on your own hardware is always unlimited, and its engine is MIT licensed, so software licensing does not block self-operation. The real questions are staffing, testing capacity and the skills needed for configuration management, metrics and secrets handling. Before deciding, write down the baseline, the patch policy, the on-call model and the handover process, and estimate the hours each one requires from your own staff. If those hours are not available, a fixed-scope implementation followed by a support retainer is one option to compare.

The biggest drivers are the number of sites and machines, how many distinct hardware profiles exist, how much benchmarking is needed to choose a model and quantization per profile, and how much hardening and monitoring each node needs. Cost splits into one-time work (hardware survey, design, benchmarking, hardening, documentation, handover) and recurring work (updates, model changes, monitoring, support). Ask for the quote to list each site, each hardware profile, the benchmark matrix, the update cadence and the support hours separately, so you can see which items scale with fleet size. No fixed figure applies across environments, so the written scope is the basis for any number.

Duration depends mainly on the number of sites, whether hardware is already on hand, how many hardware profiles need separate testing, and how much network access and approval each location requires. A pilot at one site usually takes far less time than a fleet rollout across branches, ships or plants because each new profile repeats survey and benchmarking. DeployOpen does not publish timelines; confirm the phases, dependencies and acceptance tests in the written scope before work starts.

Most of the dependencies sit with your team rather than the implementation partner. Plan to provide:

• Physical or remote access to each site, with change windows agreed in advance
• The hardware inventory or purchase decisions, including CPU generation and installed memory
• Network rules for outbound model downloads, internal clients and any metrics collection
• Named decision owners for security, model approval and acceptance testing

Confirm in the quote which of these the partner expects you to supply and which are included in the fixed scope.

Yes, a team with Linux administration, container and service management experience, and access to a test node can run llama.cpp or Ollama on a single machine. The harder part is repeating the work across many nodes: keeping model files, engine builds and configuration consistent, applying authentication and TLS, and checking performance on each hardware profile. Assess whether you have people to own that fleet process over time. A partner can build the baseline and runbooks, then hand over operations, or your team can own it from the start.

Not always, but you should plan for ongoing work. llama.cpp publishes numbered builds continuously, and Ollama ships frequent releases, so each engine upgrade and model change needs testing on every hardware profile. Ollama's own documentation notes that it binds to 127.0.0.1 port 11434 by default and that the local API does not require authentication, so exposing nodes to the network requires a proxy or gateway with access control. Decide whether your team will handle that cycle internally or whether a retainer for upgrades, testing and support better fits your release calendar. Ask the partner to state the support hours, response process and upgrade cadence in writing.

Security on small nodes depends on four controls: restricting who can reach the engine, enabling authentication, encrypting traffic and limiting what each node can reach. The llama-server supports API key authentication through an API key file or the --api-key option, TLS through PEM certificate and key files, and an optional Prometheus-compatible metrics endpoint enabled with --metrics. Ask the partner how they handle key distribution and rotation across many nodes, how they confirm metrics are not exposed beyond the monitoring network, and how they document these settings for audit evidence.

A useful quote lists the scope in writing: the sites and machines in scope, the hardware survey method, the models and quantization levels to benchmark, the response-time and throughput criteria for acceptance, the hardening baseline, the monitoring setup, the documentation and handover items, and any exclusions. It should also separate one-time deliverables from recurring support and state what each assumes about your environment. If a line item is missing, ask whether it is excluded or included, because gaps usually appear later as change requests.