Self-hosted LLM inference requirements checklist

A requirements checklist for an internal model service covering models, latency, GPUs, API compatibility, authentication, metrics, multi-node networking and upgrades, with how to verify each item.

Book a callTalk to us
On this page

How to use this self-hosted LLM inference requirements checklist

This checklist turns the decision to run open-weight models internally into written, verifiable requirements. Your team has already agreed on the goal, so the open question is what the service must do before anyone picks an engine. Use the sections below to agree on model and context needs, concurrency and latency targets, accelerator support, OpenAI API compatibility, authentication, metrics, multi-node networking and upgrade cadence, and to record who owns each answer.

Treat every requirement as something to prove, not something a vendor has promised. Each item should name the test, configuration or document that shows it is met, such as a load test against your target concurrency, a captured request showing that unauthenticated calls are refused, or a scrape of the metrics endpoint in your monitoring stack. Where a requirement cannot be tested yet, record it as an open question with an owner and a date for the answer. A requirement that only exists in a slide deck will not survive the first audit or the first incident.

The engine comes last. vLLM, Ollama and llama.cpp differ in how they handle parallel requests, multi-node serving, authentication defaults and hardware support, so the same requirements may point to different engines for different workloads. Read the requirements against the engine documentation, not the other way round, and expect some items to be met by a reverse proxy or gateway rather than by the engine itself. Once the requirements are signed off, the engine decision becomes a comparison against a fixed list instead of a debate about preferences.

Functional requirements to agree before building

This table sets out the functional requirements a team should agree before choosing an inference engine, with the reason each matters and how to test it. Read each row as a question to answer in writing, with a load test or hardware check as the evidence. The main takeaway is that Ollama's default concurrency limits and vLLM's partial endpoint protection both need to be tested against your own targets before the design is fixed.

RequirementWhy it matters and what to agreeHow to verify
Models and context lengthsName the exact open-weight models, their quantization, and the longest context each use case needs. Memory use grows with context, so a model that loads may still fail under real prompts.Load each candidate model with your longest expected prompt on the target GPU or CPU and record memory use and whether the response completes.
Concurrency and latency targetsAgree the number of simultaneous users, the peak requests per second and acceptable time to first token. Ollama defaults to OLLAMA_NUM_PARALLEL=1 per model, OLLAMA_MAX_LOADED_MODELS of 3 per GPU and OLLAMA_MAX_QUEUE of 512, so defaults rarely match a shared service.Run a load test with a tool such as a benchmark client at the agreed peak, then compare latency and queue depth against the targets and the Ollama concurrency settings.
GPU and accelerator supportConfirm the hardware vendor and driver stack. vLLM runs on NVIDIA CUDA, AMD ROCm and Intel XPU, while llama.cpp adds Metal, Vulkan and SYCL backends. Third-party accelerators need a plugin, so confirm support for each.Run a hardware compatibility check on a pilot node with the exact GPU model, driver version and container image, and record the engine startup log.
OpenAI API compatibilityList the endpoints your applications call, such as chat completions, responses, embeddings or transcription. vLLM's server exposes several of these, while Ollama offers OpenAI compatibility on port 11434. Confirm each endpoint your code uses.Send a test suite of real client requests through the proxy to the engine and check response shape, streaming behavior and error codes against the OpenAI client library your teams use.
Authentication and exposed endpointsvLLM's --api-key option covers /v1 routes but not /metrics, /health or /pause, and the docs advise a reverse proxy that allowlists only user-facing paths. Ollama's local API has no authentication by default and binds to 127.0.0.1.Scan the server from a user network segment and confirm that only allowlisted paths respond, and that unauthenticated calls to administrative routes are refused.
Metrics and multi-node operationDecide which metrics feed capacity dashboards and alerts. vLLM exposes Prometheus metrics at /metrics, and its multi-node communication is insecure by default, so nodes need an isolated network. Confirm this before planning pipeline parallelism across hosts.Scrape the metrics endpoint into your monitoring stack, trigger a test alert, and review the network design with the security team against the vLLM security guidance.

Which engine fits which requirement set

vLLM is the engine to evaluate first when the requirement is a shared model service for many concurrent users on GPUs. It is an inference and serving engine licensed under Apache-2.0, with no paid edition, so the software cost is zero and the real costs are accelerators, infrastructure and the people who operate it. Its server, started with vllm serve, exposes OpenAI-compatible routes for completions, chat completions, responses and embeddings, so existing OpenAI-based applications can point at it.

  • Licence: Apache-2.0, according to its GitHub repository.
  • API surface: /v1/completions, /v1/chat/completions, /v1/responses and /v1/embeddings, plus audio transcription and translation.
  • Concurrency: Tensor parallelism within a node and pipeline parallelism across nodes, with Ray or its own multiprocessing backend for multi-node runs.
  • Authentication: The --api-key option does not cover every route, including /metrics, /health and /pause. The security docs recommend a reverse proxy that allowlists only user-facing endpoints.
  • Metrics: Prometheus-format metrics at /metrics, covering request counts, token throughput, latency and KV cache use.

Confirm hardware support for your specific accelerators and measure throughput with your own prompt lengths and concurrency, since those results depend heavily on the workload.

Security requirements for endpoints the engine leaves open

Inference engines expose more than the chat route. vLLM and Ollama ship endpoints that an API key or a default bind setting does not protect, so the checklist needs explicit controls around the engine. Each card below names one control and the configuration review or scan that proves it is in place.

Authentication in front of the engine

The vLLM --api-key option protects the /v1, /v2, /inference and /cohere routes, but the vLLM security docs warn against relying on it alone. Decide where user authentication lives, usually an identity-aware gateway or reverse proxy that validates tokens before any request reaches the engine. Verify it by sending requests with no credential and with an expired one to each user-facing route, and confirming every attempt is refused.

Allowlisting endpoints at a reverse proxy

The vLLM docs recommend a reverse proxy that forwards only the endpoints meant for users. Routes such as /invocations, /pause, /resume, /abort_requests, /metrics and /health are not covered by --api-key, so an unlisted path must be denied by default at the proxy. Verify by scanning the engine's port from the user network and from the proxy network, and confirming that control routes return errors to every caller except the operations team.

TLS on every client connection

Prompts, files and API keys travel with every request, so the connection between clients and the proxy should use TLS. llama-server supports TLS through PEM certificate and key files, and vLLM deployments commonly terminate TLS at the proxy. Confirm which hop terminates TLS, that certificates are issued from the organization's own authority, and that plain HTTP is refused. Verify with a TLS scan and a review of the proxy listener configuration.

Isolated network for multi-node traffic

The vLLM security docs state that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption. Place the nodes on an isolated network segment, block inter-node ports from user and application subnets, and treat the Ray or multiprocessing traffic as internal only. Verify with a firewall rule review and a port scan from outside the cluster segment.

No unauthenticated bind address

Ollama binds to 127.0.0.1 on port 11434 by default, and its local API requires no authentication. Changing the bind address with OLLAMA_HOST to expose it on a server network without a proxy in front gives every host on that network unauthenticated model access. Set the bind address deliberately, keep the engine on loopback where possible, and verify with a listening-socket check on each server and a request from a second host.

Access logging for every request

Engine logs and proxy logs should record who called which route, when, with which model and what the response status was. This supports incident response and, for high-risk systems, the automatic logging duties in Article 12 of the EU AI Act, which require logs to be kept for at least six months under Article 19. Verify by sending a test request and confirming the caller identity, route and status appear in the central log store.

Integration requirements with existing systems

An inference service only earns its place when existing applications can call it without rewrites, keys can be issued and revoked, and staff sign in with the identity provider they already use. Each card below is an integration point to agree with the application and platform teams before the first GPU is racked, along with the question to put to each one.

OpenAI-compatible client compatibility

Most applications already speak the OpenAI API, so the engine's compatibility surface decides how much code changes. Confirm which routes each client calls (chat completions, responses, embeddings, or audio) and test them against the engine version you plan to run, not the one in the vendor's marketing. Ask each application team whether they set custom base URLs, model names or streaming options, since those settings break quietly when the model alias changes.

AI gateway for keys and budgets

Engines such as vLLM and llama-server authenticate callers with a shared key, which does not give you per-team keys, spend limits or revocation. Decide early whether a gateway sits in front of the engine to issue keys, enforce rate limits and record usage. A self-hosted gateway such as LiteLLM or Apache APISIX can be scoped as a separate service. Ask each application team which key owner, budget and rotation schedule they need.

Chat front ends and developer tools

Staff-facing chat tools and internal developer tools need the same endpoint but different settings, such as streaming, file uploads and model lists. Open WebUI and LibreChat are common choices, and Open WebUI's license requires written permission to remove its branding once a deployment exceeds 50 users in a rolling 30-day period. Confirm the user count and branding rules before selecting a front end, and ask each team which models they expect to see in their picker.

Observability tooling and metrics

vLLM exposes Prometheus-format metrics at /metrics, and llama-server offers a Prometheus-compatible endpoint when started with metrics enabled. Those metrics must reach the monitoring stack your operations team already runs, with alerts on queue depth, latency and KV cache pressure. Confirm which dashboards and alert routes the operations team owns, and whether prompt-level tracing will be handled by a separate service such as self-hosted Langfuse or Phoenix.

Model storage and artifact locations

Model weights are large, and their location affects backup, access control and the audit trail. Ollama stores models in a default directory per operating system unless OLLAMA_MODELS moves them, and other engines load weights from paths or registries you configure. Define one approved storage location for weights and configuration, decide who can write to it, and confirm that the application teams never pull models from a public registry at runtime.

Identity provider and endpoint access

Engines do not sign users in, so the identity provider must sit in front of them through a reverse proxy or gateway. The vLLM security docs advise allowlisting only the endpoints meant for users, because the --api-key option does not protect routes such as /metrics or /health. Confirm with the identity team which groups map to which endpoints, how service accounts authenticate, and who owns the proxy configuration.

Operational requirements and how to verify them

This table sets out the operating requirements an inference service must meet after launch, with the verification method and the owner for each. Use it to turn each requirement into a named test or runbook before the service carries real traffic. The main takeaway is that the engine's own features cover only part of the job, and monitoring, backup, rollback and handover need named owners.

RequirementVerification methodOwner
Metrics scraped from the engineConfirm Prometheus scrapes /metrics on vLLM or the metrics endpoint on llama-server (enabled with --metrics), and that dashboards show request counts, token throughput, latency and KV cache use. Confirm /metrics is reachable only from the monitoring network.Platform engineering
Capacity monitoring and alertsSet alerts on queue depth, latency percentiles and KV cache pressure. For Ollama, confirm OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS and OLLAMA_MAX_QUEUE are set to tested values, and run a load test at expected peak before go-live.Platform engineering with the service owner
Model and configuration backupStore model weights, chat templates, launch flags, environment files and proxy rules in version control or an artifact store. Test a restore on a second node and record the time it took. Confirm model directories (OLLAMA_MODELS or the vLLM model path) are included.Platform operations
Upgrade cadence and rollbackAgree a pinned engine version and a review cycle, since vLLM ships minor releases about every two weeks (v0.31.0 on October 5, 2026) and Ollama and llama.cpp publish builds frequently. Test each candidate on a staging node with the same models and API client calls, and keep the prior image or binary for rollback.Platform engineering with security review
On-call ownership and handoverName an on-call rota, escalation path and runbook for GPU failure, engine crash, queue saturation and certificate or key expiry. Run a tabletop exercise and confirm the operating team can restart, roll back and read logs without the build team.Head of platform engineering, with the business owner for service levels

The order to agree these requirements

Agreeing requirements in this sequence keeps model choices, capacity targets and security decisions tied to real usage, so each later decision has a written input. The result is a set of artifacts that the engineering, security and compliance teams can all sign against before any hardware is ordered.

  1. Inventory workloads and candidate models

    List every application, team and user group that will call the service, then record the models each one needs, the context lengths they require, and whether they use text, embeddings, audio or vision. Ask each owner for a sample of real prompts and documents so context sizes are measured rather than guessed. Done when the team has a signed workload register that names an owner, a model family and a maximum context length for each use. Confirm model licences at this stage, since the licence of the weights can differ from the licence of the engine.

  2. Set concurrency and latency targets

    Translate the workload register into expected concurrent requests, peak periods, time to first token and total response time for each workload. Use observed usage from existing tools where it exists, and label any assumed figures as assumptions to be replaced after a pilot. Ask the platform team to confirm how the chosen engine handles parallel requests and queueing, because those settings change the latency users see. Done when a targets sheet lists one latency and one concurrency goal per workload, with a named owner who accepts them.

  3. Choose hardware and the inference engine

    Match the model sizes and targets to accelerator types, memory per device, number of nodes and the engine that serves them. Confirm that the GPU or CPU vendor is supported by the engine release you plan to run, and check the engine's parallelism options for models that span several devices or nodes. Ask the hardware supplier for a written statement of supported drivers and runtimes. Done when a hardware and engine decision record names the engine version, the accelerator model, the node layout and the reasons for each choice.

  4. Fix the security and API boundary

    Decide which endpoints users may reach, how callers authenticate, where TLS terminates and which network segments hold the engine and its inter-node traffic. Engine documentation notes that some administrative and metrics endpoints are not covered by the built-in API key option, so a reverse proxy or gateway that allows only the intended routes is usually part of the design. Done when a network and access diagram, an endpoint allowlist and a list of who can reach the metrics and administration paths have been reviewed by the security team.

  5. Agree the observability and logging scope

    Specify which metrics the engine exposes will feed dashboards and alerts, how long request logs are kept, and what is recorded for each call such as caller identity, model, token counts and timestamps. Compare these needs with any record-keeping duties that apply to your use case, and confirm retention with your compliance lead. Done when a metrics and logging specification names each signal, its owner, its alert thresholds and its retention period, approved by security and compliance.

  6. Agree the upgrade cadence and evidence plan

    Set how often the engine, drivers, container images and model weights will be reviewed, tested and promoted, and who approves each change. Engine projects release frequently, so ask for a written change process that covers staging, rollback and the record of what was deployed. Define the evidence the audit or assessment team will request, such as configuration baselines, access reviews and change records. Done when an upgrade policy and an evidence index are signed, and the first upgrade test is scheduled against the staging environment.

Questions buyers ask before choosing an implementation partner

How much does a requirements exercise for an internal LLM service cost?

The cost of the requirements exercise itself is set by the written scope, and DeployOpen does not publish a price list, so ask for a fixed-scope written quote. The larger cost sits in what the requirements decide. Four components matter: GPU or accelerator capacity, the surrounding infrastructure (networking, storage, load balancing, monitoring), implementation work to configure and secure the serving stack, and the people who operate it afterward. Capacity and staffing recur every month, while design and implementation are largely one-time. The engine itself carries no license fee: vLLM is Apache-2.0 licensed and has no paid edition, so its cost comes from the hardware and the people around it. Ask any partner to separate one-time and recurring items in the quote and to state which hardware is in scope.

How long does it take to agree requirements for a self-hosted model service?

The schedule depends on how many decisions are open when the work starts, and no fixed duration applies to every organization. The main drivers are whether the model list, context lengths and latency targets are already written down, how quickly security and compliance owners sign off on the authentication and logging design, whether GPU hardware is already purchased or still under procurement, and how many applications must be migrated to an OpenAI-compatible endpoint. Ask the partner to state the duration in the written scope, the milestones it depends on and which of them are controlled by your own teams. Schedules often slip at procurement and sign-off, not at the technical design.

What does the internal team need to provide for the requirements work?

The internal team provides the decisions and the facts the design depends on. Expect to supply:

  • A list of the use cases, the user groups and the expected concurrency at peak
  • Named owners for security, identity, compliance and the GPU or server platform
  • Access to the target network, hypervisor or Kubernetes platform, and the existing identity provider
  • The data classification rules that determine which prompts and outputs must stay in the environment

The partner will need someone who can approve model licenses and make a decision on hardware sizing. Ask for a written list of inputs and the person responsible for each one before the scope is signed.

Can our own engineers agree these requirements without an outside partner?

Yes, a capable internal team can write the requirements, provided it has people who understand GPU sizing, the OpenAI API surface and the security posture of the engine. The gaps usually appear in three places: the vLLM security docs state that multi-node traffic is insecure by default and that the --api-key option does not cover every endpoint, so a reverse proxy allowlist is required; capacity planning for concurrency and latency needs load testing against real models; and the upgrade cadence matters because vLLM ships new minor releases about every two weeks. A partner is most useful for the parts your team has not run before. Ask a partner to show which of these items its own method covers and which ones remain your responsibility.

What evidence would an auditor expect from an LLM inference requirements document?

An auditor will expect the requirements to be traceable to decisions, controls and records, not just stated. Typical evidence includes the approved model inventory with versions and sources, documented authentication and network boundaries for every endpoint, the metrics and logs collected from the serving layer, and the change and upgrade records for the engine. Where the system is high-risk under the EU AI Act, Article 12(1) requires the system to technically allow automatic recording of events over its lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law says otherwise. Ask the partner which artifacts it produces and which ones your compliance team must own. DeployOpen prepares systems, controls and evidence, but independent auditors and certification bodies issue any report or attestation.

Which requirements break first if we skip the authentication decision?

Skipping the authentication decision leaves the serving engine exposed on paths its own key does not protect. The vLLM --api-key option covers endpoints under /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The vLLM security docs advise against relying on the key alone and recommend a reverse proxy that allowlists only the endpoints meant for users. Ollama's local API at localhost:11434 requires no authentication, and llama-server offers --api-key and TLS options that must be set deliberately. Your requirement should name the gateway or proxy that sits in front, the identity method for callers and the list of endpoints that are reachable from each network zone. Confirm each of these against the version you deploy.

How should the requirements handle upgrades for a fast-moving inference engine?

The requirements should set an upgrade cadence, a test gate and a rollback rule before the first deployment. vLLM published v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026, so a monthly review at minimum is a practical starting point for an in-house team to evaluate. Ollama and llama.cpp also release frequently, with Ollama at v0.40.5 on October 14, 2026 and llama.cpp publishing numbered builds continuously. Agree who approves a version change, how a model is re-validated after an engine change, and how long the previous version is kept for rollback. Ask the partner to document the upgrade procedure and the backup of model artifacts and configuration.

What should a fixed-scope written quote for this work include?

A fixed-scope quote should list the deliverables, the environments covered, the hardware and software in scope, the acceptance criteria and the exclusions. DeployOpen agrees scope, deliverables and timelines in writing before work starts and does not publish prices, so the quote follows from that document. Ask for these items specifically:

  • The engine and version list, plus the model list and context lengths to be validated
  • The concurrency and latency targets, and the load test that proves them
  • Who owns GPU procurement, network changes and identity integration
  • Which metrics, logs and backups are configured, and what the handover includes
  • Whether ongoing support, upgrades or testing are separate retainers

A quote that does not separate one-time work from recurring operations is incomplete.

What the checklist supports for EU AI Act record-keeping

Logging and retention belong in the requirements phase, not after the service is live. Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow the automatic recording of events over the system's lifetime, and Article 19(1) requires providers to keep those logs for at least six months unless other law provides otherwise. Whether either article applies to your model service depends on how you use it, and that decision belongs to your legal and compliance team, not to the platform team or to us. Once that team has made the call, the engineering requirements follow: which events the inference layer must capture, where they are stored, who can read them, how long they are kept and how they are deleted at the end of the retention period. Confirm these early, because log volume and storage design change with the request rate you agree to in the concurrency targets.

On the technical side, the checklist should name the log sources you expect to use and the gaps you must close. vLLM exposes Prometheus-format metrics at /metrics, which cover request counts and latency but are aggregate operational data rather than a per-event audit trail, so confirm with your legal team whether they count as the logs the Act describes. Request-level records usually come from a layer in front of the engine, such as a reverse proxy or an AI gateway, and that layer should be specified as part of the authentication design. DeployOpen prepares the systems, controls and evidence that support a compliance assessment, including log configuration, retention settings, access controls and documentation of what is recorded. It does not issue audit reports, attestations or certifications. Independent auditors and certification bodies make those determinations, so ask them early which evidence they expect to see.

How DeployOpen scopes and delivers this requirements work

DeployOpen turns the requirements for a self-hosted LLM inference service into a written scope before any build starts. The work runs inside your own cloud accounts, data centers or air-gapped networks, and the requirements are agreed with the people who will own the service.

How we scope

  • Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
  • We record the model list, context lengths, concurrency and latency targets for each application that will call the endpoint.
  • We confirm GPU or accelerator vendors, node counts and the network zones where inference servers and their nodes will sit.

How we build

  • We deploy the chosen engine, such as vLLM, Ollama or llama.cpp, in your environment with the agreed OpenAI-compatible endpoints.
  • We place an authenticating reverse proxy in front of the engine and allowlist only the routes meant for users.
  • We enable metrics export, document the multi-node network isolation, and test capacity under the agreed load profile.

How we hand over

  • You receive written documentation of the configuration, endpoints, access rules, metrics and model artifact locations.
  • We document the upgrade procedure for engine releases, which ship frequently, and the approval steps for model updates.
  • We hand over runbooks and evidence records so your team can operate the service and answer assessor questions.

How we support

  • Ongoing support, upgrades and continuous testing are available as a retainer after handover.
  • Retainers can cover engine patching, model rotation, backup of model artifacts and configuration, and capacity reviews.
  • We can rerun endpoint security testing and review changes to the requirements as applications and models grow.

Related

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.
Book a callhello@deployopen.com