What self-hosted LLM inference costs compared with a hosted API
The answer to whether a team should run models on its own GPUs or pay per token depends on utilization, staffing and request volume, not on the serving engine being free. vLLM is licensed under Apache-2.0, Ollama under MIT and llama.cpp under MIT, so none of them carries a license fee. The engine is the cheapest line on the budget. The expensive lines are the accelerators, the power and space that run them, and the people who keep the service patched, monitored and available. Spend the most effort on the cost components that recur. GPU capacity is paid for whether or not requests arrive, so a lightly used server costs the same per hour as a busy one, and the cost per token falls only as the hardware stays loaded. Model and engine upgrades also recur: vLLM ships new minor releases about every two weeks, so someone must test each release against your models and endpoints. Ask any partner to separate one-time items (hardware purchase, network build, initial hardening, integration) from recurring items (rental or depreciation, power, cooling, support hours, on-call rotation, testing of upgrades) and to state which volume and utilization assumptions the quote depends on. Before comparing totals, measure your own demand. Record peak and average requests per minute, prompt and completion token counts, the context lengths your applications send, and how many hours per day the service must answer. Then price the same workload against the hosted API you would otherwise use, including its per-token rates for the model you need. Model the scenario where demand is half your forecast, because idle GPUs are the most common reason self-hosting costs more than expected. Also account for OWASP's LLM10:2025 Unbounded Consumption risk: without per-user limits and budgets, a single client can exhaust the capacity you are paying for, so budget controls belong in the cost model, not in an afterthought.
Which cost components apply to in-house, partner and SaaS options
This table compares the cost components of running an open-weight model service in-house, through an implementation partner, or through a hosted per-token API, and marks each as one-time or recurring. Read each row as a question to answer in your own quote: who is paid, and what keeps that cost moving. The main takeaway is that in-house cost is mostly people and utilization, while hosted API cost scales with tokens.
| Cost component | In-house on your GPUs | Implementation partner | Hosted per-token API |
|---|---|---|---|
| GPU purchase or rental | Recurring if rented, one-time if purchased. You pay the hardware vendor or cloud provider. Idle capacity is still paid for. | Usually not paid by the partner. Confirm whether the quote includes hardware procurement or only sizing advice. | None. Capacity is inside the per-token price. |
| Power, cooling and space | Recurring. Paid to your data center, colocation provider or utility. | Not paid to the partner. Confirm who holds the colocation or cloud contract. | None. |
| Engineering time for setup and integration | One-time. Paid in staff time or to an internal team. | One-time, fixed scope. Paid to the partner under a written statement of work. | Low. Mainly application changes to call the API and handle keys. |
| Engine upgrades and releases | Recurring. vLLM and llama.cpp publish releases often, so someone must test and roll them out. | Recurring if under a retainer. Confirm release cadence and test scope in writing. | Handled by the provider. Model version changes can still affect your application. |
| On-call operations and capacity monitoring | Recurring. Paid in staff salary or on-call allowance. Scales with uptime commitments. | Recurring if the partner runs operations. Confirm hours covered and escalation paths. | Included in the provider's service. Your uptime depends on their terms. |
| Security hardening and access control | One-time build, then recurring checks. Endpoints need a reverse proxy and authentication in front of the engine. | One-time hardening plus optional retainer testing. Paid to the partner. | Provider-side controls. Your costs are API key management and data handling reviews. |
| Monitoring, logs and model artifact storage | Recurring storage and tooling costs. Paid to your storage or cloud provider. | Recurring if managed by the partner. Confirm where weights and logs are kept. | Usage logs held by the provider. Retention terms set your evidence obligations. |
| Per-token or subscription fees | None for open-weight models run locally. Ollama states that running models on your own hardware is unlimited. Its paid plans cover cloud models only. | Usually none for the engine itself. Confirm that no licence fees apply to any component. | Recurring per token or per request, paid to the API provider. Scales directly with volume. |
Which costs are one-time and which recur every month
The one-time costs of a self-hosted model service are the ones you pay before the first production request: the hardware purchase or the setup of a rental contract, the initial architecture for the serving layer, the hardening work on the endpoint, the design of who can reach it and how, and load testing to find out how many concurrent users one node really supports. Each of these is usually billed as project effort or as a capital line on the hardware, and the split depends on how your finance team classifies the spend. Ask finance now whether GPU servers are capitalized and depreciated over a fixed period, whether a rented GPU fleet is an operating expense, and whether implementation services can be capitalized at all. The answer changes how the first year looks on paper even when the total is the same.
The recurring costs are where most of the money goes over time. They include power, cooling and rack space if the hardware is in your data center, or the hourly or monthly rate if it is rented, plus the staff time to monitor capacity, rotate keys, review access, back up model artifacts and configuration, and respond to incidents. Engine upgrades also recur, and the effort is easy to underestimate. The vLLM project shipped minor releases about every two weeks in the period it published its release history, with v0.28.0 on August 26, v0.29.0 on September 9, v0.30.0 on September 22 and v0.31.0 on October 5, 2026. Ollama and llama.cpp publish even more often. Each release needs a test run against your models, your reverse proxy rules and your metrics dashboards before it reaches production, so budget for a steady cadence of upgrade work rather than a single migration.
Before you compare totals with a hosted API, confirm three things with your finance team and your implementation partner: which lines fall under capital and which under operating expense, how long the hardware is expected to be depreciated, and whether the quote separates one-time setup from a recurring operations retainer. Ask for the upgrade and patch effort to be written as a named recurring line with its own scope, so it is not hidden inside a general support figure. A quote that cannot tell you which costs repeat each month is not yet comparable to a per-token invoice.
What moves the cost up or down
Hosted APIs bill per token, so their cost follows usage directly. A self-hosted service carries a largely fixed cost for hardware, power, operations and controls, so the same cost drivers decide whether it stays cheaper. The cards below cover the levers that move that fixed cost and the utilization it is spread across.
GPU utilization drives unit cost
Hardware is paid for whether or not it serves tokens, so idle capacity is the largest hidden cost in an in-house service. Cost per token falls as utilization rises and rises sharply when GPUs sit at night or between projects. Ask for measured utilization by hour and by model over several weeks before comparing against an API price. If the workload is bursty, ask whether the GPUs can be shared across teams or models, or whether part of the demand should stay on a hosted API.
Concurrency must be configured
Engines default to conservative concurrency, so a service that works in a test can queue badly under load. Ollama, for example, defaults to one parallel request per model and a queue of 512 requests (OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE). Each setting changes how many GPUs a given latency target requires. Confirm peak concurrency, time-to-first-token targets and queue behavior before sizing hardware, and test them with realistic prompt lengths.
Model size and context length
Larger models and longer context windows consume more GPU memory and slow each response, which reduces how many users one card can serve. Quantization can lower memory use, but it changes output quality and must be tested on your own tasks. Decide which models must run, the maximum context length each application needs, and whether a smaller model can handle routine work so that the largest model is reserved for hard cases.
Unbounded consumption is a cost event
The OWASP Top 10 for LLM Applications 2025 lists LLM10:2025 Unbounded Consumption, uncontrolled resource use by model requests. On self-hosted models, exhausted GPU capacity is the equivalent of an unexpected bill, because it delays other users and cannot be turned off per request. Set per-user and per-application rate limits, maximum output lengths and request budgets at a gateway in front of the engine, and alert when usage departs from the forecast.
Environments and node count
Each environment, such as development, test, staging and production, usually needs its own model deployment, and high availability adds a second node. Multi-node serving across machines adds interconnect hardware and networking work. vLLM's documentation notes that node-to-node traffic is insecure by default and must be isolated on a private network, which adds cost for network design and testing. Ask which environments need full-size hardware and which can use smaller or shared capacity.
Security and compliance scope
Every system that handles prompts, outputs or logs may fall within audit scope, which adds costs for access control, logging, retention, backup and evidence collection. Requirements such as record-keeping under the EU AI Act can require logs kept for at least six months, which means storage that grows with usage. Confirm the regulations that apply before sizing storage and staff time, and ask a partner to separate the one-time control build from the recurring evidence work.
The request volume at which self-hosting starts to pay
The break-even point is the monthly token volume at which the fully loaded monthly cost of a self-hosted service equals the monthly spend on a hosted API for the same tokens. No break-even figure is supplied for this topic, so you have to build it from your own inputs. On the self-hosted side, add the GPU or accelerator cost (purchase amortized over its life, or rental), power, cooling and rack space, the platform engineering and on-call time needed to run the engine, and the cost of the monitoring and backup work around it. On the hosted side, use the per-token prices in written quotes from each provider, applied to token counts measured from your own traffic, not to the volume you hope to reach. Model the self-hosted cost at a utilization you can defend from load tests, because an idle GPU still costs money every month.
The result is sensitive to utilization more than to almost any other input. Low or bursty traffic favors a hosted API, because you pay only for tokens consumed and do not carry idle capacity sized for peak demand. Steady, high utilization favors self-hosting, because fixed costs are spread across many requests and the hosted bill grows with every token. Peak periods matter too: a service sized for a month-end or launch spike may sit at low utilization for most of the year, which can erase the advantage. Before you commit, ask each partner or vendor to show the capacity assumptions behind its quote, and check how the GPU count changes with concurrency, context length and latency targets.
Check the break-even model against unit costs that are easy to forget. Governance overhead is one: you may need access controls, usage budgets per team, and request logs that the organization keeps for its own records, and the OWASP Top 10 for LLM Applications 2025 entry on unbounded consumption describes how uncontrolled requests can exhaust self-hosted GPU capacity. Another is the cost of a second site or spare node for resilience, which a hosted API does not require from you. Recompute the model whenever traffic, model size or hardware pricing changes, and record the assumptions so a later review can see when the answer flipped.
How to build your own cost estimate
This sequence turns a vague sense that self-hosting might be cheaper into a costed model with written inputs, three utilization scenarios and a like-for-like comparison against hosted API pricing.
Measure tokens per day and peak concurrency
Start from the workload you already have, or from a documented projection if none exists. Count input and output tokens per day for each application, then find the busiest hour and the number of requests in flight at that moment. Your platform owner or the team that runs the applications can usually pull this from gateway logs or API invoices. The step is done when you have a daily token total, a peak concurrency figure and a note on how confident you are in each number. Without peak concurrency, you cannot size GPUs, because capacity is set by the peak, not the average.
Set latency and availability targets
Agree with the business owners how fast responses must start and finish, and how much downtime the service can tolerate in a month. Write these as measurable targets, such as time to first token and time per output token at peak load, plus the availability the service must meet. These targets decide whether one GPU is enough, whether you need redundant nodes and whether a batch overnight is acceptable. The step is complete when the targets are signed by the service owner and written next to the token figures from the previous step.
Choose candidate models and GPU classes
Shortlist the open-weight models that meet the quality bar for each use case, then note the context length and memory each needs at your concurrency. Map these to GPU classes or accelerators, and confirm with the engine documentation which hardware it supports, such as NVIDIA CUDA, AMD ROCm or CPU-only options through llama.cpp. Check each model's license for commercial use. The output is a table of model, precision, memory need and the hardware class that can serve it at your peak. Smaller quantized models can change the GPU count substantially, so test quality before fixing the design.
Price hardware or rental and power from written quotes
Request written quotes from at least two hardware vendors for purchase, and from one or more cloud or colocation providers for rental. Ask each for the full line items: servers, accelerators, networking, storage, support contracts, racks, power and cooling where you host. For purchases, record the one-time cost and the expected support renewal. For rentals, record the hourly or monthly rate and the commitment term. Confirm the power draw per node and the data center rate per kilowatt, because power and space recur every month. The step is done when every hardware and facility line has a written source and a recurring cost sits in its own column.
Add engineering, on-call and security hardening time
Estimate the people who build and run the service. Separate the one-time work, such as installing the engine, placing a reverse proxy in front of it, configuring authentication, isolating the node network and setting up backups, from the recurring work of on-call rotation, incident response and access reviews. Ask your security team to list the hardening steps the engine does not provide out of the box. Ask each partner bidding for the work to state the effort in person-days and what it includes, in writing. The output is a labeled list of one-time and recurring staff time with a named owner for each line, so the cost is not hidden in existing salaries.
Add upgrade, monitoring and capacity planning effort
Inference engines release new versions often, so budget for regular testing, staging, rollout and rollback of engine and model updates. Add the effort to collect metrics, for example from the Prometheus-format endpoints that vLLM and llama-server expose, build dashboards and alerts, and review KV cache use and queue depth as load grows. Include the time to re-forecast capacity each quarter and to plan hardware refreshes. The output is a recurring monthly effort line for maintenance, monitoring and capacity review, with its owner, so the total cost of ownership reflects the work that continues after launch.
Compare against the provider's hosted API price sheet
Take the daily token totals from step one and multiply them by the input and output prices on the hosted provider's own published price sheet, using the exact model you would compare against. Check whether the price is per million tokens, whether cached or batch pricing applies and whether any minimum commitment or enterprise terms change the rate. Convert both options to the same unit, typically a monthly cost per million tokens served. The output is a monthly figure for the hosted API at your volume, which you set beside the self-hosted total from the previous steps. Do not compare a small model on one side with a frontier model on the other.
Model three utilization scenarios
Build a low, expected and high utilization scenario using the token and peak figures you measured. Self-hosted cost is largely fixed by the hardware you run, so the cost per token falls as utilization rises, while hosted API cost moves in direct proportion to tokens. Find the monthly token volume where the two lines cross for each scenario, and state the assumptions behind each crossover point. Ask the partner or vendor to model the same three scenarios so the outputs can be compared line by line. The step closes when the three scenarios, their crossover points and the quotes behind them are in one document that the decision owner has reviewed.
When a partner, a hosted API or in-house operation fits best
When traffic is steady and high, GPUs stay busy for most of the day and the fixed cost of hardware and operations is spread across many requests. An in-house platform team can make that pay off, but the organization carries every layer: hardware selection, the serving stack, upgrades, capacity planning and on-call coverage. Serving engines such as vLLM have no license fee, so the money goes to accelerators, data center space, power and the people who keep the service running.
- GPU purchase or lease: Capital or lease payments recur whether or not the cards are busy, so utilization determines the effective cost per request.
- Power, cooling and space: These recur monthly and depend on your facility or colocation contract.
- Platform engineering time: Engineers own upgrades, scaling, metrics and incident response for a service that changes every few weeks.
- Redundancy: A second node or failover path is needed if applications depend on the endpoint.
- Security controls: Endpoints need authentication and a reverse proxy, which someone must build and maintain.
Confirm your sustained utilization figures, peak-hour capacity needs and the staff hours your team can commit before choosing this path.
When the organization lacks a team that can run GPU infrastructure, a partner can design, build, document and hand over the service. The partner is paid for scope, so the organization still carries the hardware and the ongoing operations, but avoids building expertise from scratch before the first user logs in. Fixed-scope engagements agree deliverables and timelines in writing before work starts, which makes the cost of the build visible before it begins.
- Build fee: A one-time cost for design, installation, integration, hardening and documentation, quoted from the written scope.
- Hardware still owned by you: GPUs, servers and facility costs continue to recur on your books.
- Knowledge transfer: Confirm whether runbooks, configuration and training are part of the scope or an extra item.
- Post-handover support: Ongoing support, upgrades and testing are often sold as a retainer, which is a recurring cost to plan for.
- Internal ownership gap: Someone must still approve model changes, access requests and capacity decisions.
Ask the partner to state in writing who responds to an outage, how upgrades are tested and what the handover package contains before signing.
When requests are infrequent or arrive in short bursts, dedicated GPUs sit idle for long stretches and the cost per request can rise sharply. A hosted API charges per token and needs no hardware, which can cost less to operate for this pattern. The trade-off is that prompts and outputs leave your infrastructure, and the provider sets the terms for availability, data handling and pricing changes, so the choice must pass your data classification and vendor review first.
- Per-token charges: Costs follow usage and rise with volume, so forecast monthly token counts before comparing.
- Data transfer approval: Legal, privacy and security review of the provider's data terms is a one-time cost that can delay adoption.
- Vendor dependency: Exit planning, model deprecation notices and rate limits are outside your control.
- Idle capacity alternative: On-premises hardware kept for occasional use still incurs power, space and maintenance costs.
- Gateway work: A self-hosted gateway can route between hosted and local models, adding a component to operate.
Confirm your expected monthly request and token volumes, peak-to-average ratio and whether any of the data may legally leave your environment.
Questions buyers ask about self-hosted LLM inference cost
What does self-hosted LLM inference actually cost?
The cost has four main parts: GPU or accelerator capacity (bought or rented), power, space and cooling, the engineering time to install and tune the serving stack, and the on-call time to keep it running. The engine itself carries no license fee. vLLM is Apache-2.0 licensed with no paid edition, and Ollama's documentation states that running models on your own hardware is always unlimited. Most of the spend is hardware and people, so the largest swing factor is how busy the GPUs stay. Ask any partner for a written quote that separates one-time items (hardware, installation, integration) from recurring items (support, upgrades, monitoring, capacity reviews), and that states who pays for each line.
How long does a self-hosted inference setup take?
No duration is supplied for this work, so no timeline should be assumed from this page. The time depends on the models chosen, the number of GPUs and nodes, whether the cluster already exists, how the endpoint is secured and how many applications need to connect. Ask for a written timeline tied to a written scope, with milestones for hardware readiness, a working endpoint, authentication in front of the engine, metrics and a handover. A partner that cannot break the plan into those milestones has not yet scoped the work.
What does our team need to run a model service?
Plan for people who can operate Linux servers, container platforms and GPU drivers, who understand network segmentation, and who can watch metrics and respond to incidents. For multi-node runs, vLLM's security documentation says inter-node traffic is insecure by default and must sit on an isolated network, so someone must own that design. You also need an owner for model selection and updates, and someone who can judge whether an engine release is safe to deploy. Identify these roles before the quote, because the gap between your staff and the required skills is what the partner's scope should fill.
Can our engineers build this themselves with vLLM, Ollama or llama.cpp?
Yes, a capable team can run these engines directly, and each has a different fit. vLLM is an inference and serving engine that exposes OpenAI-compatible endpoints and scales across GPUs and nodes, which suits shared services. Ollama is an MIT-licensed runtime with a REST API on port 11434 that binds to localhost by default and has no built-in authentication for that local API, so it needs a proxy before any shared use. llama.cpp, under MIT, runs GGUF models with a server that supports API keys, TLS and a metrics endpoint, and suits smaller or CPU-based deployments. The hard parts are authentication, reverse proxying, capacity planning and patching, which are where self-built projects usually stall.
What recurring work follows the launch of a model service?
Recurring work includes engine upgrades, since vLLM ships new minor releases about every two weeks (v0.28.0 on August 26, 2026 through v0.31.0 on October 5, 2026), and Ollama and llama.cpp also release frequently. Each release needs testing against your models and your clients before rollout. Other recurring items are capacity monitoring against the Prometheus-format metrics at /metrics, reviewing request queues and latency, rotating credentials, renewing certificates, backing up model artifacts and configuration, and reviewing access. Ask a partner whether these are included in a retainer, billed by the hour or excluded, and how an engine upgrade is tested before it reaches users.
Do Ollama's paid plans change the cost picture?
They change the cloud side only. Ollama's pricing page states that running models on your own hardware is always unlimited, and its paid plans cover cloud models: Pro at $20 per month ($200 per year), Max at $100 per month, Team at $500 per month and Enterprise at custom pricing. The Enterprise plan adds model access controls, cost budgets per user or API key and a dedicated support channel. If your models run on your own GPUs, those subscriptions do not replace the hardware, operations or security costs in your model. If you use Ollama's cloud models, add those plan costs to your comparison, and confirm current terms on the pricing page before you decide.
How do we compare a partner quote with a hosted API bill?
Put both costs on the same monthly basis, covering every item the self-hosted option adds. Hosted APIs bill per token, so your bill rises with volume. A self-hosted service carries mostly fixed costs for capacity, people and operations, so its cost per request falls as utilization rises and climbs when GPUs sit idle. Compare your current and forecast token volume, peak concurrency and latency needs against the GPU capacity the partner proposes, then add power, space, support and upgrade work. Ask the partner to show utilization assumptions in writing, and request a sensitivity view at lower and higher request volumes. Illustrative example: a team with light, bursty traffic may find a hosted API cheaper, while steady high volume on owned GPUs may favor self-hosting.
How DeployOpen scopes the cost of a self-hosted model service
DeployOpen plans, implements, hardens, documents and supports self-hosted platforms inside the client's own infrastructure, whether that is their cloud accounts, data centers or air-gapped networks. For a model service, that means the cost estimate starts with a written scope: which engine is chosen, how many model endpoints run, which GPUs or servers host them, how users authenticate, what monitoring is in place and who operates the service afterwards. DeployOpen does not publish a price list, and it does not estimate its own fees in advance. A quote follows from the written scope, so the useful preparation is on your side: a list of the models you plan to serve, the number of users and applications calling them, peak concurrency and the hardware you already own or are allowed to buy.
Most engagements are fixed-scope. Scope, deliverables and timelines are agreed in writing before work starts, which lets you compare an implementation quote against the hosted API spend it would replace using the same assumptions. Ask that the written scope separate one-time work (architecture, installation, gateway and authentication setup, hardening, documentation and handover) from recurring items such as engine upgrades, model updates, patching, monitoring and on-call coverage. Ongoing support, upgrades and continuous testing are available as retainers, and those should be priced as a separate line so the first-year total is not mistaken for the long-run cost. Hardware, power and space remain your costs whichever partner you choose, so the quote should state clearly what is excluded.
Before you sign, confirm three things in the scope document: the hardware assumptions and the utilization they were sized for, the handover of operations to your own staff, and the exit terms if you later move workloads back to a hosted API. Ask DeployOpen to show how each assumption changes the quote, for example what happens if peak load doubles or a new model family needs a different GPU class. A quote that cannot be traced back to these inputs is a reason to ask more questions before committing.
How DeployOpen scopes, delivers and supports a fixed-scope inference platform
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside the client's own cloud accounts, data centers or air-gapped networks. For an inference cost decision, we start by writing down the GPU, utilization and staffing assumptions so the comparison against a hosted API rests on figures the client can check.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope lists the models, expected request volume, peak concurrency and the GPU or server hardware the client already owns or plans to rent.
- We confirm which costs sit in the quote, including hardware, power and space, implementation and the operating staff who will run the service.
How we build
- We deploy vLLM, Ollama or llama.cpp in the client's environment and place an authenticated reverse proxy in front of it, since the engine's own API key does not cover every endpoint.
- We set up Prometheus-format metrics and utilization dashboards so GPU use and queue depth are measured from day one.
- We run load tests at the concurrency levels in scope and document the measured throughput so the self-hosting cost model uses observed figures.
How we hand over
- Handover includes a cost model worksheet that the client's finance and platform teams can update with their own rates and volumes.
- We document the runbooks for engine upgrades, model updates and capacity changes, including how to test a new engine release before rollout.
- Deliverables include configuration, access controls and records that compliance teams can use as evidence, though audit reports come from independent auditors.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, and we agree their scope in writing.
- Retainers can cover monitoring review, engine and model patching and periodic re-running of the cost comparison as volumes change.
- We recommend confirming with your own finance and operations teams which costs a retainer covers before comparing it with in-house staffing.
