Self-hosted LLM inference RFP questions

No. A credible partner asks for workload data first and quotes hardware only after the numbers are agreed in writing. Hardware follows from the model size, context length, concurrency and latency targets, so a quote issued before those are known is a guess. Ask the partner to name the inputs they need: the models you intend to run, the longest prompt plus response you expect, peak and average concurrent users, the response time staff or applications will accept, and the hours the service must be available. Ask for a written sizing worksheet that lists each assumption, so you can change one input and see how the GPU count moves. Ask whether the quote covers only hardware or also the engine configuration, load testing and capacity monitoring that keep the sizing honest after go-live.

Model weight size and numeric precision set the baseline memory a GPU must hold before any user request arrives. Lower precision reduces that footprint, which is why quantized formats matter. The llama.cpp project supports integer quantization from 1.5-bit to 8-bit to reduce memory use and speed inference, and vLLM and other engines support their own quantized formats. The trade-off is output quality, which varies by model and task, so the partner should test the quantized model against your own evaluation set before accepting it. Ask which precision they propose for each model, what quality test they will run, who approves the result, and whether the sizing changes if you later switch to a different model family or a larger one.

The KV cache stores the attention state for every token in every active request, so memory use grows with both context length and the number of requests served at once. A model that fits comfortably for one short chat can run out of memory when many users send long documents at the same time. vLLM reports KV cache use through its Prometheus-format metrics, which makes it a good signal for sizing and for alerting. Ask the partner how they will set the maximum context length per model, how many concurrent sequences the KV cache can hold on the proposed hardware, what happens to requests when the cache is full, and which metric will show that condition in production. A partner who cannot answer these questions has not sized the workload.

Set concurrency and latency targets as measurable numbers before any hardware discussion. Concurrency is the count of requests in flight at peak, and latency targets usually split into time to first token and total response time, both measured at the peak load you expect. Ask the partner to run a load test against a realistic prompt mix and report the results at your target concurrency, not at a single request. Ask how they will measure throughput, how queueing behaves when demand exceeds capacity, and what limits they will set per user or per application. Engine defaults matter here: Ollama, for example, documents a default of one parallel request per model, so a team that never changes that setting will see queueing long before the GPUs are busy.

Tensor parallelism splits each layer of a model across several GPUs in the same server, so they work together on every token. Pipeline parallelism places different layers on different GPUs or nodes, so a request passes through them in sequence. In plain terms, tensor parallelism is the choice when the model is too large for one GPU and the GPUs share a fast link inside one machine, while pipeline parallelism is used to span nodes when one server cannot hold the model. vLLM supports tensor parallelism within a node and pipeline parallelism across nodes, and it uses Ray or its own multiprocessing backend for multi-node runs. The vLLM security documentation states that node-to-node communication is insecure by default, so any multi-node design needs an isolated network. Ask the partner which split they propose, why, and how they will test the interconnect.

Headroom is the capacity held back for peak demand and for the loss of a GPU, server or node. The right amount depends on your peak pattern, your availability targets and how fast a replacement can be brought up, so no single percentage applies to every organization. Ask the partner to model your peak as a separate scenario, to state what capacity remains if one GPU server fails, and to explain whether a failed node degrades latency or stops the service. Ask how spare capacity is paid for, since idle GPUs are the largest recurring cost of a self-hosted service, and how utilization will be reported so you can judge whether the headroom is being used. Request the sizing for both normal and peak conditions, written down.

A single benchmark reports throughput for one model, one prompt length, one batch pattern and one engine version, so it rarely matches your traffic. Real load mixes short and long prompts, streaming and non-streaming calls, and uneven bursts across users. Engines also change between releases: vLLM ships new minor releases about every two weeks, so a number taken on one version may not hold after an upgrade. Ask the partner to show the test conditions behind any figure they cite, including prompt and output lengths, concurrency, engine version and hardware. Ask them to repeat the test on your own prompt traces and to rerun it after each engine upgrade. A sizing that cannot be reproduced on your data should be treated as an estimate, not a commitment.

A sizing worksheet should list every input and every output so that your team can check the arithmetic. Inputs include the models and their precision, maximum context length, expected requests per minute at peak, prompt and output token distributions, latency targets, redundancy target and the engine and version to be used. Outputs include the GPU model and count per model, the parallelism layout, the KV cache budget, the expected queueing behavior at peak, and the monitoring signals that will show when capacity is running short. Ask the partner to state which values are measured and which are assumed, and to attach the load test results behind them. The worksheet should be part of the written scope so it can be revised when your workload changes, and it should be revisited whenever a new model or a major engine release is added.