LiteLLM Prometheus metrics and monitoring

Prometheus does not wait for LiteLLM to send its numbers. It makes HTTP requests to the gateway's /metrics endpoint on a fixed schedule, stores each sample as a time series, and answers questions about those series with PromQL. On the LiteLLM side, the endpoint appears once you add prometheus to the callbacks in litellm_settings. The prometheus_client package is preinstalled in the LiteLLM Docker image, so you do not add it yourself when you run that image. Confirm that the image tag you deploy includes it, because a custom build may not.

The scrape target depends on where the proxy runs. In Kubernetes, Prometheus usually discovers proxy pods or their service endpoints and reaches the metrics port over the cluster network, so the first check is that the Prometheus pod can open a connection to each proxy on that port and that network policies allow it. On virtual machines, targets are often listed statically or found through a cloud or file-based discovery mechanism. Whichever you choose, confirm the network path, and confirm whether your LiteLLM version exposes /metrics without credentials so you know how to restrict it at the network layer.

Each proxy instance should be its own scrape target. Every process keeps its own in-memory counters and gauges, so when several instances run behind a load balancer, scraping the shared address would return a different instance on each attempt and the series would appear to jump. Scraping each instance directly gives every series an instance label, lets you see one instance falling behind, and makes counter resets from restarts visible. Use rate() or increase() on counters so that a restart does not look like a drop in spend. Prometheus values reliability over precision, so treat these series as operational signals, not as an exact per-request bill.

Pull fits the LiteLLM proxy because it is a long-running service that stays up long enough to be scraped on every interval. Short-lived jobs are different: a batch process that exits before the next scrape cannot be sampled, so it pushes its metrics to a Pushgateway, which Prometheus then scrapes. Do not route proxy metrics through a Pushgateway just because the proxy sits behind other infrastructure. After deployment, check that each proxy instance appears as an up target in Prometheus, and set an alert for a target that stops responding so a silent gap in spend data does not go unnoticed.

Every distinct combination of label values creates its own time series in Prometheus, so the cost of a label depends on how many distinct values it takes. On a LiteLLM gateway, end user, hashed API key, key alias, user and team are the labels most likely to grow out of control, because they follow the population of people and keys using the gateway rather than the number of things you operate. litellm_spend_metric carries several of these at once, so its series count is driven by the combinations of their values, not by any single label. Before you write dashboards, confirm the exact label set for each metric in your LiteLLM version against the Prometheus metrics documentation.

Start by deciding which questions your dashboards and alerts must answer. A finance view of spend usually needs team and model. An on-call alert about a runaway key may need key alias. A per-end-user breakdown is rarely needed as a live metric, and if your privacy or chargeback process requires it, answer that question from the spend log in the database. Write the agreed list down with its owner, and treat every other label as a candidate for removal. Review the list with finance, platform and security together, since each group tends to ask for a different breakdown.

Enforce the list in two places. At scrape time, use metric_relabel_configs in the Prometheus scrape job to drop labels that are not on the list. Dropping a label merges series that differed only in that label, so test the change in a non-production environment first to catch collisions. For dashboards and long-range views, create recording rules that pre-aggregate by the approved labels, such as team and model. Raw series remain in local storage for the configured retention, which defaults to 15 days, so the recording rules are what keep long-term views small and fast.

Illustrative example: a team-by-model breakdown produces one series for each combination of a team and a model that team uses, so growth follows the number of teams and models. A per-end-user breakdown adds a dimension sized by the number of people who call the gateway, and that number grows with every onboarding. Both views can answer useful questions, but only the first is a sensible default for a live dashboard. The second belongs in a report generated from stored logs.

Review the series count after the rollout and again after each LiteLLM upgrade, because new releases can add labels or metrics. Also review it when a large new team or a new model provider is onboarded. Set an alert on sustained growth in series for the LiteLLM metric names so the team hears about a problem before storage fills. Remember that Prometheus favors reliability over precision and is not suited to per-request billing, so use the spend log and the budget features for chargeback and treat Prometheus as the operational view.

Decide where gateway metrics live before you build alerts on them, because alert history depends on how long Prometheus keeps its data. Prometheus writes scraped samples to a local time series database organized into two-hour blocks, with a write-ahead log that protects recent samples before they are compacted into blocks. The default retention is 15 days, and you can configure retention by time or by size. Fifteen days is a default, not a retention decision. Check it against how far back your spend reviews, budget investigations and incident reviews need to look, and confirm the chosen limits in your deployment configuration before go-live.

Local storage is not clustered or replicated, and Prometheus documentation says to manage it like any other single-node database. If the node or its disk fails, the history stored on it is lost with it. That makes two decisions for the platform owner. The first is how long data stays on local disk, sized against the volume of LiteLLM series you expect. The second is whether to configure remote write to a long-term storage system, which Prometheus supports through its remote write and remote read protocols. Ask who operates that system, who controls access to gateway metrics, and how Prometheus behaves if the remote endpoint is unavailable. Also decide whether a second Prometheus instance is part of the design, since duplicate scraping is a deliberate choice you must plan and test.

Alertmanager receives alerts from Prometheus and controls how they reach people. Grouping combines related alerts into one notification, inhibition suppresses dependent alerts while a higher-severity alert is firing, and silences cover planned maintenance or known noise. These settings decide whether an on-call engineer gets one clear page or a flood of duplicates, so write the routing tree with the gateway's owners before the first rule ships. For high availability, Alertmanager can run as a cluster configured through its --cluster flags. Point Prometheus at every Alertmanager instance directly, not through a load balancer in front of them, so that each instance receives alerts and the cluster can group and deduplicate notifications. Test this with one instance stopped during a drill.

Spend alerts need to account for a known delay. LiteLLM recommends batching spend updates to the database every 60 seconds at scale, so spend dashboards and any queries built on that data can trail real usage by about that long. Prometheus is designed to favor reliability over precision and is not suited to per-request billing, so treat spend metrics as signals for alerting and not as the accounting record. Set budget alert thresholds with headroom for that lag, so a warning fires while there is still time to act on the key or team involved. In a non-production environment, run an illustrative test with a low budget and measure how long the alert takes to fire after the threshold is crossed.