Monitoring LLM traffic in Apache APISIX

Monitoring LLM traffic through Apache APISIX: what the gateway can measure

When several teams call LLM providers through one gateway, the gateway is the one place that sees every request, so it is the natural source for usage numbers. APISIX records the model name, request and response token counts, request duration and latency metrics, but it does not produce billing-grade figures. Treat its output as an operational signal for spotting a runaway team, a misbehaving integration or a slow provider, and reconcile it against provider invoices before anyone charges a cost back to a department or customer. The measurement comes from two places. The ai-proxy-multi plugin can write a summary per request when logging.summaries is enabled, and the entry includes the LLM model, the duration, and the request and response tokens. The prometheus plugin exposes metrics such as apisix_http_status, apisix_http_latency and apisix_llm_latency at /apisix/prometheus/metrics on port 9091 by default. The 3.18.0 release added built-in Prometheus histograms for time to first token and token distribution, and apisix_llm_latency gained a type label to distinguish them. Any dashboard or alert written against the earlier label set must be checked after an upgrade, and the reader should confirm the exact label values in the APISIX changelog for their version. Teams need this when they cannot yet answer simple questions: which consumer used which model, how many tokens each team consumed last month, and whether a provider failover changed latency for users. The decision to make early is where the numbers will live and who owns them, because Prometheus and the logs it feeds are operational tools with their own retention and accuracy limits. Confirm with the vendor and in your own environment how the type label, the summary log format and the metrics endpoint port are set for your APISIX release before you build reports on them.

Which signals APISIX exposes for LLM traffic

APISIX gives you four signals for LLM traffic, and each answers a different question. Request metrics show whether the gateway is serving traffic, the LLM latency histograms show how the model responds, summary logs show what each request consumed, and consumer identity shows who sent it. Read them together, and check the limits of each before you build a dashboard or a billing report on it.

Request status and latency

The prometheus plugin exposes apisix_http_status and apisix_http_latency at /apisix/prometheus/metrics on port 9091 by default. These answer whether requests to your LLM routes succeed, how many fail, and how long the gateway takes end to end. They do not separate model time from gateway time, and they say nothing about tokens. Confirm which labels your APISIX version attaches before you filter by route.

Time to first token histograms

APISIX 3.18.0 added Prometheus histograms for time to first token and token distribution, carried on apisix_llm_latency with a new type label. Time to first token shows how long a user waits before streamed output begins, which matters more for chat than total duration. Dashboards and alert rules that match the metric without the type label will need updating after the upgrade.

Per-request summary logs

With logging.summaries enabled, ai-proxy-multi logs each request's LLM model, duration, and request and response tokens. This is the only signal here that ties token counts to a single call, so it is the basis for per-request usage analysis. Summary logs are per request, so their volume grows with traffic, and you must set a retention period for them that fits your audit and cost requirements.

Consumer identity in logs

Consumer identity appears in logs only when an authentication plugin such as key-auth or jwt-auth identifies the caller before the request reaches the AI plugin. Without it, you can attribute usage to a route or an upstream but not to a team or customer. Confirm that your consumer names are stable and that the summary log line carries them, because cost reports depend on that join.

Configuration options and when each fits

This table compares the three ways to get LLM usage out of APISIX: the Prometheus metrics endpoint, summary logging through logging.summaries, and custom log formats shipped to an external system. Read each row for the data it gives, what it costs in storage and scrape load, and which team usually owns it. Most teams combine a metrics view for trends with logs for per-request and per-consumer questions.

Option	Data it gives	Cost and who it suits
Prometheus metrics at port 9091	Route and status counts, latency, bandwidth and apisix_llm_latency, including time to first token from 3.18. No prompt or per-request detail.	Costs scrape load and Prometheus storage that grows with label cardinality. Suits platform and SRE teams watching trends and alerts.
Metrics endpoint exposure at /apisix/prometheus/metrics	Same metric set, reachable only where you expose it. Confirm the bind address and access controls in the prometheus plugin docs.	Low cost if restricted to the monitoring network. Exposing it widely leaks route and usage detail, so security teams should review it.
logging.summaries on ai-proxy-multi	Per-request LLM model, duration, and request and response token counts, written to the gateway logs.	Adds one log line per request, so log volume and retention cost rise with traffic. Suits finance and product teams needing per-model usage.
Custom log format	Chosen fields from the request context, such as consumer name or token counts, if your APISIX version exposes them. Confirm variable names in the logger docs.	Cost depends on which fields you keep. Requires someone to maintain the format when fields change. Suits teams building cost reports.
External log shipping	Gateway logs delivered to a log store where you query by consumer, model or time window.	Recurring cost for ingestion and retention in the target system. Suits teams that already run a log platform and need audit evidence.
Metrics and logs combined	Trend dashboards from metrics plus per-consumer cost detail from logs, joined by time window.	Highest combined cost and two pipelines to run. Suits organizations reporting spend per team or customer.

Why dashboards break on the new type label in 3.18

APISIX 3.18.0 added a `type` label to `apisix_llm_latency`, and any query, recording rule or panel that matches that series without the label can return empty results or, when several series are summed together, double-counted results. Before you upgrade a gateway that feeds Prometheus, export every dashboard, alert rule and recording rule that references apisix_llm_latency and list which ones filter, group or aggregate on its labels. Pay particular attention to sum and avg expressions with no by clause, because those are the most likely to change meaning once the new label splits the series. Confirm the exact label values in the APISIX changelog entry for 3.18.0 before you write any replacement query, since the values determine which series each panel should select.

Once the audit is complete, split time to first token from total latency explicitly in every panel that shows LLM response times. A panel that averages all apisix_llm_latency series will mix the two measurements after the upgrade, so each query should select one type value and state its meaning in the panel title. Alerts need the same treatment: a threshold written against total latency should not fire on time to first token, and the reverse is also true. Keep the existing panels available under their old names until the new ones have been checked against real traffic, so that a reviewer can compare the two views side by side.

Test the revised queries on a staging gateway that runs the same APISIX version and sends representative LLM traffic through the same ai-proxy-multi routes you use in production. Compare the series counts and the query results for the same time window against the current version, and confirm that every panel returns data for each provider and route you expect. Record the results in the change ticket, then promote the dashboards and alert rules together with the gateway upgrade. Because Prometheus values reliability over precision, treat these metrics as operational signals rather than a billing record, and keep per-consumer cost reporting in the logs described elsewhere on this page.

Setting up LLM metrics and logs in APISIX

This sequence turns an APISIX gateway that already proxies LLM calls into a measured system: model and token fields in logs, the LLM latency series in Prometheus, and per-consumer views with retention you have chosen on purpose. Each step closes with a check so you know it worked before moving on.

Confirm the APISIX version and plugin availability
Your platform engineer checks which APISIX release runs in each environment and whether the prometheus and ai-proxy-multi plugins are present and enabled in the gateway configuration. The apisix_llm_latency metric with its type label arrived in 3.18.0, so any older build will not show time to first token. Confirm the version against the APISIX downloads page and the changelog, and note that the downloads page does not mark releases as long-term support. You are done when the running version is recorded per environment and the plugin list includes both plugins.
Enable the Prometheus plugin and verify the endpoint
Enable the prometheus plugin in the gateway configuration so it exposes metrics at /apisix/prometheus/metrics on port 9091 by default. Restrict that port to your monitoring network, since the endpoint reveals traffic detail. From a host inside that network, request the endpoint with curl and confirm the response contains apisix_http_status, apisix_http_latency and apisix_bandwidth series. You are done when the endpoint returns metrics without errors and the expected metric names are present in the output.
Enable ai-proxy-multi summary logging
Turn on logging.summaries for the ai-proxy-multi plugin on the routes that carry LLM traffic. With this setting, each request logs the LLM model, duration, and request and response tokens. Apply the change through your usual configuration path, such as the Admin API or your configuration-as-code repository, so the setting survives a redeploy. You are done when a single test request produces one log entry that contains the model name and both token counts.
Send test requests to each provider
Send one short, non-sensitive test prompt to every provider and self-hosted model behind the route, using a known consumer key. Use a prompt you wrote for testing, not production data, because the logs will hold its size and possibly its content depending on your logging settings. Check the access log for each call and confirm the model field matches the provider that served it and that token fields are populated. You are done when every provider in the route has produced a log entry with model and token values, and any gap is traced to a provider configuration issue before going further.
Point Prometheus at the endpoint and check the type label
Add the APISIX metrics endpoint to your Prometheus scrape configuration, using the address and port reachable from the Prometheus host. Prometheus pulls metrics on its own schedule, so wait for at least one scrape cycle, then query apisix_llm_latency in the Prometheus expression browser. Confirm series exist for each type label value you expect, and that the labels identify the route and model. Record the exact label names, since dashboards and alert rules depend on them. You are done when the query returns series for the test traffic from every provider.
Build per-consumer views and cost queries
Group the logged token counts and the Prometheus series by consumer, so each team or customer can be matched to its usage. Confirm that the consumer identity appears in the fields you query before writing any panel, because the metric labels may not carry it by default and you may need to rely on log fields instead. Treat the results as operational estimates: Prometheus values reliability over precision and is not suited to exact per-request billing. You are done when one consumer's test traffic shows up under its own label or log filter and the totals match your test requests.
Set retention and confirm the dashboards load
Decide how long Prometheus keeps local data, since its default is 15 days and it can be configured by time or size, and decide where longer history goes if you need it through remote write to external storage. Remember that local Prometheus storage is not clustered or replicated, so plan backups or a second instance accordingly. Set the same retention intent for gateway access logs with your log platform team. Then load your Grafana dashboards against the live data and check that the type label panels render. You are done when retention values are written down, the dashboards show the test data, and the team agrees who owns each setting.

Turning token counts into per-consumer cost reports

A per-consumer cost report is a join, not a metric. The gateway gives you token counts and the model name for each request, and the team has to multiply those counts by the price schedule each provider charges for that model and then group the result by consumer. The price table is the part nobody else maintains. Providers change input, output and cached-token rates, retire models and add tiers, so the table needs a named owner, a source for every rate (the provider's published pricing page or the contract rate card), and a date on each row. Ask your finance and platform leads to agree who signs off a change before the first report is produced, and confirm which token types each provider bills separately so the schedule has a column for each.

Do not use Prometheus as the billing ledger. The Prometheus documentation says the system values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. A scraped counter can miss samples during restarts or scrape gaps, and the local time series store keeps only a default retention of 15 days unless you configure otherwise. Use the gateway data for showback and trends: which team or consumer is growing, which model is driving spend, and whether a quota change is working. Keep the exact per-request record in the access logs you retain, and reconcile the monthly totals against each provider's invoice. Expect a difference, and agree in advance how large a gap triggers an investigation.

Version the price table the same way you version gateway configuration. Store it in Git, give each change a effective date, and never edit a past row, so a report for last quarter can be reproduced with the rates that applied then. Record which table version produced each report. Before you rely on any report, confirm with the vendor how the token fields in your APISIX release are named and whether the logged values include cached or reasoning tokens, because a mismatch there changes the cost by a large margin.

Log retention and metric storage choices

Operational metrics such as apisix_llm_latency, apisix_http_status and apisix_http_latency answer near-term questions: is a provider slowing down, are errors rising, is time to first token drifting. The APISIX prometheus plugin exposes these at /apisix/prometheus/metrics on port 9091 by default, and a local Prometheus server scrapes them. Local Prometheus keeps data for 15 days by default, and that period can be set by time or by size.

Default window: The 15-day default suits trend analysis and incident review. Confirm whether operations needs a longer window before changing it.
Single-node storage: Local storage is not clustered or replicated. Losing the Prometheus node loses that history, so manage it like any other single-node database with its own backups and disk monitoring.
Growth drivers: Series count grows with label combinations such as route, model, consumer and the new type label. High-cardinality labels, especially per-consumer or per-request values, increase disk use and query cost.
Size limits: A size-based retention cap protects the disk, but it can drop older data sooner than the time window suggests. Set both limits deliberately.
Precision limits: Prometheus values reliability over precision, so these metrics are not suitable as the record for per-request billing.

Before relying on the default, ask the compliance owner whether operational metrics fall under any retention requirement at all, or whether only the summary logs do.

Summary logs record each LLM request at the gateway with its model, duration, and request and response tokens. With logging.summaries enabled in ai-proxy-multi, that record is written to the gateway's logs and can be shipped to a log platform. This is the most useful source for per-consumer cost reporting and for answering what a specific request used, because it is recorded per request rather than as an aggregate.

Platform choice: A log platform such as Loki or ClickHouse can hold these records. Loki indexes labels and stores chunks on object storage, while ClickHouse suits analytical queries across large event sets. Choose based on query needs and existing operations skills.
Storage growth: Volume grows with request count and with how much each record contains. Prompt or response content, if logged, multiplies it sharply and raises privacy scope.
Retention policy: Set separate retention for the summary record and for the full request content, if any. Content retention is usually the stricter question.
Access control: Token and cost records can identify teams and customers. Restrict who can query them and audit that access.
Backups: Logs used as evidence need tested backup and restore, not only replication.

Confirm with compliance owners whether the summary record is itself a regulated record, how long it must be retained, and who may delete it early.

Grafana dashboards and Alertmanager for LLM usage

The dashboard set should answer four questions from the gateway's metrics: how fast responses arrive, how quickly the first token appears, how many tokens each model and consumer uses, and how often requests fail over to another provider. Build panels for latency percentiles from apisix_http_latency, time to first token from apisix_llm_latency filtered on its type label, token volume by model and consumer from your logging pipeline, and error rates on fallback paths. Because the type label arrived in APISIX 3.18.0, any panel written against the older series shape will show empty or merged data after the upgrade, so confirm each query against a live 3.19.0 gateway before you publish the dashboard. Alerts should cover two cases: quota pressure, where token consumption approaches a budget for a team or consumer, and latency regressions, where time to first token drifts above the level your users accept for a sustained period.

Run Grafana with a MySQL or PostgreSQL database for production. The Grafana installation docs state that SQLite does not scale for production, and high availability requires two or more active-active Grafana servers behind a load balancer sharing that database, which the Grafana high availability guide describes. The same guide warns that alerting needs extra configuration to avoid duplicate notifications and that Grafana Live has limitations in high availability, so test notification delivery on a second node before relying on it. Provision data sources, dashboards and alert rules from YAML or JSON files so that the dashboard set lives in version control and can be rebuilt after a failure. Confirm which identity method your organization needs: the Grafana authentication docs list SAML, team sync and SCIM provisioning as Grafana Enterprise features, so an open-source deployment will not include them and you should plan a supported OAuth or OIDC path instead.

Prometheus pulls metrics from APISIX, evaluates rules and sends firing alerts to Alertmanager. Alertmanager can run as a cluster for high availability using its cluster flags, and Prometheus should be configured to point at every Alertmanager instance rather than at a load balancer in front of them. Decide who receives quota and latency alerts, how grouping and silences work for an incident affecting several models, and which alerts page someone versus post to a channel. Remember that Prometheus local storage is not clustered or replicated and defaults to 15 days of retention, so the budget and audit evidence you keep for LLM usage needs a long-term store through remote write, and the Prometheus storage documentation explains the trade-offs. Prometheus also warns it is not suitable where exact per-request billing is required, so treat dashboard figures as operational estimates and reconcile invoices from the provider or gateway logs.

Security and privacy of LLM usage data

Usage data from LLM traffic is sensitive even when it contains no prompt text: token counts, model names and consumer identifiers reveal how teams work and what they spend. The cards below cover the places this data leaks or lingers, and the control and owner for each.

Prompts and responses in logs

Request and response bodies must stay out of access logs, error logs and the summaries APISIX writes for AI plugins. Confirm which fields your logging plugin captures before enabling it in production, and test with a dummy prompt containing a recognizable string. The security engineering lead owns the log schema, and the platform team owns the logging configuration and its review.

Metrics endpoint exposure on port 9091

The prometheus plugin serves metrics on port 9091 by default, at /apisix/prometheus/metrics. Keep that listener off public networks, restrict it to the Prometheus servers with network policy or firewall rules, and confirm whether your deployment adds authentication in front of it. Label values such as consumer names should never be exposed to anyone outside the monitoring team. The platform team owns the listener and its network rules.

Dashboard hardening in Grafana

Grafana dashboards show per-model and per-consumer usage to anyone with access. Enable secure and SameSite cookies, a Content Security Policy, Trusted Types, hide the version from unauthenticated users, require authentication for the metrics endpoint, and enable domain verification against DNS rebinding. The observability team owns the Grafana configuration and reviews it after each upgrade.

Access to per-consumer cost data

Cost reports tie token volume to named teams or customers, which is commercially sensitive and can expose internal projects. Limit dashboard folders by role, use Grafana's team-based access where your edition supports it, and confirm whether SAML and team sync are available for your deployment. Finance and the platform owner should agree who may see per-consumer figures before the dashboards are shared.

Retention of personal data in logs

Log lines that carry consumer identifiers, client IP addresses or user identifiers are personal data under many privacy regimes. Set a retention period for each store, including the log pipeline, Prometheus and any long-term storage, and document the purpose. Prometheus keeps local data for 15 days by default, so confirm whether remote storage extends that and who approves it. The data protection officer and the platform team own the retention schedule.

Gateway secrets and upstream keys

Provider API keys used by the AI plugins must never appear in dashboards, metric labels or log output. Store them in your secrets manager, rotate them on a schedule you set, and check that exported configuration files do not contain them. The security team owns key rotation and the platform team owns the secret references in the gateway configuration.

Questions teams ask before monitoring LLM traffic

How much effort does LLM traffic monitoring through APISIX take?

The effort depends mostly on how many model providers and consumers you route, how many dashboards and alerts you need, and how long you keep logs and metrics. Each provider adds a route, credentials and a label set to validate. Each consumer adds a row to your cost views. Dashboards multiply quickly when finance, platform and product teams each want their own view. Retention rules add storage design and a review of your privacy obligations. Ask for a written scope that lists each of these counts before accepting any quote.

Can our own team set this up without a partner?

Yes, if you have people who already run APISIX, Prometheus and Grafana in production. The core work is configuration: enabling the prometheus plugin, enabling logging summaries on the AI plugins, writing PromQL queries and building dashboards. The harder parts are agreeing on a token and cost model with finance, handling the apisix_llm_latency type label across existing panels, and deciding what is safe to log. A partner is most useful when those decisions are open or when your team cannot take on the build alongside production duties.

What do we need to provide before the work starts?

Plan for access to the gateway configuration and Admin API, the Prometheus and Grafana instances you plan to use, and a test environment where changes can be checked. You will also need current provider price lists for each model you route, because cost reporting depends on them. Your retention policy for logs and metrics should be written down, including who may read prompt content. Finally, name the people who will approve dashboards and alert thresholds. Missing any of these usually delays the cost reporting part most.

How long does an LLM monitoring implementation take?

It is easier to plan in phases than in calendar time. The first phase confirms the APISIX version, plugin configuration and which fields the AI plugins will log. The second phase collects metrics and log records in a test environment and checks them against known requests. The third phase builds dashboards, cost reports and alerts with the teams that use them. The last phase covers retention settings, runbooks and handover. The length of each phase follows from the number of providers, consumers and dashboards in your written scope.

Can APISIX produce invoice-grade token and cost numbers?

Treat gateway numbers as operational estimates unless you verify them against provider invoices. Prometheus documents that it values reliability over precision and is not a good choice where 100% accuracy is required, such as per-request billing. Gateway logs can record model, duration and request and response tokens when logging summaries are enabled, which is useful for chargeback and trend analysis. For invoicing, reconcile those records with provider billing exports and agree the tolerance and the source of truth with finance before you publish figures.

Will upgrading to APISIX 3.18 or later break our existing dashboards?

It can. The APISIX changelog records that version 3.18.0 added Prometheus histograms for time to first token and token distribution, and that apisix_llm_latency gained a type label. Queries that match the metric without filtering on that label may return different series or double counts after the change. Review every panel and alert that uses apisix_llm_latency, test them against a staging gateway on the target version, and update the queries before rolling out. The changelog also describes a fail_mode for AI plugins, which changes how non-AI traffic is treated, so confirm that behavior in staging as well.

What retention and privacy controls should we plan for LLM logs?

Decide what you need to keep before you decide how long to keep it. Prometheus stores data locally with a default retention of 15 days, configurable by time or size, and local storage is not clustered or replicated. Longer history usually means remote write to external long-term storage, which carries its own cost and access rules. Prompt and response content is a separate question from token counts, and many organizations should log only metadata. Confirm your retention policy with your compliance and legal owners, then configure the gateway and storage to match it.

How DeployOpen scopes LLM monitoring on APISIX

DeployOpen starts every LLM monitoring engagement with a written scope, and the scope decides what gets built. Before any configuration changes, the document fixes the APISIX version you run, the LLM providers and self-hosted models whose traffic passes the gateway, the consumers (teams, applications or customers) you need to report on, the dashboards your operations and finance readers expect, and the retention periods for metrics and logs. Those five items are the inputs that drive the work. A gateway on an older release, or a consumer model that cannot be mapped to a key or header, changes the effort, so both are confirmed in the scope rather than discovered during the build. Deliverables and timelines are agreed in writing, and the quote follows from that scope.

Inside your infrastructure, the build then covers the gateway configuration that emits token and model data, the APISIX prometheus plugin scrape path, the log format that carries request and response tokens, and Grafana dashboards and data sources. A specific risk sits in the 3.18 change: apisix_llm_latency gained a type label, so any panel or alert that matches the old series without that label will go blank or misreport. DeployOpen tests that change against your own version and queries before cutover, and records which panels and rules were updated. The result is documented for your operations team, including what to check after upgrades. Ongoing support, upgrades and continuous testing are available as retainers, and any of these can be added to the written scope.

Monitoring LLM traffic through Apache APISIX