How token-based rate limiting caps LLM spend in APISIX
When several teams or customers call paid LLM APIs through one gateway, the provider invoice arrives as a single number with no owner attached. The ai-rate-limiting plugin in Apache APISIX lets you place a ceiling on that consumption before requests reach the provider. It counts tokens rather than requests, so it can limit total tokens, prompt tokens or completion tokens within a time window. You can apply the limit to each LLM instance, so one provider or model has its own budget, or to each consumer, so each team or customer has its own. When a consumer exhausts its quota, the plugin returns HTTP 503 by default, and you can confirm in the plugin documentation which rejection status and fields your version uses before you write client error handling.
Request-based limits such as a fixed number of calls per minute do not reflect LLM cost, because one request can carry a short question or a very long document and produce a very different token count. Token-based limits fit when the goal is to bound spend or capacity per team, product or customer, and when prompts vary widely in size. Request limits still help against bursts and abuse. Most teams combine both, with the request limit protecting the gateway and the token limit protecting the budget. Decide which windows match your chargeback or finance cycle, which consumers need separate budgets, and what should happen when a team reaches its limit mid-month.
Before you rely on these limits, confirm three things with your own tests: how your chosen models report token usage, which window types and reset behavior your APISIX version supports, and how the plugin behaves when the counter store is unavailable. The plugin's budget counts are only as accurate as the usage data returned by each provider, so the gateway enforces a budget but does not guarantee the cost figure on any invoice. Treat the gateway limit as a control you test, not a promise about the final bill.
Choosing what to count: total, prompt or completion tokens
Counting total tokens adds prompt and completion tokens into one budget for each window. It is the simplest limit to explain to a finance team, because a single number tracks how much model capacity a team or customer has used. The trade-off is that one number hides which direction the cost is growing, so a long prompt and a long answer can exhaust the same budget in different ways.
- Simple to communicate: one allowance per window maps easily to a monthly or daily usage cap.
- Blind to direction: a team that sends very large prompts and one that requests long answers can hit the same limit with different cost profiles.
- Good default for broad caps: it works well as an outer ceiling above more granular limits.
- Consistent across providers: it avoids differences in how each provider splits input and output charges.
- Dependent on reported usage: enforcement follows the token counts the upstream returns, so confirm how your chosen providers report usage.
Before you pick this option, check whether your provider's invoices weight input and output differently, because a single total may not match your actual spend.
Counting prompt tokens limits what callers send to the model, including the system prompt, conversation history and any retrieved context. It is useful when input size drives cost or when you need to control how much data leaves your network toward a provider. Because many applications resend long histories on every call, a prompt cap can slow growth in those applications without changing the length of answers.
- Controls data sent out: a prompt budget limits how much internal content reaches an external provider.
- Targets history growth: chat applications that resend context on every turn hit prompt limits first.
- Less control over answers: a caller can still receive long completions within the prompt allowance.
- Useful for retrieval systems: retrieved documents count toward the prompt budget, which makes it visible.
- Check provider accounting: confirm how cached or reused input is reported, since that affects what the gateway counts.
If you rely on a prompt limit to control data egress, test that the counted tokens match what the upstream actually processes.
Counting completion tokens limits the length of model output for a window. Output tokens often carry a different price than input tokens, so this limit fits teams whose spend is driven by generated text, such as summarization or drafting workloads. The gateway can only count completion tokens after the response is produced, so a single long answer may pass the check and push usage past the budget before the next request is refused.
- Matches output-heavy spend: it suits workloads where generated text dominates cost.
- Reactive by nature: the limit is applied as responses come back, so overshoot within one request is possible.
- Predictable for short prompts: applications with fixed, small inputs get a clear output ceiling.
- Pair with a total cap: a completion limit alone does not stop large prompts from consuming capacity.
- Confirm streaming behavior: check how streamed responses are counted and when the counter is updated.
Before relying on this limit for cost control, test how a long streamed answer updates the counter and what the caller sees when the window closes.
Scope decides whose usage is counted. A limit per LLM instance caps traffic to one model endpoint regardless of who calls it, which protects a provider account or a self-hosted GPU pool. A limit per consumer caps what one team, application or customer can use, which is the basis for chargeback and tenant fairness. Many designs use both: an instance limit protects shared capacity and a consumer limit allocates it.
- Instance scope protects capacity: it guards a provider quota or a self-hosted cluster from being exhausted by any caller.
- Consumer scope allocates budget: each team or customer receives a separate counter that maps to their share.
- Combined limits need ordering: a consumer can be within budget while the shared instance is exhausted, so test both paths.
- Identity must be reliable: consumer limits are only as accurate as the authentication that identifies each caller.
- Account for shared keys: a single provider key used by many applications makes instance counters hard to attribute.
Confirm how consumers are identified in your design, because a consumer limit built on an unreliable identity will misallocate spend.
The time window sets how often the counter resets. Short windows stop bursts quickly and limit the damage from a runaway client, but they can block legitimate peaks. Longer windows, such as a month, align with invoices and budgets but allow a client to spend its full allowance in one burst. The window and the counter storage together determine how precisely a budget is enforced, so the choice affects both finance reporting and operational stability.
- Short windows contain bursts: they reduce the cost of a misbehaving client but may reject normal peaks.
- Calendar windows match billing: a monthly reset lines up with invoice periods, which simplifies reconciliation.
- Layered windows are common: a short per-minute limit can sit beside a monthly cap.
- Reset timing needs agreement: confirm whether resets follow UTC, a fiscal calendar or a contract date.
- Counters must persist: shared counters in Redis keep windows consistent across gateway nodes, so plan that storage with the window.
Check the window against your provider invoice cycle and test how the counter behaves across a reset boundary before you go live.
Configuration options and when each fits
This table compares the main ai-rate-limiting configuration keys, the limit type, time window, scope, rejection status and counter backend, and shows when each setting fits a spending cap. Read each row as a decision: pick the value that matches how your teams and customers are billed, then confirm the exact key names and accepted values in the plugin documentation for your APISIX version.
| Configuration key | Options to weigh | When it fits | What to confirm |
|---|---|---|---|
| Limit type | Total tokens, prompt tokens or completion tokens | Total tokens fits a single shared budget. Prompt or completion limits fit cases where input size or output length drives your provider cost differently. | Confirm which token types your upstream reports in usage fields and how the plugin counts them. |
| Time window | Fixed window measured in seconds | Short windows protect against bursts from a single runaway job. Longer windows match monthly or team budgets but allow larger spikes inside the window. | Confirm the supported window unit and how the counter resets at the window boundary. |
| Scope | Per LLM instance or per consumer | Instance scope caps total spend on one provider or model. Consumer scope gives each team or customer its own budget, which suits chargeback and tiered plans. | Confirm that consumer scope requires an authenticated consumer and which auth plugin identifies it. |
| Rejected status code | Default 503 or a configured code | Keep the default if your clients already handle 503 retries. Set a 429 response when client libraries expect standard rate-limit semantics. | Confirm the default and whether the response body or headers can be customized. |
| Counter backend | redis, redis-cluster or redis-sentinel | A single redis instance fits a lab or one gateway node. Redis Cluster or Sentinel fits several gateway nodes that must share one count and survive a Redis failure. | Confirm the connection keys for each mode and the failover behavior of your Redis topology. |
| Redis topology for HA | Standalone, Sentinel with at least three Sentinel instances, or Cluster | Sentinel fits when you do not need data sharding. Cluster fits larger keyspaces, though it changes how keys are spread across nodes. | Confirm the Sentinel quorum and Redis version you will run, and the license terms of your Redis edition. |
| Fallback when quota runs out | Reject the request or shift traffic to another instance through ai-proxy-multi | Rejection fits strict budgets. Shifting traffic fits availability goals, but the second provider then carries spend you may not have planned for. | Confirm how the two plugins interact in your APISIX version and where the counter is read in the request flow. |
Which Redis version and license fit your organization
The shared Redis counter is the piece of this design that carries a license decision, not just a technical one. The version you pick determines which license terms apply, and the hosting model determines who operates the service. Work through these choices with your legal and procurement teams before the counter goes into production.
Redis 8 and later licensing
Redis 8.0 and later can be used under a license of your choice: RSALv2, SSPLv1 or GNU AGPLv3. The choice matters for APISIX because the rate limiter writes counters to Redis, and any modified Redis code offered as a network service may carry source-disclosure duties. Ask counsel which option fits your use and record the decision in your architecture file.
Redis 7.4 to 7.8 terms
Redis 7.4.x through 7.8.x remain under the dual RSALv2 or SSPLv1 license, and neither is OSI-approved. If your standard requires OSI-approved open source, these versions may not satisfy it even though the software is widely used. Confirm with your procurement or legal team whether a version in this range is acceptable for the counter that enforces your LLM budgets.
Redis 7.2 and earlier terms
Redis 7.2.x and earlier remain under the BSD-3-Clause license. Redis states that teams moving from 7.2 or earlier with unmodified artifacts see no license impact. This line is the simplest in license terms but will not receive the newer 8.x features, so check whether the security support you need is still available for the version you choose before standardizing on it.
Self-managed or vendor hosting
Redis Open Source is self-managed. Redis Software adds compliance, reliability and resiliency features for self-managed deployments, and Redis Cloud is a managed service on AWS, Google Cloud and Azure. Neither of the commercial options is open source. Decide whether your team will operate Sentinel and failover, or whether a vendor contract is the better fit for your data residency and support requirements.
Windows host limitation
Redis Open Source runs on Linux, macOS and the BSD systems, but on Windows only through Docker. If your APISIX nodes or Redis hosts run on Windows servers, plan a Linux container host for the Redis tier. Confirm the supported container image and version with your Redis build pipeline before you commit to a Windows-based design.
How to set up token quotas per consumer
This sequence takes a gateway from an inventory of LLM instances and consumers to a working per-consumer token budget backed by shared Redis counters, with fallback to other providers when a quota runs out. Each step ends with a check you can run before moving on.
Confirm LLM instances and consumers
Your team lists every LLM instance the gateway will front, including provider, model, endpoint and the credential each one uses, and marks which instances are approved for which teams or customers. Then you list each consumer that will receive a quota, such as an internal team, a product or an external customer, and the window each budget should cover. Confirm the limit types you need: total, prompt or completion tokens. Done when every consumer maps to a named budget owner and every instance has a documented provider and model name that you can test directly against the provider's API.
Deploy the Redis backend and Sentinel quorum
Your platform team deploys Redis in infrastructure you control, with replicas and at least three Sentinel instances, since the Redis Sentinel documentation describes a robust deployment as needing that minimum. Confirm the Redis version and license with your legal and procurement teams before deployment, because licensing differs across Redis releases. Done when Sentinel reports a healthy master, a replica promotes correctly after a controlled failover in a test environment, and the gateway hosts can reach the Sentinel and Redis ports.
Define consumers and API keys
Your team creates one APISIX consumer for each budget owner and attaches the authentication plugin your design uses, such as key-auth or jwt-auth, so the gateway can identify the caller before it counts tokens. Keep consumer names stable, because the per-consumer counter depends on them. Store issued keys in your secrets system rather than in route configuration. Done when a test request with each consumer's key is accepted, and a request without a key is rejected before it reaches any LLM route.
Configure ai-rate-limiting with limit type and window
Your gateway engineer adds the ai-rate-limiting plugin to the LLM route with the chosen token type, limit value and time window, and sets the key to consumer or instance depending on whether the budget belongs to a caller or to a model. Point the Redis settings at the Sentinel-managed deployment so counters are shared across nodes. Confirm the response status the plugin returns when a quota is exceeded, since the default is 503, and decide whether clients should see that code or a remapped one. Done when a configured test consumer receives successful responses until the limit is reached and then receives the rejection, with the remaining counts visible in the response headers you enabled.
Pair with ai-proxy-multi for traffic shifting
Your gateway engineer places ai-proxy-multi on the same route so that when one instance's quota is exhausted or it returns errors, traffic moves to an approved alternative. Define priorities and weights for each instance, choose the balancing algorithm, and set max_retries. Each instance needs a unique name in the configuration, which the plugin requires. Confirm that a fallback provider is also covered by the consumer's policy, since shifting traffic to an unapproved model can break your data handling commitments. Done when a test that exhausts the primary instance's budget results in a successful response served by the next instance, and the logs show which instance served it.
Test quota exhaustion across two gateway nodes
Your QA or security engineer runs two APISIX nodes against the same Redis backend and sends requests for one consumer alternately to each node until the limit is hit. The aim is to show that both nodes count against one shared total rather than each keeping its own tally. Record the request number at which rejection starts on each node and confirm both match the configured limit. Also confirm that a second consumer's budget is unaffected during the test. Done when the rejection point is identical on both nodes and the second consumer continues to receive responses.
Test counter sharing after a Redis failover
Your platform engineer triggers a controlled Sentinel failover in the test environment while requests continue to flow through both gateway nodes. The check is that counters survive the promotion of a replica and that the consumer's remaining budget stays consistent afterward. Confirm with your Redis and gateway teams how the plugin behaves while the backend is unavailable, and document the fail behavior your organization accepts. Done when the consumer's remaining budget after failover matches the pre-failover count within the behavior you documented, and the failover event is captured in your monitoring.
Shifting traffic to another provider when a quota runs out
A quota rejection and a provider failover are different events, and the gateway treats them differently. When ai-rate-limiting finds that a consumer or LLM instance has used its token allowance for the window, it rejects the request itself. By default that rejection returns HTTP 503, and no upstream provider is contacted. When ai-proxy-multi fails over, it is reacting to a response from an upstream, such as a 429 or 5xx, and it retries another instance under its max_retries setting. A quota rejection is therefore not a signal that a provider is unhealthy, and a failover path will not help a client whose own budget is spent. Teams need to decide which of the two outcomes they want for each consumer before they write any plugin configuration.
Teams usually combine the two plugins in one of two ways. The first places ai-rate-limiting on the consumer or route and uses ai-proxy-multi priorities to send overflow to a second instance, such as a cheaper model or a second account with another provider. The second keeps the budget check on the primary instance and gives a separate, cheaper instance its own route for consumers who have exhausted the main quota. Either design changes who pays for overflow traffic, so finance and the owners of each provider account should agree on the spillover rules first. Clients also need a defined behavior for the 503 case, such as backing off and retrying after the window resets, or switching to a lower-cost route in their own code. Without that contract, a quota rejection will look to the client like an outage.
The exact combination must be validated in a test environment before production. Confirm the plugin ordering, the behavior of priorities when the primary instance rejects a request, whether the rejection status can be changed for a given route, and how shared Redis counters behave during a failover between instances. Run a test that exhausts a quota and checks that overflow reaches the intended instance, that the client receives the status it expects, and that usage is logged against the right consumer. Check the current plugin documentation for your APISIX version, since the behavior of these plugins has changed between releases, and APISIX ai-rate-limiting plugin documentation is the starting point for that check.
Security and operational risks to plan for
Token quotas only hold if the counters behind them are reliable, protected and auditable. The cards below cover the shared Redis dependency, the credentials that guard it, the consumer keys that identify callers, and the logs that prove spend. Each one names a control to put in place before the quota is trusted for budget decisions.
Redis as a shared dependency
Every node in the APISIX cluster writes to the same counters, so a Redis outage affects every LLM route that uses ai-rate-limiting. Decide in advance whether requests fail closed or open when the counter store is unreachable, and document that choice for finance and security. Confirm the failure behavior of your chosen Redis topology with the APISIX plugin docs and test it in staging by stopping the Redis primary.
Redis credentials and network access
The Redis instance stores spend counters that decide whether callers are blocked, so anyone who can write to it can reset a budget. Restrict Redis to the APISIX network segment, require authentication, and keep the password in your secrets store rather than in the plugin configuration committed to Git. Confirm with your Redis deployment team which ACL users APISIX needs and whether TLS is enabled on the connection.
Consumer key handling and rotation
Per-consumer budgets depend on the gateway identifying each team or customer correctly, which usually means an API key issued through a consumer. Set a rotation schedule and a revocation procedure that removes the consumer key without deleting its counter history. Check that a rotated key keeps the same consumer identity, or your budget reporting will split one customer into two.
Quota bypass through unidentified callers
A request that reaches the LLM route without a consumer identity may fall outside the per-consumer limit, or be counted against a shared bucket that blocks everyone. Make authentication mandatory on every LLM route, and test that a request without a key is rejected before it reaches the model. Review routes for any path that proxies to a provider without the limiter attached.
Token usage logs for audit
Finance and auditors will ask how a figure in the spend report was produced, so the token counts behind each decision need to be recorded. Log the consumer, model, prompt and completion token counts, and the response status for every limited request, and set a retention period that matches your finance and compliance policy. Confirm with your logging team where those records land and who can read them.
Counter state during failover
When Sentinel promotes a replica, writes not yet replicated from the old primary can be lost, which may let a consumer exceed its window by the amount of the lost writes. Decide whether that small overrun is acceptable for your budget, or add headroom to the limit. Test a failover in staging and compare the counter values before and after to measure the effect on your own traffic.
What teams ask before running LLM token limits in-house
What drives the implementation effort for LLM token limits in APISIX?
The effort follows three things: how many consumers and budgets you need to enforce, how the shared counters are laid out in Redis, and how many LLM providers sit behind the gateway. A single team with one model provider and one budget is a small configuration task. Per-customer budgets across several providers add consumer design, counter keys, provider-specific token accounting and failover testing. Redis topology matters because the ai-rate-limiting plugin can share counters through redis, redis-cluster or redis-sentinel, and each option brings different operational duties. Ask for a written scope that lists each of these three counts before you compare quotes.
Can one engineer build LLM token limits in APISIX alone?
One experienced engineer can build a narrow version, such as a single gateway with one provider and one Redis instance. The harder parts are rarely the plugin configuration itself. They include deciding how budgets map to consumers, testing what happens when a quota runs out, confirming how each provider reports token usage, and making sure the Redis counters survive a node or failover event. Teams that own the gateway and Redis, and have someone who can review the Lua or configuration changes, usually cover the build internally. Teams without that depth often benefit from a second pair of hands for design review and verification.
What access and inputs does the client need to provide?
The work depends on inputs that only your organization can supply. Typically this includes:
- Access to the APISIX Admin API or the configuration repository for the gateway
- A Redis deployment the gateway can reach, with credentials and network rules agreed in advance
- The consumer list, including teams or customers that need budgets and how they are identified
- Accounts and API keys for each LLM provider in scope, with usage and billing visibility
- Test traffic or a non-production environment where budget exhaustion can be exercised safely
Missing any of these usually moves the start date, so confirm them before the scope is signed.
How long does validation take for token limits?
Validation time depends on scope rather than a fixed number. A single-instance limit tested with one consumer is short to verify. Validation grows when you add per-consumer budgets, several time windows, multiple providers with different token reporting, and a failover path that moves traffic when a quota runs out. Testing should cover quota exhaustion, counter behavior across gateway nodes, Redis failover, and how the gateway responds to clients. DeployOpen does not publish durations, so ask for the test cases and their order in the written scope and check that each maps to a scenario you care about.
What recurs after token limits go live?
Several tasks recur after launch. Budgets need tuning as usage patterns change, and teams often adjust time windows and limits in the first months. API keys for consumers and providers rotate on your schedule and need the gateway configuration updated. Upgrades to APISIX and to Redis are periodic work, and Redis licensing and release lines should be checked at each upgrade because they have changed. Someone must also review counter and cost reports so budget breaches are noticed. Ask whether the quote covers these as a retainer, as separate work, or as handover to your team.
Which Redis licensing questions should we check before choosing a topology?
Redis licensing changed across releases, so confirm the version and license before you standardize on it. According to the Redis licenses page, Redis 8.0 and later can be used under a license of your choice among RSALv2, SSPLv1 and AGPLv3. Redis 7.4.x to 7.8.x remain under dual RSALv2 or SSPLv1, which are not OSI-approved, while Redis 7.2.x and earlier remain under BSD-3-Clause. Your legal team should review which license fits your use, especially if you modify Redis and offer it as a network service. The licenses FAQ on the same site explains the obligations in more detail.
How is the cost of token limits quoted?
DeployOpen does not publish a price list. A quote follows from a written scope that names the consumers, providers, Redis topology, test environments, documentation and handover expectations. Most engagements are fixed-scope, so deliverables and timelines are agreed in writing before work starts. Ask the quote to separate one-time build and validation from any recurring retainer for support, upgrades or continuous testing. Compare quotes on the same scope items rather than on headline totals, and confirm what is excluded, such as provider contract changes or Redis hosting.
How DeployOpen scopes and delivers this in your infrastructure
DeployOpen plans, implements and supports this work inside your own infrastructure, and the scope is agreed in writing before any build starts. DeployOpen is a Chainscore Labs company. For an APISIX token budget project, the written scope should name the APISIX version you will run, the deployment mode (traditional, decoupled or standalone), the cloud accounts, data centers or air-gapped networks that host the gateway and its Redis counters, and each LLM provider or self-hosted model in the route set. It should also list which consumers and teams get budgets, which limits apply (total, prompt or completion tokens, per instance or per consumer), what happens when a quota runs out, and what evidence you will receive, such as the ai-rate-limiting configuration, test results and runbooks. Ask for these items to appear as deliverables with acceptance criteria, so the quote reflects the actual work.
Pricing follows from that written scope. DeployOpen does not publish a price list, so a quote depends on the number of routes and consumers, how many providers and fallback paths need testing, whether Redis is new or already operated by your team, and how much of the Redis topology (single node, Sentinel with at least three Sentinel instances, or Redis Cluster) you need to design and verify. The one-time costs are design, configuration, integration with your identity and logging systems, and acceptance testing. Recurring costs depend on whether you need ongoing support, upgrades and continuous testing, which DeployOpen offers as retainers. When you request a quote, ask which items are fixed, which are estimates tied to a named assumption, and what changes trigger a scope revision.
After delivery, the gateway, Redis and budget settings still need an owner. Confirm with your own team who approves changes to quota values, who reviews the 503 responses that appear when a budget is exhausted, and how upgrades to APISIX and Redis will be tested before rollout. Before signing, confirm the licensing position of the Redis version you plan to run, since the Redis licenses page describes different terms for 7.4.x to 7.8.x and for 8.0 and later. A retainer can cover these operational tasks, while the scope document should record who is responsible for each one.
