How LLM load balancing and failover with ai-proxy-multi keeps AI traffic flowing
A single LLM provider is a single point of failure, and provider rate limits (HTTP 429) or upstream 5xx errors surface to your users as broken features. The ai-proxy-multi plugin in Apache APISIX addresses this by spreading requests across several LLM instances with weighted roundrobin, chash or semantic balancing, and priorities, then retrying on another instance when one returns a rate limit or 5xx response, up to the number set in max_retries. Teams usually need it once an AI feature is in production, serves more than one team, or depends on a provider whose quotas or regional availability they do not control. The plugin supports OpenAI, DeepSeek, Azure OpenAI, Anthropic, Gemini, Vertex AI, Amazon Bedrock and OpenAI-compatible APIs, so a self-hosted model can sit in the same pool as a cloud provider.
Before you build the pool, confirm the version requirement in the plugin documentation and the apache/apisix changelog. Starting with APISIX 3.19, each instance in the instances list needs a unique name, so a configuration that worked on an earlier release may be rejected on upgrade. Plan the naming scheme now, because logs, metrics and fallback behavior all refer to instances by name, and a duplicate name breaks the mapping you rely on during an incident.
Decide three things before writing configuration: which providers belong in the pool, how traffic should split between them, and what counts as a failure worth retrying. Keep weights and priorities in a reviewed change, and test failover by forcing errors against a non-production route. Confirm with your provider contracts which models are interchangeable, because a fallback that answers with a weaker or differently priced model changes cost, quality and data handling for that request.
Which balancing algorithm fits your traffic
ai-proxy-multi offers four ways to decide which LLM instance receives a request: weighted round robin, consistent hashing, semantic selection and priorities. They are alternatives you choose between per route, and each suits a different traffic pattern. Pick the one that matches what you need to control, then confirm the exact configuration keys against the plugin documentation.
Weighted round robin
Spreads requests across instances in proportion to a weight you set for each one. It suits a steady mix of similar prompts where you want predictable split, for example sending most traffic to a negotiated provider account and a smaller share to a second account. Check how weights interact with failed instances, because a heavily weighted instance that starts returning errors still receives its share until fallback takes over. Confirm the weight and retry settings in the plugin reference.
Consistent hashing (chash)
Routes requests with the same hash key to the same instance, based on a value such as a consumer or tenant identifier. It suits workloads where a customer's requests should stay on one provider, for example to keep per-tenant context or quota accounting in one place. Hashing spreads load only as evenly as your keys do, so a few large tenants can concentrate traffic on one instance. Review the key choice with your traffic data before you rely on it.
Semantic selection
Chooses an instance based on the content of the request rather than a fixed rule. It can suit mixed workloads where some prompts go to a cheaper model and others to a stronger one. Because the selection logic depends on how the plugin classifies requests, treat it as a design decision to test, not a default. Ask how semantic selection is configured in your APISIX version, what it costs in added latency, and how it behaves on prompts it cannot classify.
Priorities
Orders instances so that lower-priority ones are used only when higher-priority ones are unavailable or fail. It suits a primary provider with a contracted backup, where you want the backup to stay idle until needed. Priorities define the failover order, so pair them with fallback behavior on 429 and 5xx responses and set max_retries deliberately. Test the failover path in staging, including the case where the backup also rate limits.
Configuration options for ai-proxy-multi and when each fits
This table maps the main ai-proxy-multi settings to the situations where each one matters, so you can see which choices shape traffic spread, failover and cost. Key names and defaults should be checked against the plugin reference for your APISIX version before you write configuration. Read each row as a design question: what you need the gateway to do, and which setting answers it.
| Setting | When it fits | What to confirm in the plugin reference |
|---|---|---|
| Instances with names and weights | Splitting traffic across several provider accounts or models in fixed proportions, such as sending most requests to a primary deployment and a share to a second one. | That each instance has a unique name (required in 3.19), how weights are interpreted, and whether weights apply within a priority group. |
| Algorithm (roundrobin, chash or semantic) | Roundrobin suits even spread across equivalent instances. Chash suits keeping one consumer or session on the same instance. Semantic suits routing by request meaning. | The exact hash key used by chash, and the configuration and dependencies the semantic option needs before you enable it. |
| Priorities | Keeping a preferred provider in use and sending traffic to lower-priority instances only when higher ones are unavailable or failing. | How priority interacts with weights and with fallback, and which failure conditions move traffic to the next priority. |
| max_retries | Limiting how many alternative instances a single request may try after a failed attempt. Lower values protect latency, higher values protect completion rates. | The default value, whether retries apply to non-idempotent LLM calls, and the total time budget a retried request can consume. |
| Fallback conditions (429, 5xx, rate limits) | Moving requests off an instance that is throttled or returning server errors. Useful when provider quotas are the most common failure. | Exactly which status codes and conditions trigger fallback in your version, and whether 4xx errors other than 429 are passed through unchanged. |
| Health or timeout settings | Tuning how quickly the gateway stops sending traffic to a slow or unresponsive provider, important for streaming responses and long completions. | Which timeout keys exist, their units and defaults, and whether active health checks are supported for each provider type. |
| Provider auth | Storing one API key per instance so each provider account can be rotated, budgeted and audited separately. Needed when mixing cloud and self-hosted endpoints. | How keys are referenced (plain values or environment variables), whether Azure OpenAI, Vertex AI and Bedrock use different auth fields, and where secrets are stored. |
| Override model and endpoint | Mapping one client-facing model name to different upstream model names or OpenAI-compatible base URLs, such as a self-hosted vLLM server. | Which provider types accept an endpoint override and how model names are passed through for each supported provider. |
How fallback on 429 and 5xx responses behaves in practice
When an instance in ai-proxy-multi returns a rate limit (429) or a 5xx error, the plugin treats it as a failed attempt and moves the request to another instance in the balancing group, as long as the retry budget allows. The failing instance is skipped for that request, and the next attempt is chosen by the same algorithm you configured, whether weighted round-robin, chash or semantic. The plugin's max_retries setting caps how many extra attempts a single client request can make. Each attempt is a full upstream call, so a request that fails twice before succeeding is billed by the provider up to three times. Confirm in the plugin documentation exactly which status codes and timeouts count as retryable in your version, because that list decides when a retry happens at all.
Retries multiply load at the moment a provider is already struggling. If a provider starts returning 429 for most requests and every client request retries against it first, the gateway adds traffic to the remaining instances just as they are absorbing a shift. Keep max_retries low, and place the failing provider at a lower priority so most traffic reaches a healthy target first. Plan for the cost side too: during an incident, retried calls can add token spend without producing a usable answer if the first attempt was accepted but timed out on the client side. Ask your provider how they bill requests that return an error after generating tokens.
To test this, configure a deliberately failing instance, such as a key with a bad secret or a test upstream that returns 429 or 503, and send a known volume of requests through the route. Measure how many requests succeed on fallback, how many upstream calls each client request produces, the added latency per retry, and whether the failing instance is dropped from rotation as expected. Compare the provider-side call counts with your gateway logs to confirm the billed volume. Repeat the test with a slow instance that times out rather than errors, since that path often generates the most duplicate calls. Record the results before you set the retry limit for production.
Setting up ai-proxy-multi with unique instance names
This sequence takes an APISIX gateway from a confirmed 3.19 install to a route that spreads LLM requests across named provider instances, retries on failure and keeps serving when one provider is blocked. Each step ends with a check you can run before moving on.
Confirm APISIX 3.19 in your environment
Your platform or infrastructure team confirms that the gateway runs APISIX 3.19.0 and that the admin API responds in the environment where the route will live. Version 3.19 requires unique instance names for ai-proxy-multi, so older configurations need review before they are reused. Check the version reported by the running instance against the release notes in the APISIX changelog, and record it in the change ticket. The step is done when the version is recorded and the admin API returns a healthy response.
Create the LLM route and consumers
The gateway team creates one route for the LLM endpoint that clients will call, and one consumer for each application or team that will use it. Choose the authentication plugin for consumers according to your access model, as covered in the authentication pages. Keep the route simple at this stage and attach the ai-proxy-multi plugin in a later step. The check is a request with a valid consumer credential that reaches the route, and a request without one that is rejected.
Store provider credentials outside the configuration
The security or platform team places each provider API key in the secret store or environment that your deployment uses, then references it from the plugin configuration. Confirm with the APISIX documentation for your deployment mode how secret references are resolved, and whether the admin API returns them in plain text. The goal is that no raw key appears in route definitions, version control or change tickets. The check is an export of the route configuration that contains references only, with no key values.
Define named instances for each provider
The gateway team adds one entry per provider account or deployment to the ai-proxy-multi instances list, such as separate entries for a primary cloud provider and a backup provider. Each entry needs a name that is unique across the whole list, because version 3.19 rejects duplicates. Use names that describe the provider and purpose, so logs and metrics stay readable. The check is a saved configuration that accepts every instance, and a deliberate duplicate name that the gateway refuses.
Set weights and priorities for each instance
The gateway team chooses the balancing approach for the route, such as weighted round robin for proportional traffic, or priorities to send traffic to a primary instance first. Set the weight and priority fields for each named instance, and confirm in the plugin documentation which fields apply to the chosen algorithm. Hash-based or semantic balancing needs its own inputs, so decide those before go-live. The check is a batch of test requests whose distribution across instance names matches the intended split in the logs.
Configure retry limits for fallback
The gateway team sets the max_retries value so that a request that receives a rate limit, a 429 or a 5xx response moves to another instance, with a ceiling on how many attempts are made. A high retry count can multiply token spend and latency during an outage, so agree the limit with the cost owner before setting it. Confirm in the plugin documentation which status codes trigger fallback in your version. The check is a saved configuration and a test in which a forced error produces a logged retry to the next instance.
Test failover by blocking one provider
On a non-production route, the platform team blocks traffic from one provider, for example by revoking its key or denying egress to its endpoint, and then sends a batch of test requests. The expected result is that requests complete through the remaining instances, with the failed instance visible in the logs as a fallback. Record the error rate and latency during the test, then restore the provider and confirm traffic returns to the normal split. The step is done when the test results and the restored configuration are filed with the change record.
Securing provider keys and prompt data during failover
Failover multiplies the places where provider credentials and prompt content can leak, because every extra instance is another key and another upstream that sees user data. The cards below cover where keys live, how they are scoped, where prompts travel, what logs keep, and who can change the instance list.
Provider key storage
Each entry in the ai-proxy-multi instances list carries its own provider API key. Keep those values out of route JSON exported to Git, and inject them from a secrets store at deploy time. Ask whether your APISIX deployment mode and configuration workflow can reference secrets rather than literal values, and confirm how the Admin API returns stored keys when someone reads the route back.
Least-privilege keys per instance
Issue a separate key for each provider instance, scoped to the project, model set and spending limit that instance needs. A leaked key for a backup provider then exposes only that account. Confirm with each provider which permissions and project scopes its keys support, and plan rotation so one instance can be re-keyed without touching the others.
Keeping prompts on your network path
Failover sends the same prompt to whichever provider is healthy, so every provider in the pool becomes a destination for user data. List the providers approved for each data class, and keep self-hosted model endpoints inside your network so prompts reach them without crossing the public internet. Confirm that the data terms of each provider match your approved list before adding it as a fallback.
Logs that exclude prompt content
Access logs, error logs and gateway plugin logs can capture request bodies during troubleshooting, which copies prompts and completions into a second store. Keep body logging off for LLM routes, log model name, instance, status and token counts instead, and set retention on the log store to match your policy. Test a forced 429 and a forced 5xx to confirm that failover logs contain no prompt text.
Admin API access for instance changes
Adding, reweighting or removing a provider instance is a change to where your prompts go, so treat the Admin API as a privileged control. Restrict it to named operators and a change pipeline, use distinct admin keys, and keep the Admin API off public listeners. Review who can edit priorities and fallback settings, and record each change with an approver.
Questions teams ask before building LLM failover on APISIX
How many LLM providers does an APISIX failover setup usually need?
There is no fixed number. Most teams start with one primary provider and one or two fallbacks, then add more only where a contract, region or model need justifies it. The ai-proxy-multi plugin supports OpenAI, DeepSeek, Azure OpenAI, Anthropic, Gemini, Vertex AI, Amazon Bedrock and OpenAI-compatible APIs, so the choice of providers is rarely limited by the gateway. Each provider you add still needs its own account, credentials, model mapping, health behavior and test cases. Cost rises with each one, so decide the provider list from your workload before the build starts. The plugin documentation lists the supported provider types and options.
Which factors drive the cost and effort of an APISIX failover build?
Five factors shape most of the effort:
- Number of providers and models, since each adds configuration, credentials and tests.
- Authentication models, such as API keys, cloud identity roles and private endpoints, which differ by provider.
- Testing depth, from a single happy-path check to repeated tests of 429 and 5xx responses under load.
- Retry policy, including
max_retries, priority order, weights and what counts as a failure worth retrying. - Observability, meaning how much logging, metrics and alerting you need to see which provider served each request.
A quote should show each factor separately, with the environments to be covered. One-time work includes design, build and testing. Recurring work covers key rotation, provider changes and upgrades.
Can an in-house platform team build LLM failover on APISIX alone?
Often yes, if the team already runs APISIX in production and is comfortable with the Admin API, plugin configuration and its own deployment pipeline. The harder parts are usually outside the gateway: negotiating provider access, deciding what a failed request should do for each application, and proving that retries do not duplicate costly or sensitive calls. Teams without recent APISIX experience usually need time to learn the plugin's behavior under error conditions. Confirm with your own engineers whether they can cover design, testing and on-call handover within your schedule. DeployOpen can take on the build as a fixed-scope engagement in your infrastructure, or work alongside your team.
What do we need to provide before a failover project starts?
Prepare four inputs before scoping:
- Provider accounts, with API keys or cloud identities, billing owners and agreed rate limits.
- Traffic data, including which models and applications are used, typical request and token sizes, peak periods and current error rates.
- A test window in a non-production environment, plus a production window for controlled cutover.
- Named decision makers who approve routing priorities, fallback behavior and which data may leave your network.
Without the traffic data, weights and priorities are guesses. Without a test window, 429 and 5xx behavior is only tested after a real outage. Ask the partner to list these inputs in the written scope.
How do APISIX upgrades affect an ai-proxy-multi configuration?
Upgrades can change plugin fields, defaults and required settings, so review the release notes before moving any failover route. The topic requirement to use unique instance names is an example of a change you must act on: APISIX 3.19.0, released on September 28, 2026, requires unique instance names in this plugin. Check the project's CHANGELOG for each version between your current release and the target, then confirm the exact fields that change in your version. Plan to rename or validate instances, retest fallback paths and roll back the gateway if a route fails to load.
Will a new APISIX release break existing failover routes?
It can, which is why failover routes should be part of every upgrade test. Run the same scenarios on the new release in a staging environment: a normal request, a forced 429, a forced 5xx, a provider timeout and a priority change. Compare which provider served each request and whether retries stayed within max_retries. Keep the previous gateway package ready for rollback, and do not upgrade during peak traffic. Ask any partner to show the test cases and rollback steps in writing before the work starts.
How long does it take to put LLM failover into production?
No duration can be stated in advance because it depends on the factors covered above: number of providers, authentication models, the depth of testing and how quickly your team can supply accounts and traffic data. Delays most often come from provider access approvals and from agreeing on routing and retry rules. Ask for a timeline in the written scope that lists the inputs you must provide, the test windows and the cutover date. A fixed-scope agreement should tie each milestone to a deliverable you can check.
Is LLM failover a one-off build or an ongoing service?
The initial build is usually a fixed-scope project with agreed deliverables and timelines. Changes that follow, such as adding a provider, updating credentials, handling a new APISIX release or responding to a provider's API change, are recurring work. DeployOpen offers ongoing support, upgrades and continuous testing as retainers, so you can choose between a one-time build, a retainer or an internal team. Ask for the scope of support in writing, including response expectations, before deciding which model fits your team.
When a single provider is enough and when balancing pays off
A single provider is usually enough when traffic is modest, one model meets the quality bar, and an outage of a few hours is tolerable for the feature. In that case, ai-proxy-multi adds a configuration layer without adding much protection, because there is no second instance to fail over to. Balancing starts to pay off when one of three conditions holds: request volume regularly hits the provider's rate limits or quota, a business process cannot stop during a provider incident, or different workloads need different models, regions or price points. Before deciding, confirm with each provider the rate limits, quota scope (per key, per organization or per region), and data residency terms that apply to your account, since these determine whether a second instance in the same region actually adds capacity.
Cost-based routing is the most common reason to split traffic. Using weight values on instances, or priorities that send routine requests to a cheaper model and reserve a stronger one for harder tasks, lets you move spend without changing client code. Self-hosted models behind an OpenAI-compatible endpoint can sit in the same instance list as cloud APIs, which keeps sensitive prompts on infrastructure you run while overflow goes to a cloud provider. Treat such a design as a decision about data handling and cost together, and confirm with your security team which request types are allowed to leave your network.
Be clear about where gateway failover stops. ai-proxy-multi can retry another instance after a rate limit, 429 or 5xx response, but it cannot make two models behave the same. A fallback model may return different answers, support different context lengths, handle tool calls differently or apply different safety behavior, so test each fallback pair against your own prompts before it goes into a priority chain. Provider-side quotas also remain your responsibility: a gateway that retries into an exhausted quota only adds latency. Plan the quota and contract limits with each provider, and use the gateway to route around them, not to hide them.
How DeployOpen scopes and delivers LLM failover on APISIX
DeployOpen plans, builds and hands over LLM load balancing and failover on self-hosted Apache APISIX, inside infrastructure you control. Each engagement starts from a written scope that names your providers, routing rules, failure tests and the teams who will operate the gateway afterward.
How we scope
- We inventory the LLM providers and self-hosted models your applications already call, with their API types and credential ownership.
- We agree in writing which balancing algorithm, priority tiers and fallback conditions apply to each route before build starts.
- Engagements are fixed-scope, so deliverables, test cases and timelines are agreed in writing before work begins.
How we build
- We configure ai-proxy-multi instances with unique names, weights or hash keys, and priorities, then reference them from routes.
- We set max_retries and fallback behavior for 429 and 5xx responses, and test each failure path with controlled upstream errors.
- We keep provider keys in your secrets store and deploy gateway configuration through your own change process.
How we hand over
- You receive the route and instance configuration, a list of tested failure scenarios and the expected result for each.
- We document the runbook for adding a provider, rotating a key and reading which instance served a request.
- Handover sessions walk your platform and security engineers through the configuration and the test evidence.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer, covering changes to providers and APISIX releases.
- We recheck fallback behavior after each APISIX upgrade, since plugin options can change between releases.
- Ask us to confirm a quote covers provider onboarding, retesting and the support hours you need before signing.
