Why LiteLLM routing decides whether model access survives a bad day
This page explains how to choose a router strategy, set fallback order, retries and cooldowns, and spread traffic across provider accounts and self-hosted models. Those settings decide what happens when a provider rate-limits a key, a region degrades or a private model server runs out of capacity. For most gateways, the failure is not a rare event. It is a normal Tuesday afternoon, and the question is whether your applications keep working or pass the error straight to users. Picture a gateway serving a customer support assistant, an internal search tool and a document summarizer. The three applications share a handful of provider accounts plus one self-hosted model. When the primary account for a model group starts returning rate-limit errors, every application that depends on it is affected at once unless the router moves traffic elsewhere. In LiteLLM, that behavior is configured per model group. Each group lists the deployments that can serve a model name, and the router applies its strategy, retries, cooldowns and priority ordering within that group. A deployment that is not in the group cannot take over, so the group membership is the first design decision to review.
Start with the routing strategy. LiteLLM supports simple-shuffle, which the documentation recommends as the default, along with usage-based, latency-based, least-busy and cost-based routing. Simple-shuffle spreads requests across deployments in the group, which suits accounts with similar limits and prices. Latency-based and least-busy routing need the router to hold state about recent response times and in-flight requests, and cost-based routing depends on accurate pricing for each deployment. Confirm which of these your provider accounts and models can support, and whether your team can explain to an auditor why traffic moved from one account to another during an incident. Retries, cooldowns and fallback order interact more than most teams expect. A retry on the same deployment can hide a short blip but slow every request during a real outage. A cooldown removes a deployment from rotation after repeated failures, and the duration and failure threshold determine how quickly traffic recovers when the provider stabilizes. Priority ordering defines which deployment or model group is tried next. Write these values down per model group, including what should happen when every deployment in the group is cooling down, and confirm the exact parameter names and behavior against the routing documentation for the version you deploy.
The configuration becomes a shared-state problem once more than one proxy instance runs. Rate limit counts and router state must be visible to every instance, or each pod makes routing decisions from its own partial view and limits are exceeded in aggregate. LiteLLM's production guidance requires Redis 7.0 or newer in that case, and PostgreSQL remains the primary database for keys, budgets and spend. Ask your platform team who operates Redis, how it is made highly available, and what happens to routing if it becomes unreachable. Decide whether the gateway should fail closed or fail to local state, because that choice changes both your cost exposure and your risk during a Redis outage. The hard part is testing failure paths, not the happy path. A routing configuration that passes a smoke test with all deployments healthy tells you little. Plan tests that disable one provider account, return rate-limit errors from a deployment, stop the self-hosted model server, and cut Redis, then record which deployment served each request, how long users waited and whether budgets and limits still held. Use a non-production environment with the same model group layout, and keep the results as evidence for the operations team and for any compliance review that asks how availability is managed.
Confirm three things before you commit to a design. First, which provider accounts and private models are in each group, and whether their limits and data terms allow them to share traffic. Second, the strategy, retry, cooldown and fallback values for each group, with an owner who can change them during an incident. Third, a written test plan that covers every failure path your applications depend on, with a pass or fail result recorded for each. Your implementation partner or internal team should be able to show these artifacts, and the quote for the work should name them explicitly so the scope is clear before build starts.
Which router settings and extension points carry the behavior
LiteLLM routing behavior comes from a small set of configuration points: the model list, router settings, failover order, shared state and key scoping. Each one changes what a request can reach and what happens when a provider fails, so they should be reviewed together before traffic moves to the gateway.
Model list and model groups
The model list maps each public model name to one or more deployments, such as separate provider accounts or self-hosted endpoints. Deployments that share a model name form a model group, and the router balances and fails over inside that group. Check that every deployment in a group is interchangeable for the application, including context length, output behavior and data location, because the router will not check this for you.
Strategy and retry settings
The router strategy decides how a deployment is picked from a group: simple-shuffle (the documented recommendation), usage-based, latency-based, least-busy or cost-based. Retry counts and cooldown rules determine how quickly a failing deployment is removed from rotation and when it returns. Test the chosen strategy with realistic request sizes and provider error responses before relying on it, since the behavior under rate limits differs from the behavior under timeouts.
Priority ordering and fallbacks
Priority ordering defines which deployment or model group is tried first and which is used only after earlier options fail. Fallbacks can move a request to a different model group, which may have different capabilities, prices or approval status. Write the failover order down per application and confirm that every fallback target is an approved model for the data the application handles.
Session affinity
Session affinity keeps related requests on the same deployment, which matters when a provider caches context or when an application expects consistent behavior across a conversation. It also reduces even spread across accounts, so it trades balance for consistency. Decide per application whether affinity is needed, and confirm how it behaves when the pinned deployment enters cooldown.
Redis as shared router state
When more than one proxy instance runs, router state and rate limit counters must be shared or each instance will make its own decisions. LiteLLM's production guide requires Redis 7.0 or newer for this case. Confirm that cooldown state, usage counts and limits are read from the same Redis, and plan for what happens to routing if Redis is unavailable.
Virtual keys limit reachable model groups
Virtual keys can be restricted to specific models, so a caller can only request the model groups its key allows. Routing and fallbacks operate after this check, so a fallback cannot move a request to a model group outside the key's allowed list. Verify this by testing each key against every model group, including fallback targets, and by reviewing keys whenever a new model group is added.
Choosing between shuffle, usage, latency, least-busy and cost-based routing
Start with simple-shuffle. The LiteLLM routing documentation lists it as the recommended default, and it spreads requests across the deployments in a model group without trying to predict which one is fastest or cheapest. That simplicity is the point: it has few moving parts to explain during an incident, and it gives you a baseline to compare against before you switch to anything that adapts to live conditions. Usage-based routing tries to balance load according to tokens or requests already consumed by each deployment. Latency-based routing favors deployments that have been responding faster recently. Least-busy routing sends new requests to the deployment with the fewest requests currently in flight. Cost-based routing prefers the deployment with the lowest configured price. Each one optimizes a single signal and ignores the others, so none of them is a general improvement.
The usual reason to leave the default is a mismatch the default cannot see. Deployments with uneven capacity, such as a self-hosted model on a smaller GPU pool next to a hosted provider account with a large quota, can be overloaded by an even split. Latency differences between regions can make a nearby deployment consistently better for interactive users, and a strategy that tracks response times will route toward it. Spend differences between provider accounts matter when one account has a negotiated rate or a separate budget; cost-based routing can then shift volume, but it will also keep favoring the cheapest route even when that route is slow or throttled. Illustratively, a team with a fast regional endpoint and a slower overflow endpoint might test latency-based routing for chat traffic and keep shuffle for batch jobs.
Do not choose on documentation alone. Before you change the strategy, collect your own per-deployment figures for a representative period: median and tail latency, error and timeout rates, rate-limit responses, and token counts by account. Compare those figures across each candidate deployment for the same model group. Then test the strategy you are considering against replayed or staged traffic, and watch how it behaves when one deployment is slowed or fails, since adaptive strategies can concentrate load on a deployment that looks healthy for a short time. Note that the router's behavior depends on the shared state it keeps, so multi-instance setups need the same test on every proxy instance.
Before you commit, confirm the current behavior of each strategy, along with its tuning options, in the LiteLLM routing documentation for the exact version you run. Parameter names, default windows for latency or usage measurements, and how cooldowns interact with the chosen strategy can change between releases. Ask the team that owns each provider account for their rate limits and pricing terms, and record which deployment should receive traffic first when everything is healthy. Write down the rollback step, usually a return to simple-shuffle, so that a routing change can be reversed quickly if the measurements you collected do not hold up in production.
Routing and fallback decisions and what each one costs you operationally
This table compares the LiteLLM routing strategies, the retry and fallback choices, and session affinity, with the operational cost each one adds. Read each row as a decision: when the option fits, and what you must maintain once it is live. The main takeaway is that the simplest strategy is the easiest to run, while the more adaptive options depend on shared state and tuning you must validate.
| Option | When it fits | Operational cost |
|---|---|---|
| Simple-shuffle (recommended) | Several deployments of the same model with similar capacity and pricing, and no need for measured latency or spend signals | Low. Keep the deployment list and weights accurate; validate that traffic spreads as intended after each config change |
| Usage-based routing | Deployments with different rate limits (TPM and RPM) where you want traffic to follow remaining capacity | Needs shared usage counters, so Redis must be reachable from every proxy instance; confirm limits are set per deployment |
| Latency-based routing | Deployments in different regions or providers where response time varies and matters to users | Latency windows and thresholds need tuning and retesting after provider or network changes; monitor for routing flapping |
| Least-busy routing | Requests with widely varying output length, where in-flight request count is a better signal than rotation | Tracks in-flight state, so confirm it stays consistent across several proxy instances before relying on it in production |
| Cost-based routing | Models with different prices where you want the cheapest healthy deployment to take traffic first | Pricing data must be kept current for every model you route; stale prices send traffic to the wrong deployment and skew spend reports |
| Retries on the same deployment | Short, transient errors such as brief timeouts or rate-limit responses from one endpoint | Adds latency per attempt and can amplify load on a struggling provider; set retry counts and backoff deliberately |
| Fallback to another model group | A provider or model outage that a retry cannot fix, or when a different model is an acceptable substitute | Needs an approved fallback list per application; the substitute model must meet the same data and quality requirements |
| Session affinity | Multi-turn conversations where keeping a user on one deployment matters for consistency or caching | Affinity state must be stored and expired correctly; it can reduce the balancing benefit and needs load testing |
How to configure routing and fallbacks in sequence
This sequence turns provider accounts and self-hosted models into named model groups with defined strategies, retries, cooldowns and fallback chains. The result is a router configuration that sends traffic only to deployments approved for each data class, and that you confirm with test requests and log entries before applications depend on it.
Inventory every deployment behind each model
Platform engineering lists each provider account, region, self-hosted endpoint and model version that can serve a request, with an owner and the data classifications each deployment is approved to handle. Security and compliance review the list before any routing is configured. The step is done when every deployment has an owner, an approved data class and a documented reason for being in the pool. Deployments without an approved data class stay out of every group until that decision is recorded.
Define model groups with shared names
Engineers give deployments that serve the same purpose one shared model group name in the proxy configuration, so applications call a single alias while LiteLLM chooses the deployment behind it. Groups should be split by capability and data class, so a group never mixes deployments with different approvals. The check is a request to the proxy's model list that returns each group name, followed by a test call to one group and a log entry showing which deployment in that group handled it.
Choose the routing strategy for each group
For each group, the team selects a routing strategy from the options the LiteLLM router supports: simple-shuffle, usage-based, latency-based, least-busy or cost-based. LiteLLM's documentation recommends simple-shuffle as the default, and the other strategies suit specific traffic patterns that should be tested before adoption. Enable session affinity only where a conversation must keep reaching the same deployment. The check is a batch of test requests whose logs show the expected spread across deployments, with the strategy stored in version-controlled configuration.
Set retries, cooldowns and timeouts
Platform engineering sets how many times the router retries a failed request and how a deployment enters cooldown after repeated failures. Values should come from the provider error patterns observed in testing and be confirmed against the current routing documentation for the installed version. Cooldown settings decide how quickly traffic leaves a failing deployment and when it returns. The check is a deliberate failure on a non-production deployment, such as an invalid key, showing the expected retries, then a cooldown entry in the logs.
Define priority order and fallback chains
Engineers set the priority order inside each group and write the fallback chain that LiteLLM follows when the primary deployment fails. Each step in the chain must be approved for the same data class as the request. For restricted data, the chain should end at self-hosted or approved private deployments, and security should confirm the configuration returns an error when no approved target remains. The check is a test with the primary disabled that shows the intended fallback, and a restricted-tagged test that fails instead of routing to an unapproved provider.
Enable Redis for multi-instance routing
When more than one proxy instance runs, the team enables Redis 7.0 or newer so instances share router state, rate limits and cache. Without shared state, one instance may not see a cooldown that another instance has triggered. Platform engineering confirms the Redis version and connection settings in each proxy's configuration. The check is a test that triggers a cooldown through one instance and then confirms that a second instance respects it, with the outcome recorded before the proxies go into service.
Restrict virtual keys to approved groups
Virtual keys need PostgreSQL, so the database must be in place before keys are issued. Administrators create keys for each team or application with an allowlist of model groups that matches the data classes that team handles. Keys with no approved group for a workload should not be issued for it. The check is a request with a key to a group outside its allowlist that returns an authorization error, followed by a successful request to a group on the allowlist.
Confirm fallback behavior in logs
Before applications move onto the gateway, the team reviews a fallback test end to end and checks the log record it produces. The record should show the original deployment, the fallback deployment and the reason for the switch, and it should contain no prompt content that the logging destination is not approved to hold. Compliance reviews the result against the fallback rules for each data class. The step is done when a fallback log entry matches the documented chain and the team signs off the configuration for that group.
Deciding which fallbacks are acceptable for regulated data
A fallback in a LiteLLM router is a data-handling decision as well as an availability one. When the primary deployment fails, the router sends the same prompt to the next deployment in the chain. That next deployment may belong to a different provider account, a different region, or a different model with different contractual and retention terms. Before you configure fallbacks for regulated data, treat every entry in the chain as a destination for that data and approve it the same way you would approve a new processor. Availability gains do not justify a route your data protection owner has not reviewed.
Start by mapping each model group in your configuration to the approved provider list. For every deployment behind a model name, record the provider, the account or subscription, the region or endpoint, and the agreement that governs it. Then check that each fallback target appears on the same list as the primary for the data classes it will receive. A deployment that is approved for public marketing content may not be approved for patient or account records. The router does not know your classification scheme, so the mapping has to live in your configuration review, with the model group names, deployment entries and any priority ordering documented together. Confirm with each provider's current terms, rather than with assumptions, which data types and regions they cover.
Keep private models as the last resort or as the only target for keys that carry regulated data. A common pattern is to scope keys by data classification using LiteLLM virtual keys, restricting each key to specific models, so that a key used for regulated workloads can only resolve to self-hosted or approved private deployments. Under that pattern, a failure produces an error and an alert rather than a silent move to a public API. If you do allow an external fallback for some keys, use priority ordering so the private model is tried first, and limit the external entries to keys and teams that have a documented reason to use them. Test this with a deliberately failed primary, and confirm in the logs which deployment served each request.
Involve your data protection and security owners before you set the chain, not after the first incident review. They can decide which data classes may leave the private environment at all, which regions are acceptable, and whether a fallback needs a separate record of processing or a data transfer assessment under your own policies. Ask them to approve the written fallback matrix, which should list each model group, its permitted data classes, its fallback order and its owner. Revisit the matrix whenever a provider is added, a region changes, or a team asks for a new model. Changes to the router configuration should go through the same review as any other change to where data flows.
Illustrative example: a team routes customer support summaries and internal engineering questions through the same model group. Under a shared chain, a regional outage could move support data to an external provider that the data protection owner never approved for customer records. Splitting the model group into two, with private deployments only for keys tagged for customer data and an approved external fallback for internal keys, removes that path. The cost is more configuration to maintain, and the benefit is a fallback rule that can be explained and audited.
How routing behaves on single node, high availability and Kubernetes
A single proxy instance keeps all routing state in its own process memory. The router tracks which deployments have recently failed, which are in cooldown, how many requests are in flight to each one, and how much traffic each key has used. Because one process owns all of this, the routing decisions are internally consistent, and the setup is the simplest to reason about and test. The trade-off is that the instance is a single point of failure for model access, and its memory-held counters reset when the process restarts.
In practice, a single instance is suitable for pilots, internal evaluation and low-stakes workloads where a short outage is acceptable. Check the following before relying on it:
- Failover scope: retries and fallback order only cover provider or deployment failures, not the proxy itself.
- Restart behavior: confirm what state is lost on restart, including cooldowns and in-memory usage counts.
- Database dependency: virtual keys and budgets need PostgreSQL, so confirm that database is also highly available or accept its outage risk.
- Capacity: confirm the vCPU and memory per worker against the production guide's suggested sizing.
- Routing strategy: confirm whether simple-shuffle, usage-based or latency-based routing fits the single-process model you are testing.
Confirm with your platform team how a single instance will be restarted during patching and what application teams should expect during that window.
When several proxy instances sit behind a load balancer, each instance receives a share of requests, so no single instance sees the full traffic picture. Without shared state, one instance could allow a key to exceed its requests-per-minute limit because its peers are counting separately, and one instance might keep sending traffic to a provider that another instance has already seen failing. LiteLLM's production guide requires Redis 7.0 or newer once more than one proxy instance runs, to hold shared rate limits, router state and cache. This is what keeps the instances behaving as one gateway.
The key design point is that cooldown and failure counts must be consistent across instances. If instance A has put a deployment into cooldown and instance B has not, traffic will keep reaching a failing endpoint through B. Check the following:
- Shared Redis: confirm Redis is 7.0 or newer and is reachable from every proxy instance on the same logical state.
- Redis availability: decide what happens to rate limits and cooldowns if Redis is unreachable, and test that behavior.
- Cooldown settings: confirm the failure threshold and cooldown period, then verify they produce the same outcome on every instance.
- Load balancer health checks: remove unhealthy instances from rotation, but confirm the checks do not hide provider failures that routing should handle.
- Shared database: confirm all instances point at the same PostgreSQL primary for virtual keys and spend.
Run a test where one instance receives failing requests and confirm the other instances stop routing to that deployment within the expected window.
On Kubernetes, LiteLLM's guidance is to run one Uvicorn worker per pod and scale by adding pods. This gives the orchestrator control over restarts, resource limits and replica counts, and it keeps each pod's memory footprint predictable. The router state model is the same as the load balanced setup: every pod needs the shared Redis and PostgreSQL to hold consistent rate limit counts, cooldowns and key budgets. Scaling pods up or down does not change the logic, but it changes how often state is read and written, so Redis and database capacity should be sized with the pod count in mind.
Kubernetes adds its own failure modes that routing teams must account for. Check the following:
- Worker count: keep one Uvicorn worker per pod, and set CPU and memory requests against the suggested sizing of 1 vCPU and 4 GiB per worker.
- Pod restarts: confirm readiness and liveness probes do not restart pods during slow provider responses, which would clear in-flight routing context.
- Autoscaling: confirm that scaling events do not overwhelm Redis or the database with spend updates, and review the batching interval for spend writes.
- Secrets: store LITELLM_MASTER_KEY, which must start with sk-, in a secret manager rather than a plain manifest.
- Network policy: confirm pods can reach Redis, PostgreSQL and every permitted provider endpoint.
Before go-live, run a load test that scales the replica count up and down, and confirm rate limits and cooldowns stay consistent throughout.
In an air-gapped network, the gateway can only route to endpoints that are reachable inside that network. Public provider APIs are usually not available, so the routing table must point at self-hosted models or at provider endpoints reached through an approved internal path. A fallback that points to an unreachable provider will fail at request time, which means the fallback order is only valid for the endpoints the network actually permits. The same route configuration can behave very differently in an connected test environment and in the isolated production network.
Fallbacks in this setting depend on what has been installed. Each provider integration and each self-hosted model server must be deployed inside the enclave and reachable from the proxy. Check the following:
- Reachable endpoints: list every endpoint in the routing configuration and confirm each one resolves and responds from inside the network.
- Installed providers: confirm which provider integrations and model servers are present in the deployed image and configuration.
- Fallback order: remove any fallback that depends on an external endpoint, and confirm the remaining order still covers the workloads that need it.
- Health checks: confirm internal health checks detect failed self-hosted models, since no external provider will signal them.
- Updates: plan how new provider versions and routing changes reach the isolated network through your own transfer process.
Test a deliberate failure of the primary internal endpoint and confirm the fallback that actually exists in the enclave takes over.
Access control and secrets for routing configuration
A routing configuration decides which provider accounts and models a request can reach, so the credentials, keys and stores behind it need the same care as the routes themselves. The cards below cover where secrets live, how the master key and virtual keys are scoped, how Redis is protected, and what the logs record about which deployment served a request.
Provider credentials outside the config file
Each entry in the router's model list points to a provider account, and the API key for that account should be read from a secret manager at startup, not written into the config file. This keeps keys out of version control and out of copies of the file that operators share. Check which secret stores your team approves, confirm how credentials are rotated without a config edit, and confirm which values the proxy pods are allowed to read.
Master key kept out of config
LITELLM_MASTER_KEY, which must start with sk-, grants administrative access to the proxy, including issuing virtual keys and changing routing. LiteLLM's production guidance recommends storing it in a secret manager rather than in the config file or a plain environment file. Decide who can read it, when it rotates, and how a rotation is rolled out across every proxy instance.
Virtual keys scoped to model groups
Virtual keys can be restricted to specific models, so an application key only reaches the model groups it was approved for. Map each team or application to the model groups in the routing config, and make sure a fallback does not send a restricted key to a provider or model it was never approved to use. Test this with a key that should be denied and confirm the denial is logged.
Redis access and network placement
Once several proxy instances share router state and rate limits, Redis holds data that drives routing decisions and cooldowns. Require authentication and TLS for Redis connections, place it on a private network reachable only from the proxy pods, and restrict its management ports. Confirm that the Redis version meets the 7.0 minimum in the production guide and that backups do not expose cached request data.
Logs that show the serving deployment
Routing decisions are easier to audit when each log entry records which deployment and provider account served a request, along with fallback attempts and the reason for a retry or cooldown. Prompt and response bodies should be excluded or limited by policy, since they are often sensitive. Agree the logging fields with your security and privacy teams before go-live and verify them with a test request that follows a fallback path.
Enterprise secret manager integrations
LiteLLM Enterprise lists integrations with AWS, Azure, Google, HashiCorp Vault and CyberArk for reading secrets. Whether one of these fits depends on the secret manager your organization already runs and the standards it applies to privileged credentials. Confirm with the vendor which integration covers your store, which license tier includes it, and how a rotated secret reaches running proxy instances.
Testing that routing works and fails safely
Before any routing configuration reaches production, it should be proven under the same conditions it will face there. The core question is whether LiteLLM sends traffic where you intended, retries and falls back only when it should, and never lets a request reach a model the calling key is not allowed to use. Build a test plan around the router settings you chose, the deployments behind each model group, and the virtual keys that call them, then run it against the exact LiteLLM release you plan to deploy, not a nearby one.
Start with distribution. Drive a sustained load through a model group that has several deployments, such as two provider accounts for the same model, and confirm the share of requests each deployment receives matches the routing strategy you selected. If you chose simple-shuffle, expect roughly even spread. If you chose usage-based, latency-based, least-busy or cost-based routing, confirm the behavior under load rather than assuming it. Record request counts per deployment, error rates and p95 latency, and confirm that no single provider account exceeds the rate limit or spend ceiling you agreed with its owner.
Next, force failure. Disable or point one deployment at an endpoint that returns errors, then check three things in the logs: that the configured number of retries happens, that the failing deployment enters cooldown after repeated failures and stops receiving traffic, and that the priority-ordered fallback to the next model group completes without the calling application seeing an error. Repeat the test with the failing deployment restored to confirm it returns to rotation after the cooldown. Ask your LiteLLM contact or check the routing documentation for the exact cooldown and retry parameters in your version, because your test expectations must match them.
A fallback must also be tested for what it should not do. Create a virtual key restricted to a specific set of models, force the primary model for that key to fail, and confirm that the fallback does not route the request to a model outside the key's allowed list. A fallback that silently widens model access is a data governance problem, not just a reliability one. Include a second check that restricted-key failures return a clear error to the caller. Redis loss and instance restarts need their own tests: stop Redis during load, restart one proxy instance, and confirm how rate limits, router state and cache behave, including whether requests fail closed or open. Your platform team should confirm the expected behavior with the LiteLLM release you run, and decide in advance which behavior the business accepts.
Finally, confirm that spend and rate limits are counted once. Route a known number of requests through a fallback path and through a retried request, then reconcile the counts in the spend logs and the budget totals against your own count. Double counting inflates budgets and blocks legitimate teams, while missed counts undermine cost reporting. Save every test plan, configuration version, log extract and reconciliation result as evidence, attached to the change request so approvers can see what was tested, on which release, and with which outcome. Plan for this testing effort to recur with each planned upgrade, because routing behavior can change between releases.
What to monitor, back up, upgrade and document after routing goes live
This table lists what operations teams should watch, protect, retest and record once LiteLLM routing, fallbacks and load balancing are in production. Each row names the signal or task, why it matters for routing specifically, and what to confirm in your own environment. Read it as a checklist to agree with the platform owner before cutover, because most routing incidents show up first as silent fallback traffic or stale router state rather than as outright outages.
| Area | What to watch or do | Why it matters and what to confirm |
|---|---|---|
| Per-deployment error rates | Track error rate separately for each model deployment in the router group, not only for the model alias. | An alias can look healthy while one provider account fails and the router sends its share to the others. Confirm your dashboards break errors down by deployment. |
| Cooldown events | Alert when a deployment enters cooldown after repeated failures, and record how long it stays out of rotation. | Cooldowns hide failing capacity from callers. Confirm the cooldown thresholds you configured match the failure pattern you expect from each provider. |
| Fallback frequency | Count how often requests move to a fallback model or deployment, grouped by source model and application. | A rising fallback rate often means the primary is degraded or over a limit. Confirm which fallback paths are approved for each data class before traffic moves. |
| Latency by deployment | Compare response latency across deployments in the same group, including time to first token where your clients use streaming. | Latency-based routing and load spreading both depend on accurate timings. Confirm the routing strategy in use still fits the latency profile you observe. |
| Rate limit hits | Record provider rate limit responses and the router's retry counts, per deployment and per provider account. | Shared accounts across teams or deployments can exhaust limits quickly. Confirm the limits you hold at each provider and the thresholds that trigger a reroute. |
| Redis health and memory | Monitor Redis availability, memory use, eviction behavior and latency from every proxy instance. | Redis holds shared rate limits, router state and cache once more than one proxy runs. Confirm the Redis version meets the 7.0 minimum in the production guide and that eviction settings cannot drop router state silently. |
| PostgreSQL state | Watch database connections, lock waits, storage growth and the health of the tables that store keys, budgets and spend. | Virtual keys and budgets require PostgreSQL, and spend batching writes to it every 60 seconds at scale. Confirm backup and restore are tested against the same schema version as the proxy. |
| Routing configuration backup | Keep the proxy config, router settings, model lists and fallback order in version control with reviewed changes. | A routing change can redirect sensitive traffic within minutes. Confirm the repository holds every setting that affects routing and that deployments are traceable to a commit. |
| Fallback policy documentation | Document, for each data class, which models and provider accounts may receive fallback traffic and which may not. | Assessors and data owners will ask where a request can go when the primary fails. Confirm the policy is approved by the data owner and matches the config in version control. |
| Failover tests after upgrades | Re-run the failover and cooldown tests after every upgrade within a supported minor version and after every config change to the router. | Router behavior can change between releases. LiteLLM states it maintains support for the four most recent stable minor versions, so confirm which version you run and retest it. |
| Provider account limit reviews | Review provider quotas, rate limits and spend caps on a set schedule, and after any change in traffic or model access. | Limits set at the provider define how far load balancing can help. Confirm who owns each provider account and how limit increases are requested and recorded. |
Questions teams ask before routing work
What drives the cost of LiteLLM routing work?
The main drivers are the number of model groups, the number of provider accounts, the number of environments, and how much fallback policy needs review. Each model group needs its own routing decision: which deployments serve it, which strategy applies, and what order failover follows. Each provider account adds credentials, quota checks and limits to test. Each environment, such as staging and production, repeats the test and sign-off cycle. Fallback policy review often takes the most time when data classification rules restrict which providers may receive which data. Recurring costs appear if you ask for routing retests after provider or LiteLLM upgrades. Ask for a quote that lists each count, the deliverables per environment and the approvers named in writing.
How long does LiteLLM routing setup take?
Duration depends on the environment and the scope agreed in writing, and no fixed timeline applies to every team. The number of model groups, provider accounts and environments sets most of the work. An existing endpoint inventory shortens discovery. A team with one environment and two providers has a shorter path than a team that must test failover across several regions. Timing also depends on how quickly your approvers review fallback order and whether test traffic is available. Ask for a written schedule with milestones for discovery, configuration, test runs and cutover. Confirm which dependencies sit with your team, including provider account access and the provisioning of PostgreSQL and Redis.
What do we need to provide before routing work starts?
Start with four inputs: provider accounts, an endpoint inventory, data classification, and test traffic with named approvers. Provider accounts need API access with the scopes and quotas your team agrees to share, and you should confirm who owns each account and how keys are rotated. The endpoint inventory lists every hosted and self-hosted model that could serve a request, along with its region. Data classification tells the design which providers may receive which data, which affects fallback order. Test traffic means representative or synthetic requests sent to a non-production environment. Approvers sign off on strategies, cooldowns and fallback paths. Confirm this list in writing before the scope is final.
Can our internal team set up LiteLLM routing without a partner?
Yes, a capable internal platform team can configure routing from the documentation, and many teams do. The routing docs describe simple-shuffle as the recommended strategy, along with usage-based, latency-based, least-busy and cost-based options, plus retries, cooldowns and priority ordering. The harder work is testing failover against real provider errors, setting cooldowns that do not repeatedly remove healthy deployments, and keeping router state consistent when several proxy instances run. Multiple instances need Redis 7.0 or newer for shared router state. Decide based on whether your team has time for test cycles and on-call ownership after go-live. A partner is useful when the team lacks that capacity or wants an independent review of the design.
Is LiteLLM Enterprise required for routing features?
Based on the routing docs and the Enterprise feature list, the routing strategies are not listed as Enterprise-only features, but confirm this against your version before you commit. The routing docs cover simple-shuffle, usage-based, latency-based, least-busy and cost-based routing, plus retries, cooldowns, priority failover and session affinity. The Enterprise list covers SSO and SCIM beyond the free limit, JWT authentication, audit logs, organization and team RBAC, IP allowlisting, secret manager integrations, and team-level guardrails and logging. Admin UI SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. Ask the vendor to confirm in writing which features your planned design uses.
Which routing strategy should we pick first?
Start with simple-shuffle, which the routing docs recommend, unless your traffic data supports another strategy. Usage-based and least-busy routing respond to load across deployments, latency-based routing favors faster deployments, and cost-based routing favors cheaper ones. Each strategy changes which provider receives traffic, so test each one against your rate limits and budgets before rollout. Retries and cooldowns protect callers from a failing deployment, but aggressive settings can take healthy capacity out of the pool. Session affinity matters when a conversation must stay on one deployment. Have your team or DeployOpen document the chosen strategy, the fallback order and the reason for each choice so approvers can review them.
What usually breaks in LiteLLM fallback setups?
The most common failures come from fallback chains that send regulated data to providers the data classification did not approve, cooldowns that remove every deployment at once, and retries that multiply load during a provider outage. Testing should simulate provider errors, rate limit responses and timeouts, and confirm that each fallback path respects model access rules and virtual key restrictions. Watch for budget exhaustion on a fallback provider, since spend shifts when traffic moves. Verify that logs show which deployment served each request, and that prompts are stored only where policy allows. Ask for a test matrix that lists each failure type, the expected behavior and the owner of each result.
How DeployOpen works on this
DeployOpen plans, configures, tests and documents LiteLLM routing, fallbacks and load balancing inside your own infrastructure. The work starts with a written scope that names the models, routing strategy and failover order, then builds and verifies the router configuration before handing it over with runbooks your team can operate.
How we scope
- Inventory the model providers, provider accounts and self-hosted models that the router must reach
- Agree the routing strategy per model group, such as simple-shuffle, usage, latency, least-busy or cost-based
- Define retry counts, cooldown behavior, priority order for failover and any session affinity needs
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
How we build
- Configure model groups, deployments and fallback chains in the router settings under version control
- Test failover by simulating provider errors, rate limits and unavailable self-hosted endpoints
- Check that routing decisions and spend limits behave as written, with results recorded for your review
- Confirm shared router state runs correctly when more than one proxy instance is active
How we hand over
- Deliver the router configuration, model group map and fallback order as documented, versioned artifacts
- Provide runbooks for changing weights, adding a provider account and rotating credentials safely
- Walk your platform and security teams through test results, known limits and the operating checklist
- Document which settings your team must confirm with each provider, since behavior depends on their limits
How we support
- Offer ongoing support, upgrades and continuous testing of routing behavior as a retainer
- Retest fallback and load balancing after each LiteLLM upgrade within the supported minor versions
- Review routing changes when providers alter rate limits, model names or pricing
- Confirm support scope, response expectations and access in the written agreement before go-live
