How LiteLLM high availability keeps model access running
High availability for a LiteLLM gateway means several proxy instances sit behind a load balancer, they share state through Redis, they depend on a resilient PostgreSQL database, and the router fails over to another model provider when one stops responding. Running more containers is the easy part. The hard part is the shared state between instances and the failure paths that connect the proxy, Redis, PostgreSQL and each provider. If those paths are not tested, a single proxy or database failure can stop every team that calls models through the gateway.
Applications now send model traffic through one gateway, so that gateway becomes a shared dependency. A single proxy process, a single database or a single provider account can each stop access for every team that relies on it. The LiteLLM production guide names PostgreSQL as the primary supported database. It also requires Redis 7.0 or newer once more than one proxy instance runs, because Redis holds shared rate limits, router state and cache. Before you plan capacity, confirm the Redis version and topology you intend to operate against the current production documentation, since that guidance can change between releases.
Several components need explicit decisions. The load balancer needs health checks that call the proxy's health endpoints, not just a TCP check, so that an instance which has lost its database connection is removed from rotation. Redis needs a deployment model with replication and failover that your platform team can operate, and you should confirm how the proxy behaves during a Redis failover window. PostgreSQL needs its own high availability design, and you should confirm how the proxy reconnects after a primary switch. Virtual keys, budgets and spend tracking all depend on that database, so a database outage affects authorization as well as routing.
Provider failover is a separate control from infrastructure failover. The router supports retries, cooldowns after repeated failures and priority ordering for fallbacks, and it offers several routing strategies such as simple-shuffle, usage-based, latency-based, least-busy and cost-based. Choose the strategy and fallback order with the workloads in mind, and test that a provider timeout or quota error moves traffic to an approved alternative without exposing data to an unapproved region or vendor. Confirm whether each fallback model is approved for the data classes it will receive.
Verification should be a scripted test plan, not a diagram. Stop one proxy instance under load, stop a Redis node, fail over the PostgreSQL primary and block one provider endpoint, then record what applications see, how long requests fail, and whether spend and rate limits stay accurate afterward. Ask your team to define the acceptable behavior for each test before running it. The resulting runbook is the artifact your operations and security reviewers will need.
Which LiteLLM components need redundancy and why
A LiteLLM gateway is only as available as its weakest dependency. Each component below either serves requests, holds state that several proxy instances must share, or sits on the path to a model provider, so losing it changes what applications experience.
Proxy instances
The LiteLLM proxy receives every OpenAI-format request and forwards it to a provider. LiteLLM recommends one Uvicorn worker per pod on Kubernetes and scaling horizontally, so capacity and failure isolation come from the number of pods. If one pod is lost and no others are running, every request routed through it fails. Confirm the replica count, readiness and liveness probes, and how many pods can be lost during a node drain.
Load balancer in front
The load balancer spreads client connections across healthy proxy pods and removes unresponsive ones from rotation. Without it, applications must know individual pod addresses, and a single failed endpoint keeps receiving traffic. Check that health checks call the proxy's health endpoint, that connection draining is configured for rolling upgrades, and that the balancer itself is deployed redundantly in your environment.
Redis for shared state
LiteLLM's production guide requires Redis 7.0 or newer once more than one proxy instance runs. Redis holds shared rate limits, router state and cache, so instances agree on usage and cooldowns. If Redis is missing, instances can disagree on limits and routing state, and cache-dependent behavior degrades. Confirm the Redis deployment model, failover behavior and what happens to in-flight limit counters during a failover.
PostgreSQL for keys and spend
LiteLLM virtual keys require PostgreSQL, which is the primary supported database. It stores keys, budgets and spend records. When the database is unreachable, key validation and budget enforcement can fail, so requests may be rejected even when providers are healthy. Confirm the high availability design of the database, backup and restore targets, and how spend updates are batched at scale.
Provider paths
Each request ends at an upstream provider such as OpenAI, Anthropic, Gemini, Bedrock or Azure. A provider outage or rate limit stops requests on that path, even if every LiteLLM component is healthy. Routing features such as retries, cooldowns and priority ordering decide whether traffic moves to another deployment. Confirm which providers have a second deployment or account, and which model aliases have a fallback. The routing sibling topic covers how to choose and tune these settings.
Which HA topology fits your traffic, team count and recovery target
This table compares four ways to run LiteLLM high availability, from a single managed proxy to a multi-region design with separate databases. Read each row to see when the option fits and what your team would own to run it, then use the questions in the last row to choose before you size the platform.
| Topology | When it fits | Operational cost to own | Questions to answer first |
|---|---|---|---|
| Single proxy with managed restarts | Low traffic, one team, and a recovery target that tolerates a restart and a short gap in model access | One proxy instance, a process supervisor or pod restart policy, alerting on health checks, and a PostgreSQL database you back up and restore | Can the business pause AI calls during a restart? Who gets paged, and who can run a manual restore of the database? |
| Several replicas in one cluster or zone | Steady traffic that needs load spreading and survival of a single instance failure, with one team operating the cluster | Redis 7.0 or newer for shared rate limits and router state, a load balancer, horizontal scaling with one Uvicorn worker per pod, and key and config sync through a shared PostgreSQL database | Does the zone's failure matter to you? Is Redis managed or self-run, and who patches it? |
| Replicas spread across availability zones | Traffic that must survive loss of one availability zone and where cross-zone latency is acceptable for gateway calls | Zone-aware scheduling, cross-zone traffic and latency to PostgreSQL and Redis, failover tests that remove a zone on purpose, and checks that health probes behave during zone loss | Which PostgreSQL and Redis options does your cloud offer across zones? Have you measured added latency from your applications to the gateway? |
| Multi-region with separate databases | Strict recovery targets, regional data residency needs, or applications in several regions that cannot depend on one region | Separate PostgreSQL and Redis per region, a decision on how virtual keys, budgets and spend are synced or kept apart, config drift control, routing and provider failover rules per region, and regular cross-region failover drills | Which data may leave a region? How will budgets and spend be reconciled across regions? Who owns the runbook for a regional outage? |
How to build LiteLLM high availability in sequence
This sequence builds LiteLLM so model access continues when one proxy instance, one database node or one provider fails. Each step depends on the one before it, so the order matters: confirm the topology first, then persistent and shared state, then the proxies and routing. The sequence closes with failure tests that show whether the design holds.
Confirm topology and hosting decisions
Your team and DeployOpen agree on the target topology before any host is built. This covers how many proxy instances run, which Kubernetes or virtual machine environment hosts them, and whether PostgreSQL and Redis are managed services inside your cloud account or self-run in your data center. Record which provider endpoints each instance must reach and which network zones may carry prompts. The step is done when the architecture sheet names every component, its host, its failure domain and its owner, and the security team has approved where traffic and secrets flow.
Provision PostgreSQL with backups and failover
Provision PostgreSQL as the primary supported database for LiteLLM, because virtual keys, budgets and spend records depend on it. Use a highly available setup in your environment with automated backups, point-in-time recovery and a tested failover path. Confirm the engine and version against the LiteLLM production guide before procurement. The step is done when a restore from backup succeeds into a scratch database and a test proxy instance can create and read a key record through the connection string. Ask the platform team which failover behavior the managed service provides and what it costs to run.
Provision Redis 7.0 or newer
Provision Redis at version 7.0 or newer. The LiteLLM production guide requires this once more than one proxy instance runs, because shared rate limits, router state and cache live in Redis. Place Redis in the same network zone as the proxies, with transport encryption and authentication enabled to match your standards. Size the instance with the platform team based on expected load and observed traffic. The step is done when a test pod in the proxy network connects, returns a successful ping, completes a set and get round trip, and reports a server version of 7.0 or higher.
Store the master key in a secret manager
Store LITELLM_MASTER_KEY only in your secret manager. The key must start with sk-. Keep it out of configuration files, Git repositories and container images. Grant read access only to the service identities that run proxy instances, and inject the value at startup through your orchestrator's secret mechanism. The step is done when a scan of the repository and image layers finds no copy of the key, and a proxy started with the injected value accepts an authenticated admin request while refusing an unauthenticated one.
Deploy proxy instances behind a load balancer
Deploy two or more proxy instances behind a load balancer. On Kubernetes, LiteLLM recommends one Uvicorn worker per pod and horizontal scaling, so set replica counts and resource requests from the production guide's suggestion of 1 vCPU and 4 GiB memory per worker as a starting point, then adjust after load testing. Configure the load balancer health check against the proxy health endpoint so unhealthy targets leave rotation automatically. The step is done when each instance returns a healthy response directly and through the load balancer.
Connect every instance to shared state
Connect every proxy instance to the same PostgreSQL database and the same Redis endpoint, using identical configuration. Set the spend update batching interval for scale. LiteLLM documents batching database spend updates every 60 seconds at scale, so confirm that behavior in the current documentation before adopting it, and expect dashboard spend to trail actual usage by up to that interval. The step is done when a virtual key created through instance A is returned by the key information endpoint through instance B with the same budget, and each pod's logs show a successful Redis connection.
Configure provider entries and failover order
Add each provider account and private model as a model entry, then set the routing strategy, priority ordering for failover, retries and cooldowns after repeated failures. LiteLLM documents simple-shuffle as its recommended routing strategy, so start there and test latency-based or cost-based routing against your own traffic before changing it. The step is done when a test in a non-production environment disables the primary provider credential, shows requests moving to the secondary entry in the order you set, and shows the primary returning to rotation after its cooldown ends.
Run failure tests before go-live
Run failure tests in a production-like copy before go-live. Stop one proxy instance during a load test, restart a Redis node, fail over PostgreSQL and revoke a provider credential. Record what clients see and how long each recovery takes, then compare the results with the recovery targets your service owners set. The step is done when each scenario has a written result, the runbook reflects what you observed, and the service owners accept any remaining gaps in writing.
What happens to requests when Redis, PostgreSQL or a provider fails
Before go-live, map what each failure does to live traffic. Four failure modes matter for a LiteLLM high availability design: a proxy instance stops responding, Redis becomes unavailable, PostgreSQL becomes unavailable, and an upstream provider returns errors or times out. Each one touches a different part of the request path, and the outcome for a given request depends on the LiteLLM version, the deployment topology and the configuration your team chose. Do not assume that a request which succeeds in a single-instance test will behave the same way when a component drops out during peak load. A proxy instance failure is the simplest case. The load balancer should remove the failed instance from rotation after its health checks fail, and requests in flight on that instance are the ones most likely to error. Your decision is the health check design: which endpoint the load balancer probes, how often, and how many failures trigger removal. Confirm in your version whether the readiness and liveness endpoints reflect dependency health or only process health, because a pod that is up but cannot reach its database may still receive traffic. Redis carries shared rate limits and router state once more than one proxy instance runs. Confirm what your configuration does when Redis is unreachable: whether each instance falls back to local counters, whether rate limits become per-instance and therefore looser than intended, and whether router cooldown and usage state diverge between instances. Treat a loss of coordination as a budget and limit risk, not only a latency risk, and decide whether that trade-off is acceptable for each team and provider account. PostgreSQL is the component most likely to fail closed. Virtual key validation and spend tracking depend on it, so you must test whether requests using existing keys are rejected, served from any cache, or allowed without budget checks during an outage. Ask which requests keep succeeding and which fail, and record the answer per key type. Provider failures are handled by retries, cooldowns and priority failover in the router. Confirm the retry count, cooldown trigger, failover order and timeout values in your configuration, and rehearse a provider outage in staging to see whether failover actually completes within the timeouts your applications can tolerate. Behavior varies by version and setup, so verify each case in your own environment rather than relying on documentation summaries. Run a scripted failure test for each component, record which requests succeed, which fail closed and which degrade, and repeat the test after every version upgrade. Ask your DeployOpen or internal team to include these results in the go-live evidence pack.
What drives the effort and cost of a LiteLLM HA build
This table lists the factors that change how much work a highly available LiteLLM build takes and what it costs to run. Read each row as a question to answer about your environment before requesting a quote, with the right-hand column showing what keeps the effort contained. The main takeaway is that the number of providers, the database and cache hosting model, and compliance evidence needs usually shape scope more than the proxy itself.
| Driver | Why it raises effort or cost | What keeps it down |
|---|---|---|
| Providers and private models to fail over between | Each provider or self-hosted model needs its own credentials, model entries, health checks, cooldown and fallback order, and testing of failover paths. Cross-provider failover also needs compatible behavior checks. | Start with the providers that carry real traffic, define one model group per use case, and keep fallback order in one reviewed config file. |
| Teams, applications and virtual keys | Each team needs keys, budgets, rate limits and model restrictions. Key ownership, onboarding and offboarding then become recurring operational work. | Issue keys per team rather than per developer, rely on templated key policies, and automate issuance where your identity process allows it. |
| Traffic volume and spend write rate | Higher request rates raise the load on the proxy, Redis for shared rate limits and router state, and PostgreSQL for spend records. Load tests and capacity sizing take longer at high volume. | Size from measured load in a staging environment, batch spend updates as LiteLLM recommends at scale, and scale proxy pods horizontally. |
| PostgreSQL and Redis: managed or self-run | Self-run databases and caches add patching, backup, failover, monitoring and restore testing to your team's duties. Managed services shift part of that work to the cloud provider but still need configuration and verification. | Use a managed PostgreSQL service and a managed Redis service where your policy allows, and confirm the Redis version meets the 7.0 minimum LiteLLM names. |
| Kubernetes versus VM hosting | Kubernetes adds ingress, probes, secrets objects, horizontal autoscaling and cluster upgrades to the build. VM hosting needs its own load balancer, process supervision and rolling update procedure. | Reuse the platform your team already operates. Keep one worker per pod as LiteLLM recommends and template the deployment manifests or configuration. |
| Number of environments (dev, staging, production) | Each environment needs its own database, cache, keys, provider accounts and configuration. Failover tests must run outside production, which adds test effort. | Keep one shared configuration with environment-specific values only, and promote the same artifact through each environment. |
| Identity and secret manager integration | Admin UI single sign-on, team assignment from identity claims and storage of the master key and provider credentials in a secret manager each need configuration and testing. Some of these options belong to the Enterprise edition, which changes licensing scope. | Decide early which integrations you need, confirm which features your chosen edition includes, and standardize one secret manager path for all environments. |
| Compliance evidence requirements | Assessors may ask for configuration baselines, access reviews, logging evidence, failover test records and change history. Producing and retaining this evidence adds work during the build and afterwards. | Define the evidence list before the build, keep configuration in version control, and record failover tests as repeatable runs. Confirm the requirements with your assessor or compliance lead, since DeployOpen prepares evidence but does not issue assessments. |
How the HA design changes on Kubernetes, VM fleets and air-gapped networks
On Kubernetes, the availability design mostly maps onto platform primitives you already run. LiteLLM's production guidance recommends one Uvicorn worker per pod and scaling by adding pods, so capacity grows horizontally and a failed pod is replaced by the scheduler rather than repaired in place. Redis and PostgreSQL remain shared dependencies outside the proxy deployment, and every replica must point at the same instances for rate limits, router state and virtual key data to stay consistent.
- Replica count and spreading: Run several proxy pods and spread them across nodes or zones with the scheduler's anti-affinity or topology rules, so one node loss does not remove all model access.
- Probes: Configure readiness probes so traffic reaches only pods that can serve requests, and liveness probes so hung processes restart. Keep the liveness check narrow so a brief database slowdown does not trigger restart loops.
- Secrets: Load LITELLM_MASTER_KEY, provider credentials and database URLs from the cluster secret store or an external secret manager, never from the image or a plain config file.
- Rollouts: Use the deployment's rolling update settings so old and new pods overlap during upgrades, and check that database migrations are compatible with both versions during that window.
Confirm with your platform team which ingress or service features handle connection draining, session affinity and health checks, and whether your cluster's network policies allow the proxy to reach each provider endpoint.
On VM or container fleets, no orchestrator replaces failed instances, so availability depends on a managed load balancer, a process supervisor and a repeatable way to distribute configuration. Each instance is an independent proxy that must reach the same Redis and PostgreSQL services. Fleets are often easier to reason about for teams that already run stateless services this way, but the operational steps become manual unless automated.
- Process supervision: Run the proxy under systemd or a container restart policy so crashed processes return without waiting for an operator.
- Load balancer health checks: Point the load balancer at the proxy's health endpoint and remove instances that fail, then confirm how quickly the balancer drains connections during a restart.
- Config distribution: Keep the proxy configuration in version control and push identical files to every instance, so routing and model aliases do not drift between nodes.
- Rolling restarts: Restart one instance at a time, wait for it to pass health checks, then continue, which keeps capacity available during changes.
Confirm whether your load balancer supports sticky sessions if you use session affinity in routing, the connection draining timeout, and whether its health checks can distinguish a healthy proxy from one whose database is unreachable.
In air-gapped or private data center networks, the design must work without internet access. Images, Python packages and provider endpoints come from internal sources, and health checks must not depend on external services. The trade-off is that every dependency you normally take for granted, including image updates, license validation questions and model endpoints, needs an internal path that someone maintains.
- Internal registries: Mirror the proxy image, its dependencies and the PostgreSQL and Redis images into an internal registry, and pin versions so a rebuild does not pull something unreviewed.
- Offline or private model endpoints: Route to self-hosted models or private provider deployments inside the network. Each endpoint needs its own health checks and failover order, since external provider failover may not be available.
- No external health dependencies: Make sure the proxy's health and startup checks, and any callbacks or logging destinations, resolve only to internal services. Outbound calls that fail can slow requests and trigger cooldowns.
- Time and certificates: Plan internal certificate authorities and time synchronization, because TLS failures between the proxy, Redis and PostgreSQL look like outages.
Confirm with your provider or model vendor which endpoints work offline, whether any LiteLLM feature you rely on calls an external service, and how your security team will approve and update mirrored artifacts.
Securing shared state in an HA LiteLLM deployment
Running several LiteLLM proxy instances multiplies the places where secrets, credentials and prompt data can leak. The four areas below are where high availability changes the security work, and each one needs a decision before the second replica goes live.
Master key outside config and images
LITELLM_MASTER_KEY must start with sk- and should live in a secret manager that every proxy pod reads at start, not in config.yaml, environment files baked into images or Helm values committed to source control. With several replicas, a key copied into one image or manifest is copied into all of them. Confirm who can read the secret, how it is rotated across all pods at once, and whether your chosen edition integrates with your secret manager (Enterprise adds AWS, Azure, Google, HashiCorp Vault and CyberArk integrations; confirm which of these your edition includes).
Database, Redis and TLS paths
Every proxy pod opens connections to PostgreSQL for virtual keys and spend, and to Redis 7.0 or newer for shared rate limits and router state. Each of these is a path that must be encrypted, authenticated and limited to the proxy subnet or namespace. Check that Redis is not reachable from application networks, that database users have only the privileges LiteLLM needs, and that certificate validation is enabled on both connections rather than skipped for convenience.
Provider credentials that rotate safely
Provider API keys for OpenAI, Anthropic, Azure, Bedrock and others are shared by all replicas, so one revoked or leaked key affects the whole fleet. Scope each provider key to the models and accounts it needs, and agree a rotation runbook that adds the new key, rolls replicas in a controlled order and then revokes the old key. Confirm with your platform team whether a rolling restart is acceptable during business hours and how the router will behave while some pods hold the old credential.
Logs that exclude keys and prompts
When several instances write to shared logging, a single misconfigured logger can expose virtual keys, provider keys or full prompt bodies across every replica and every downstream log store. Decide which request fields are logged, redact authorization headers and key values before they leave the proxy, and set retention on the shared sink. Test the logging configuration with a known canary key and prompt, and confirm the result in the log platform itself.
How to test that LiteLLM high availability works and fails safely
Test high availability before go-live, in a non-production environment that matches the production topology: several proxy instances behind the load balancer, a shared Redis service, and PostgreSQL configured the way it will run in production. The goal is to prove two things. First, that model access continues through a single failure. Second, that when something does fail, the gateway fails in a way your security and finance teams have agreed to, meaning no key or budget rule is silently bypassed and no spend disappears. Write the test plan with pass and fail criteria before anyone runs it, so the results are not interpreted after the fact.
Start with consistency. Send the same requests with the same virtual key to each proxy instance directly, bypassing the load balancer, and confirm that every instance makes the same decisions: accepted or rejected, the model allowed or blocked, and the remaining budget and rate limit counters. Then exercise the edges: a key that has just reached its budget, a key at its requests-per-minute ceiling, and a key restricted to one model. If instances disagree, the most likely cause is that shared state is not reaching them, so check the Redis and PostgreSQL connection settings on each instance before testing anything else. Confirm the Redis version meets the minimum in the LiteLLM production guide, and confirm the database connection string is identical across instances.
Next, run failure tests under steady load. Stop one proxy instance while a load generator sends traffic through the load balancer, and confirm that traffic moves to the remaining instances with a small, bounded error count that your load balancer health checks explain. Then restart Redis and PostgreSQL in the test environment, one at a time, and record exactly which requests fail, which succeed, how long recovery takes, and whether the proxy returns clear errors or hangs. Decide in advance what is acceptable. For example, you may accept that in-flight requests fail during a database restart but that no request is ever served without a budget check. Record the answer as observed behavior, not as an assumption from the vendor documentation.
Then test provider failure. Simulate an error from one provider, for example by pointing a test model deployment at an endpoint that returns server errors or by revoking its credential in the test account. Confirm that retries, cooldowns and priority failover behave as your routing configuration says they should. The routing sibling topic covers how to choose the strategy and set the values, so this test only checks outcomes: the request is retried the expected number of times, the failing deployment is taken out of rotation and returns after its cooldown, and the request is served by the next priority deployment without the calling application changing anything. Also confirm that a failover does not send data to a provider or region your approvals do not cover. After recovery, verify that spend is recorded correctly in PostgreSQL for requests served before, during and after each failure. Reconcile the totals against the provider billing for the test accounts.
Finish with alerting and the go-live exit checklist. Confirm that alerts fire for an instance outage, a database or Redis outage, a provider cooldown, and a budget threshold, and that each alert reaches the on-call person who owns it. Then check that the following are true: the test results are written up with the observed failure behavior and recovery times; each open risk has an owner; the runbook covers restarting each component; backups and restore for PostgreSQL have been tested separately; and the team has agreed the acceptable failure behavior in writing. Keep the signed-off results as the baseline for operations. Repeat the same tests after each upgrade, configuration change or topology change, and compare against that baseline before accepting the change.
What LiteLLM high availability costs and what it needs from your team
What drives the cost of running LiteLLM in high availability?
The main cost drivers are the compute for several proxy instances, a PostgreSQL database and a Redis service that are both highly available, and the people time to build, test and operate them. Infrastructure is recurring: you pay your cloud or data center provider for proxy pods, database and cache capacity, plus backups and replicas. Staff time is mostly one-time for design, deployment and failure testing, then recurring for monitoring, upgrades and on-call. Costs rise with the number of providers, the request volume, the number of environments and the depth of failure testing. Ask any quote to separate one-time build effort from recurring infrastructure and support, and to state which items are included.
Can our own team build LiteLLM high availability without a partner?
Yes, if your team already runs stateful services on Kubernetes or virtual machines and has time to test failures. The design is mostly standard: multiple stateless proxy instances, a shared PostgreSQL database, and Redis 7.0 or newer for shared rate limits and router state once more than one instance runs. The harder part is proving behavior under failure, including instance loss, Redis or database failover, and provider outages with retries, cooldowns and fallback order. If nobody on the team has run those tests before, plan for that learning time or buy it in a fixed-scope engagement.
How long does a LiteLLM high availability setup take?
Duration depends on the environment and the integrations in scope, so no fixed timeline applies across projects. The main variables are whether you use Kubernetes or virtual machines, whether PostgreSQL and Redis are existing managed services or new builds, how many model providers and private models must fail over, how many identity and logging integrations are needed, and how formal your change and testing process is. Ask for a timeline in the written scope that lists these items separately, with the test cases and acceptance criteria agreed before work starts.
What does our organization need to provide for a LiteLLM high availability project?
You need to provide access and decisions, not just approvals. Expect to provide cloud or data center access with rights to create compute, database, cache and networking resources, and provider accounts with keys or contracts for each model provider you want to fail over between. You must also decide who owns the PostgreSQL and Redis services (your platform team, a managed cloud service or a vendor), where the master key and provider credentials are stored, and who approves network exposure. Finally, name a single owner who can make decisions and accept test results.
Does LiteLLM Enterprise support change the high availability picture?
It can add contractual support, but it does not change the architecture. LiteLLM's enterprise documentation lists a dedicated support channel with the engineering team and optional 24/7 SLA terms, including a 1 hour response for critical issues and 72 hours for security patches. These terms are LiteLLM's published options, not something DeployOpen provides, and their applicability depends on your contract with LiteLLM. Confirm the exact terms in your order form before you rely on them. Enterprise licensing is also separate from high availability itself, so check which governance features your design needs.
Which failures should our high availability design be tested against?
Test every failure that could cut off model access, and record the result for each one. At minimum, cover a proxy instance stopping during active requests, Redis unavailable or failed over, PostgreSQL primary failure with a replica promoted, and a provider returning errors or timeouts so that retries, cooldowns and priority-based failover behave as configured. Also test a rolling restart and a configuration change applied during load. Agree the pass criteria, the test tooling and the evidence you will keep before testing starts, and retest after each LiteLLM upgrade.
What should we ask for in a written scope for LiteLLM high availability?
Ask for a scope that names each component, environment and integration, and that states what is delivered and how it is accepted. Useful items to request include the target topology for each environment, the list of providers and private models to fail over, the chosen Redis and PostgreSQL ownership model, the failover and load balancer design, the health check definitions, the failure test plan with pass criteria, the runbooks and handover, and the post-go-live support boundaries. Also ask which costs are one-time and which recur, and which items depend on your side of the project.
How DeployOpen scopes, builds and hands over LiteLLM HA
DeployOpen plans, implements, tests, documents and supports self-hosted LiteLLM high availability inside your own cloud accounts, data centers or air-gapped networks. Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
How we scope
- The written scope names the proxy instances, load balancer, Redis and PostgreSQL components in your environment.
- We record which providers and private models need failover, and the order in which traffic should move between them.
- The scope states which health checks, routing strategies and failure scenarios will be tested before go-live.
How we build
- We deploy several proxy instances behind your load balancer, with LITELLM_MASTER_KEY held in your secret manager.
- We configure Redis 7.0 or newer for shared rate limits, router state and cache across all proxy instances.
- We set up PostgreSQL as the primary database, along with routing retries, cooldowns and priority ordering for provider failover.
How we hand over
- We run documented failure tests, including instance loss, Redis unavailability and simulated provider outages, and record the results.
- We deliver configuration, runbooks and architecture documentation so your team can operate the failover paths independently.
- We walk your operations staff through health check dashboards, alerting and the procedure for restoring a failed component.
How we support
- Ongoing support, upgrades and continuous testing of failover paths are available as a retainer after go-live.
- Retainer scope should be agreed in writing, including which LiteLLM minor versions and failure tests are covered.
- Ask us to confirm how upgrades to supported minor versions are tested against routing and budgets before rollout.
