LiteLLM proxy server vs Python SDK: when each deployment model fits
Run the LiteLLM Python SDK inside each application when one or two services call providers, a single team owns the keys and budgets are not a governance requirement. Run the LiteLLM Proxy Server as a shared gateway when several teams or applications call LLM providers and the organization needs central keys, per-team budgets, rate limits and one place to log and disable access. The SDK is a drop-in replacement for the OpenAI client with a Router for retry, fallback and load balancing. The proxy adds per-key, team and user spend tracking across providers, an admin UI and centralized logging, which the SDK does not provide because it tracks cost per response through callbacks.
The proxy is a stateful service, and the SDK is not. LiteLLM's production guide names PostgreSQL as the primary supported database for production proxy deployments, and virtual keys need it. Once more than one proxy instance runs, the guide asks for Redis 7.0 or newer, because without it each instance enforces rate limits on its own and cache hits stay local. An application that imports the SDK needs none of this. Check the condition before deciding: count the teams that need separate budgets, count the instances you expect to run, and confirm whether anyone outside the platform team must hold a key.
Keep the SDK when your check shows one owner, one or two services and no requirement for shared spend control, since the proxy then adds a database, a deployment and a credential to operate for little benefit. Move to the proxy when the answer changes to several owners, multiple instances or an audit need for per-team spend. Plan the proxy's operating effort before you commit: size pods from the guide's recommendation of 1 vCPU and 4 GiB per proxy pod, protect LITELLM_MASTER_KEY in a secret manager, and confirm the sizing against your own traffic. Read the LiteLLM proxy production best practices before you finalize the design.
How the SDK and the Proxy Server differ on deployment factors
This table compares the LiteLLM Python SDK and the LiteLLM Proxy Server on where code runs, spend tracking, key management, state, scaling, admin UI, identity and licensing. Read each row against your own answer to one question: do you need controls shared across the organization, or only per-application calls. The proxy adds components to run, while the SDK adds none beyond your application.
| Factor | Python SDK | Proxy Server |
|---|---|---|
| Where the code runs | Inside each Python application, as a drop-in replacement for the OpenAI client | As a separate self-hosted gateway service that applications call over the OpenAI request format |
| How spend is tracked | Cost is tracked per response through callbacks in the application | Spend is tracked per key, team and user across all providers |
| Keys and budgets | Budgets are handled in your own code or callbacks, not by a central key store | Virtual keys carry per-key, team and user budgets, rate limits and model restrictions |
| State and database | No shared database is needed for the core call path | Virtual keys need PostgreSQL; Redis 7.0 or newer is recommended once more than one instance runs |
| Scaling unit | Each application process scales with the application itself | Proxy instances, with one Uvicorn worker per pod on Kubernetes and horizontal scaling |
| Admin UI | Not included | Included, with SSO for Google, Microsoft Entra ID, Okta and generic OIDC |
| Identity and SSO | Uses the identity of the calling application | Admin UI SSO is free for up to 5 users from v1.76.0; beyond that an Enterprise license is needed |
| Core license | MIT, except content under the enterprise/ directory | MIT, except content under the enterprise/ directory |
| Provider coverage | Same provider coverage as the proxy, more than 100 LLM providers | Same provider coverage as the SDK, more than 100 LLM providers |
What runs where: components of the SDK and the proxy
A LiteLLM deployment is either a library inside each application or a shared service with its own data stores and secrets. The cards below separate the parts so you can see who runs each one and what stops when it fails.
In-app SDK and Router
The LiteLLM Python SDK is a drop-in replacement for the OpenAI client inside your application, and its Router adds retry, fallback and load balancing. Application teams own it, and it ships with each service's release. If a provider fails, the Router handles retries and fallbacks inside that process, but spend is only tracked per response through callbacks, so you must confirm where those records end up.
Proxy pod or VM
The Proxy Server is a self-hosted gateway that all applications call over the OpenAI request format, and it enforces virtual keys, budgets and rate limits. Platform teams operate it, typically as Kubernetes pods or a single VM. If it goes down, every application routed through it loses model access at once, so confirm your health checks, restart policy and how many instances you run.
PostgreSQL for keys and spend
Virtual keys need a PostgreSQL database, which holds per-key, user and team budgets and spend records. LiteLLM names PostgreSQL as the primary supported database for production. The database team should own backups and access controls, because losing it means lost key definitions and budget history. Ask how spend writes behave at your volume, since LiteLLM batches spend updates every 60 seconds at scale.
Redis for shared state
Redis holds rate limits, router state and cache so that several proxy instances behave as one. LiteLLM asks for Redis 7.0 or newer once more than one instance runs. Without it, each instance enforces limits on its own and cache hits stay local, so a team can exceed its quota by spreading requests across pods. Decide whether Redis is in your managed service catalog before you scale out.
Admin UI and SSO
The proxy includes an admin UI for managing keys, teams and spend. SSO with Google, Microsoft Entra ID, Okta or generic OIDC is free for up to five users from v1.76.0, and Enterprise is needed beyond that. Identity and security teams should decide who gets admin access before rollout, and confirm the SSO tier against your user count and the current license terms.
Secret store for the master key
LITELLM_MASTER_KEY, which must start with sk-, grants administrative control of the proxy. LiteLLM recommends keeping it in a secret manager rather than in configuration files. Security operations owns rotation and access logging. If the key leaks, an attacker can create keys and change budgets, so confirm who can read the secret and how rotation will be handled without downtime.
Why the proxy needs PostgreSQL and Redis in production
The LiteLLM production guide names PostgreSQL as the primary supported database for production proxy deployments, and it asks for Redis 7.0 or newer as soon as more than one proxy instance runs. Virtual keys need PostgreSQL to store keys, per-key, per-user and per-team budgets, and spend records, so a proxy without it cannot offer the governance that made you choose the proxy over the SDK. Redis carries shared state between instances: rate limits, router state and cache. The guide's warning is specific. Without Redis, each instance enforces limits independently and cache hits stay local, so a team limit set at 100 requests per minute can be exceeded by a factor of the number of pods. Plan for Redis from the first multi-instance test, not after an incident.
Before cutover, provision PostgreSQL and Redis as managed or self-operated services inside your own network, then document who owns each. Back up the PostgreSQL database on a schedule you can restore from, because it holds keys and spend history that auditors and finance teams will ask for. Treat LITELLM_MASTER_KEY, which must start with sk-, as a secret and keep it in a secret manager rather than in a configuration file. Redis is usually treated as cache, but the proxy relies on it for limit enforcement, so confirm how your Redis tier handles failover and what happens to rate limits when it restarts. Also confirm the spend batching interval the guide describes for high volume, since spend shown in the admin UI may lag the database by that interval.
One-time setup covers schema creation, network paths, secrets, TLS and the first load test. Recurring work is the larger cost: database patching and restore tests, Redis capacity checks, and upgrades. The LiteLLM releases page and the enterprise documentation describe support for the four most recent stable minor versions, so plan a recurring upgrade cadence within that window. Before production, ask LiteLLM or your implementation partner to confirm the sizing for your traffic, since the guide's figure of 1 vCPU and 4 GiB per proxy pod is a starting point, not a measured result for your workload. Run a failover test in your own environment that restarts Redis and PostgreSQL during load, and confirm that limits and key checks behave as your governance policy requires.
Moving from in-app SDK calls to a shared proxy
This sequence takes an organization from scattered provider calls inside applications to one governed gateway, with virtual keys replacing shared provider credentials. Each step names who does the work, what is one-time and what recurs, and the check that shows it is finished.
Inventory every direct provider call and key
Search application repositories, CI variables and deployment configs for calls to OpenAI, Anthropic, Gemini, Bedrock, Azure and other providers, and for the API keys they use. Record the owning team, the models called, the monthly request pattern and whether each key is shared across applications. This is one-time work for the platform team with input from application owners, and the list must be kept current as new services appear. It is done when every application that calls a model appears on the list with its key owner and provider.
Stand up the proxy with PostgreSQL
Deploy the LiteLLM Proxy Server in your own infrastructure with PostgreSQL as the database, since the production guide names PostgreSQL as the primary supported database and virtual keys need it. Platform engineers own the deployment, the database backups and the sizing decisions, using the per-pod CPU and memory guidance in the production docs as a starting point. It is done when a test request routed through the proxy returns a response and a key created on the proxy shows in the database.
Add Redis when more than one instance runs
If you run more than one proxy instance, add Redis 7.0 or newer so rate limits, router state and cache are shared. Without it, each instance enforces limits on its own and cache hits stay local, which can make budgets and limits look correct in a test and drift in production. This adds a recurring operating component with its own availability and patching needs. It is done when a rate limit set on one key is enforced consistently across two instances during a pilot test.
Move the master key into a secret manager
Store LITELLM_MASTER_KEY, which must start with sk-, in a secret manager rather than in configuration files, images or shell history. Security engineers should define who can read it and how it rotates, and confirm that the chosen secret manager is supported in the edition you run. Secret manager integrations are listed as an Enterprise feature, so confirm the license position before you rely on one. It is done when the key is absent from repositories and config files and the proxy starts from the secret store.
Create virtual keys for each team and application
Issue virtual keys per application or team with budgets, TPM and RPM limits and a restricted model list, then map each existing application to its own key. Platform and application teams decide the budget and model policy together, and the owner of each key is recorded. Keys stop being shared provider credentials at this point, so spend becomes visible per key, team and user. It is done when every inventoried application has a named virtual key and its limits are confirmed in a test.
Switch applications to the proxy endpoint
Change each application's base URL and key to the proxy endpoint and the virtual key, one service at a time, and keep the direct provider key available for rollback until the service has run cleanly. Application teams make the change and run their own regression tests, and the platform team watches logs and spend for each key. Expect the Python SDK calls to be replaced or pointed at the proxy, since the SDK tracks cost per response while the proxy tracks it per key. It is done when each service's traffic appears in proxy logs and no direct provider calls remain for it.
Retire direct provider keys and confirm the pilot
After all services run through the proxy, revoke the direct provider keys at each vendor and remove them from application configuration and CI variables. Before you do this, confirm in a pilot that fallbacks, retries and load balancing behave as your routing configuration says, and that the admin UI and SSO meet your identity requirements. Confirm also whether the SSO free limit of five users and the Enterprise features cover your user count and audit needs. It is done when provider consoles show no active keys in use by applications and the pilot checklist is signed off.
Upgrades, backups, monitoring and skills for each option
This table compares what it takes to upgrade, back up, monitor and staff the LiteLLM Python SDK and the LiteLLM Proxy Server. Read it as a running-cost checklist: the SDK adds work to each application's release cycle, while the proxy adds a service with its own database, secrets and on-call rotation.
| Area | Python SDK inside applications | Proxy Server as a shared gateway |
|---|---|---|
| Upgrades | The SDK version is pinned in each application's dependencies, so upgrades ride along with normal application release work. | The proxy is a running service that every team depends on, so upgrades need a change window and a rollback plan. LiteLLM supports the four most recent stable minor versions, so plan upgrade windows against that policy, starting from v1.105.1 (published October 14, 2026). |
| Backups | No LiteLLM-owned state. Back up the application code, its configuration and any cost records your own callbacks write. | PostgreSQL holds virtual keys, budgets and spend records, so it needs regular backups and a tested restore. Confirm how your team restores keys and spend history after a failure. |
| Monitoring | Monitoring stays with each application. Track provider errors, latency and per-response cost in the telemetry the application already emits. | Monitor the proxy's health, request latency, rate-limit hits, database connections and Redis availability. Centralized logging is a proxy feature, so alerts can cover all teams from one place. |
| On-call ownership | Each application team owns failures in its own LiteLLM calls. Central ownership is limited to shared library guidance. | A platform team usually owns the proxy and its database. Application teams should know how to report gateway errors and what the escalation path is. |
| Core skills | Python packaging and dependency management, plus familiarity with the OpenAI request format and the Router settings each application uses. | Container runtime and Kubernetes or VM operations, PostgreSQL administration, Redis operations and secret management for LITELLM_MASTER_KEY. |
| Production dependencies | None beyond the application's own runtime and provider credentials. | PostgreSQL is the primary supported database. Redis 7.0 or newer is required once more than one proxy instance runs, for shared rate limits, router state and cache. |
| Recurring effort | Recurring work is mostly dependency bumps and provider API changes, handled inside each application's backlog. | Recurring work includes minor-version upgrades, database maintenance, key rotation and reviewing budgets. Confirm the effort your team needs with your own pilot before committing. |
| Cost of getting it wrong | Drift between applications, where teams run different LiteLLM versions and budget rules, is hard to audit. | An outage or lost database blocks every team routed through the gateway. Confirm restore time targets and fallback behavior with your own environment. |
What the LiteLLM docs suggest for sizing on a VM versus Kubernetes
On a single virtual machine, the LiteLLM production guide suggests sizing the host around worker processes, with NUM_WORKERS set equal to the number of vCPUs. The guide also suggests 1 vCPU and 4 GiB of memory per worker as a baseline, so a four-vCPU host would start with four workers and roughly 16 GiB of memory. This is a starting point, not a capacity figure. Your own traffic pattern, the models you call and the callbacks you enable will decide whether that host holds.
- Worker count: Set
NUM_WORKERSto the vCPU count, then measure CPU and memory under your real request mix before adding hosts. - Shared state: Once you run more than one proxy instance, the guide calls for Redis 7.0 or newer, because rate limits, router state and cache hits are otherwise kept per instance.
- Database: PostgreSQL is the primary supported database for production, and it holds virtual keys and spend records, so size it for the write load of spend updates.
- Secrets: Keep
LITELLM_MASTER_KEY, which must start withsk-, in a secret manager rather than in a plain environment file. - Single point of failure: One VM gives you no failover for the gateway itself, so plan a second instance and a restart procedure before users depend on it.
Before go-live, run a load test that mirrors your peak hour, including streaming responses, and confirm the numbers with your own team rather than relying on the guide's baseline.
On Kubernetes, the LiteLLM production guide recommends 1 vCPU and 4 GiB of memory per proxy pod, set as both requests and limits, with one Uvicorn worker per pod. Capacity then grows by adding pods through horizontal scaling rather than by adding workers inside a pod. This keeps each pod's resource use predictable and lets the scheduler and autoscaler do the work. The per-pod figures are the documented starting point, and your cluster's autoscaling thresholds need to be tested against your own load.
- Pod resources: Set requests and limits to 1 vCPU and 4 GiB per pod, then adjust after observing actual use.
- Worker model: Run one Uvicorn worker per pod and scale out, instead of raising worker counts inside each pod.
- Shared services: Each replica needs the same PostgreSQL and Redis 7.0 or newer, so both must be sized for the total number of replicas.
- Spend batching: At scale, the docs describe batching spend updates to the database every 60 seconds, so expect spend figures in reports to lag by up to that interval.
- Autoscaling: Confirm how quickly new pods become ready and whether they warm their configuration fast enough for your traffic spikes.
Before you commit to replica counts, confirm with your platform team how your cluster handles pod disruption and rolling upgrades, and load test with the replica count you plan to run in production.
Neither deployment model sizes itself. The same guidance applies on a VM and on Kubernetes: the load on each proxy process rises with the work it does per request. Three factors matter most, and they interact. The guide's per-pod and per-worker figures only hold while these stay near the level you tested at, so the sizing conversation should start from them rather than from the baseline alone.
- Request volume: Concurrent and streaming requests consume workers and memory, and peaks, not averages, set the headroom you need.
- Number of providers: Each provider adds routing, credential handling and retry logic, and more models behind a virtual key means more routing decisions per call.
- Logging and guardrail callbacks: Every logging destination and guardrail check runs in the request path or alongside it, so enabling them can raise CPU and latency per call.
- Caching and rate limits: Cache lookups and per-key limit checks add work, and their state depends on Redis when you run several instances.
- Database write load: Spend tracking across keys, teams and users generates writes, which the batching interval reduces but does not remove.
When you request a sizing review, ask your implementation partner to load test these five factors separately, so you can see which one drives your host or pod count.
Questions buyers ask before choosing the SDK or the proxy
Can the LiteLLM Python SDK alone enforce team budgets?
No, not across a shared organization. The SDK tracks cost per response through callbacks, so each application sees the spend of its own calls. Budgets that must hold across several applications, teams or users need a central place that records spend, which is what the Proxy Server provides through virtual keys that carry per-key, team and user budgets. In practice, a team can start with the SDK inside one service and move shared traffic behind the proxy once more than one application needs the same limit. Confirm with your own engineers which services need shared limits before you decide.
What databases does a production LiteLLM proxy need, and why?
PostgreSQL is the primary supported database for production proxy deployments, and virtual keys need it to store keys, budgets and spend. Once more than one proxy instance runs, LiteLLM's production guide says to add Redis 7.0 or newer, because shared rate limits, router state and cache otherwise stay local to each instance. Without Redis, each instance enforces limits independently and cache hits are not shared. Plan for both services to be backed up, monitored and upgraded alongside the proxy, and confirm the storage and high availability model for each in your own environment.
What does the work need from our team, and can we run it ourselves?
A production proxy needs people who can operate containers or Kubernetes, manage a PostgreSQL database and a Redis service, handle secrets such as LITELLM_MASTER_KEY, and watch logs and spend. Teams that already run these services can usually keep the proxy in-house. Teams without that skill set should compare the internal effort against a fixed-scope engagement that covers planning, deployment, hardening, backup and documentation. The answer depends on your existing platform skills, so assess them before choosing.
What drives the cost of running a LiteLLM proxy?
Cost has four main parts. Infrastructure is the compute, database and Redis you run in your own accounts or data centers, and it scales with traffic and the number of proxy instances. Staff time covers deployment, upgrades and on-call work. Model provider charges are paid directly to each provider and are separate from the proxy. Licensing matters only if you need features from the enterprise/ directory. LiteLLM's production guide suggests 1 vCPU and 4 GiB of memory per proxy pod, which gives a starting point for sizing. Ask any implementation partner for a quote that names each of these components, states what is one-time and what recurs, and lists the assumptions behind the traffic estimate.
Does SSO require an Enterprise license beyond 5 users?
Yes, according to LiteLLM's documentation. From v1.76.0, SSO for the admin UI is free for up to 5 users, and an Enterprise license is needed beyond that limit. The same Enterprise tier adds SCIM, JWT authentication with identity provider tokens, audit logs, organization and team RBAC, IP allowlisting and secret manager integrations. Confirm the current limit and terms with LiteLLM before you commit, because licensing terms can change between releases.
What is the license difference between the MIT edition and Enterprise?
LiteLLM is MIT licensed except for content under the enterprise/ directory, which uses its own license in enterprise/LICENSE. The gateway, SDK, virtual keys and budgets sit outside that directory. Your legal team should read the LICENSE file in the repository and confirm which features you plan to use fall under which terms. Enterprise support and features are sold under a separate agreement, and the contract defines what you receive.
How long does a migration from direct SDK calls to the proxy take?
It depends on scope, and you should scope it before estimating. The main variables are how many applications call models directly, how many provider keys and budgets must move, whether you need SSO and audit logs at launch, and how much of your logging and secret management must be rebuilt. A pilot with one application and one team is usually a smaller project than a rollout across an organization. A fixed-scope written plan from DeployOpen or your internal team should list the applications, environments, deliverables and acceptance tests before work begins.
What should we ask for in a quote for a LiteLLM deployment?
Ask for a written scope that lists the applications, providers, environments and users in scope, along with the deliverables: deployment design, PostgreSQL and Redis setup, secret handling, backup and restore, hardening, documentation and handover. The quote should state what is one-time and what is a recurring retainer for upgrades and support, and it should name any third-party licenses or Enterprise terms that the scope requires. Ask how the team will test failover, budgets and rate limits before cutover. DeployOpen does not publish a price list, so a quote follows from that written scope.
How DeployOpen scopes, delivers and supports a LiteLLM deployment
DeployOpen plans, implements, integrates, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For LiteLLM, that means deciding with you whether the SDK, the Proxy Server or both fit your workloads, then delivering the chosen setup against a written scope.
How we scope
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- The written scope names which applications use the SDK, which teams the Proxy Server serves and which providers are in use.
- We confirm whether PostgreSQL, Redis, identity integration and secret management sit inside your environment and who owns each.
How we build
- We configure virtual keys, budgets, rate limits and model access to match your team and project structure.
- We set up the router, fallbacks and retries for the providers you have approved, and test failover behavior before go-live.
- We deploy the proxy with PostgreSQL and Redis in your infrastructure, following LiteLLM's production guidance on sizing and worker configuration.
How we hand over
- You receive documentation of the deployed architecture, key and budget policies, and the configuration owners maintain.
- We record the license position for each component, including which features depend on an Enterprise license key.
- Handover includes a walkthrough of how to add a provider, rotate the master key and restore the database from backup.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped separately from the build.
- Upgrades are planned against LiteLLM's supported minor versions, with testing of routing and budget behavior before rollout.
- Security testing of the gateway and its integrations can be scoped alongside the deployment or as a separate engagement.
