What the gateway does in a private AI stack
The LiteLLM proxy exposes an OpenAI-compatible API in front of many model providers and self-hosted runtimes such as vLLM or Ollama. Applications call one internal endpoint. The gateway holds provider credentials, checks each caller's key, applies model access rules, budgets and rate limits, retries or falls back when a deployment fails, and records spend and request logs.
LiteLLM does not run inference, so capacity and model quality still belong to the runtimes behind it. What it adds is one place to decide which teams may call which models, how much they may spend and where their prompts may go. That position also makes it a high-value target: one deployment can hold keys for every provider the organization uses.
Components of a production deployment
A single container works for evaluation. Production adds a database, shared state and controlled migrations.
Proxy
The gateway process, configured by a config file and environment variables. On Kubernetes, LiteLLM recommends one worker per pod and scaling by pods.
PostgreSQL
Stores virtual keys, users, teams, budgets and spend. Required for virtual keys.
Redis
Shares rate-limit and routing state across instances. LiteLLM calls for Redis 7.0 or newer once more than one instance runs.
Admin UI
Manages keys, teams, models and spend. The production guide notes that the master key doubles as its login password.
Migrations job
Applies schema changes once per upgrade. Both official Helm charts run it and keep serving pods from changing the schema.
Logging callbacks
Send request metadata, and optionally content, to Langfuse, OpenTelemetry, Datadog or S3-compatible storage.
Open-source core and enterprise features
LiteLLM is MIT-licensed outside its enterprise directory, which has a separate license. Enterprise features need a license key, and the list changes, so confirm it before committing to a design.
| Capability | Open-source core | Enterprise license |
|---|---|---|
| Admin UI single sign-on | Free for up to 5 users. | Required beyond 5 users, and for SCIM provisioning. |
| Virtual keys and spend | Keys with model lists, budgets and TPM or RPM limits, with spend tracked per key, user and team. | Automated key rotation, model-specific budgets per key, tag-based budgets and spend reports. |
| Guardrails | Custom guardrails and Presidio-based PII masking. | Guardrails per key or team, and several built-in moderation and prompt-injection integrations. |
| Governance | Logging to configured callbacks. | Audit logs with retention, IP-based access lists, secret manager integrations and team-based logging. |
Choose a deployment shape
One host runs the proxy with NUM_WORKERS set to the vCPU count, connected to PostgreSQL. This suits a pilot. Add Redis before running a second instance, otherwise each instance enforces rate limits on its own.
Run one worker per pod and scale pods with an autoscaler. LiteLLM publishes a monolithic Helm chart and a componentized chart that requires external PostgreSQL and Redis. Run migrations as a pre-sync job.
Mirror pinned images into an internal registry, route only to internal model endpoints, send logs to internal collectors and block outbound traffic except to approved providers.
Database connections equal instances times workers times the pool limit, and autoscaler maximum replicas count toward instances. Check that figure against PostgreSQL's connection limit.
Harden the gateway and its supply chain
On March 24, 2026, malicious LiteLLM versions 1.82.7 and 1.82.8 were published to PyPI outside the project's normal release pipeline. LiteLLM's security update notes that 1.82.8 also included a .pth file, which Python processes at interpreter startup. The project reported that its official proxy Docker image, which pins dependencies, was not affected, and advised anyone who installed the bad versions to rotate every secret on those systems.
Pin exact version tags, deploy from the official images (LiteLLM states its images are signed), mirror them internally after review and promote through staging. Keep unrelated credentials off the gateway host, and restrict its outbound traffic to approved providers, log destinations and the database.
Keep the master key in a secret manager and rotate it with the documented procedure. Set LITELLM_SALT_KEY before adding models and never change it, because it encrypts stored provider credentials. Issue one virtual key per application with an explicit model list, budget and rate limit, and keep the Admin UI on an internal network.
Decide what logs may contain. The turn_off_message_logging setting keeps prompts and responses out of logging callbacks while spend metadata is still tracked. Set a request timeout well below the long default.
Routing and failure behavior
Reliability rules change where a request ends up, so each needs a data-handling review and a test.
Retries
Each model group is retried a set number of times. Keep counts low enough that clients do not time out first.
Fallbacks
After retries fail, requests move to the next model group. A fallback from a private model to a public provider moves data outside your boundary, so approve each path.
Cooldowns
A deployment that exceeds its allowed failures per minute leaves rotation for a cooldown period. Alert on cooldowns.
Context window fallbacks
Oversized requests can move to a larger-context model when pre-call checks are on. Confirm that model is approved for the same data.
Back up, upgrade and recover
Back up PostgreSQL, which holds keys, teams, budgets and spend. Store the salt key and master key separately with the same retention, because a restored database without the original salt key cannot decrypt its stored provider credentials. Keep the config file and Helm values in version control, with secrets referenced and never committed.
Read the release notes before each upgrade, run the migrations job against a staging copy of production, and test representative keys, models, fallbacks and spend records. Keep the previous tag for rollback, noting that a schema change may make rollback depend on the pre-upgrade database backup.
LiteLLM operating ownership
The gateway sits between application teams, model owners and security, so each group owns a distinct part of it.
Platform team
- Run the proxy, PostgreSQL, Redis and migrations.
- Monitor latency, errors, cooldowns and connections.
- Pin, verify and promote new versions.
AI or model owners
- Approve models, providers and fallback paths.
- Set budgets and rate limits per team.
- Review provider terms and data locations.
Security
- Hold the master key, salt key and provider credentials.
- Review egress rules and logging settings.
- Track the project's security advisories.
Application teams
- Use one virtual key per application and environment.
- Handle rate-limit and fallback responses.
- Report unexpected spend or model behavior.
LiteLLM deployment questions
Do we need PostgreSQL for a pilot?
Only for virtual keys, teams, budgets or spend views. A config-only proxy shares the master key across every caller and should not reach production.
Can the gateway keep prompts out of our logs?
Yes. Message logging can be turned off globally or per callback while spend metadata is still recorded. Check each log destination's own retention as well.
How does LiteLLM relate to Langfuse?
LiteLLM routes and controls requests, and Langfuse stores traces, prompts and evaluations. LiteLLM can send request logs to Langfuse through a callback, and both can run in the same private network.
Will SSO work without a license?
For up to 5 Admin UI users, per LiteLLM's enterprise documentation. Larger admin groups, SCIM and audit logs need an enterprise license.
Scope and handover
DeployOpen deploys LiteLLM inside your cloud account, data center or isolated network, connects approved providers and private model runtimes, configures PostgreSQL, Redis and migrations, integrates SSO within your license, restricts egress and sets logging to match your data policy. We document the platform, plan and test upgrades, and can provide ongoing support. AI and LLM security testing of the gateway and its applications is a separate engagement.
Handover includes the model and provider register with approved fallbacks, the key and budget structure, pinned versions with the image verification step, egress rules, logging settings, the backup and key escrow procedure, and named owners for each operating area.
