What a LiteLLM migration involves
A LiteLLM migration usually starts from one of three situations: application services that call provider SDKs directly and are spread across many repositories, provider API keys hard-coded in source or stored in scattered configuration files, or an existing gateway that the organization wants to replace. Each starting point carries different risk. Direct SDK calls mean every service has its own retry logic, model names and credentials, so each one needs a code change. Keys in code or config mean the first task is inventory, not coding. An existing gateway means the work includes moving routing rules, logging and access policy, not only endpoints.
On the application side, the main change is pointing the client at the LiteLLM proxy's OpenAI-compatible endpoint instead of the provider's endpoint. Services then request logical model aliases, such as a name meaning "general chat" or "summarization", rather than provider-specific model identifiers. Each application or team receives a virtual key in place of a raw provider key, and that key carries its own model restrictions and budgets. Confirm with your own engineers which SDKs and request formats each service uses, since some features may need adjustment when routed through an OpenAI-format interface.
What the work includes
A LiteLLM migration moves applications from direct provider SDKs, scattered keys or another gateway onto one OpenAI-compatible endpoint. The work packages below show what the client receives at each step.
Inventory of call sites and keys
Every application, script, notebook and job that calls a model provider is recorded with its SDK, endpoint, provider account and the key it uses. Shared and orphaned keys are flagged. The client receives an inventory register that becomes the baseline for the migration and for later cutover sign-off.
Gateway deployment in your environment
The LiteLLM proxy, PostgreSQL and, when more than one proxy instance runs, Redis are deployed in the client's own cloud account, data center or Kubernetes cluster. Secret handling for the master key and provider credentials is set up during this step. The client receives a deployment record with versions, network paths and configuration owners.
Provider connections and alias map
Each approved provider account and private model is connected to the gateway, then exposed under stable model aliases so applications request a name rather than a vendor model ID. Changing a provider later becomes an alias update instead of an application release. The client receives an alias map that lists each alias, its target and its owner.
Team structure and virtual key issuance
Teams, users and applications are mapped to LiteLLM teams and virtual keys, with each key limited to the models that workload needs. Old provider keys are never copied into the gateway. The client receives a key issuance record showing the owner, team, allowed models and creation date for every key.
Application changes to the gateway endpoint
Applications and scripts are changed to send requests to the gateway endpoint with a virtual key, usually by updating a base URL, key reference and model name. Each change is reviewed against the inventory so no path is missed. The client receives a change log per application and a list of any calls that could not be moved.
Staged cutover and rollback plan
Traffic is moved in stages, starting with low-risk workloads, while the previous path stays available for rollback. Each stage has entry criteria, test results and a defined way to revert. The client receives a cutover runbook with stage order, go and no-go checks, and tested rollback steps.
Decommissioning of old keys and SDK paths
After each stage is accepted, direct provider keys and unused SDK code paths are revoked or removed, so the gateway becomes the only route. Revocations are confirmed against the inventory to catch anything still in use. The client receives a decommissioning record listing each revoked key and removed path with the date and approver.
Handover and operating documentation
The client's team receives the documents and walkthroughs needed to run the gateway after the migration closes, including key request steps, alias change steps and the escalation path. Open items from the inventory are assigned to named owners. The client receives a handover pack with the runbooks, ownership matrix and a list of open items.
What drives cost and duration
This table lists the factors that most change the effort of a LiteLLM migration, why each one matters and what to do to keep it under control.
| Driver | Why it matters | How to keep it down |
|---|---|---|
| Applications and direct SDK call sites | Each application that calls a provider SDK directly needs its client code, configuration and tests changed to use the gateway endpoint, so effort grows with the count of call sites rather than the count of teams. | Inventory every SDK import and HTTP call during discovery, group applications that share a client library, and migrate shared wrappers first. |
| Provider accounts and keys in circulation | Every provider account and key found during discovery must be replaced by a gateway-issued virtual key or moved into the proxy configuration, and each one needs an owner and a rotation plan. | Count keys by provider and owner before scoping, retire unused keys early, and confirm which accounts can be consolidated behind the proxy. |
| Streaming, tool calling and response handling | Where the old code path handles streamed chunks, function or tool call payloads or provider-specific response fields differently from the OpenAI request format, adapters, tests and sometimes application logic must change. | Test each feature path against the gateway with representative prompts, and list provider-specific fields that applications read so they are mapped or removed. |
| Model alias changes | Applications that reference provider model names directly must move to gateway aliases, and every alias change needs a regression check on output quality, latency and cost. | Define a single alias catalog before migration starts, map old model names to aliases in one place, and freeze alias changes during each cutover wave. |
| Teams needing their own keys and budgets | Each team that needs a separate virtual key, budget and model restriction adds setup, approval and reporting work, and budgets must be agreed with finance before keys are issued. | Group teams with similar needs into shared budget pools where policy allows, and agree the budget owner and reporting cadence in the scope. |
| PostgreSQL and Redis once more than one proxy instance runs | Virtual keys need PostgreSQL, and the production guide requires Redis 7.0 or newer once more than one proxy instance runs for shared rate limits, router state and cache, which adds components to build, back up and monitor. | Decide the instance count early, size both services for the expected key and request volume, and confirm backup and failover design with the platform team. |
| Number of environments | Each environment, such as development, staging and production, needs its own proxy configuration, secrets, database and provider connections, and each one must be tested before a cutover. | Keep environment configuration in version control, limit production-only settings, and reuse the same deployment definitions across environments. |
| Enterprise-only governance needs | SSO beyond the free five-user limit, SCIM, JWT authentication, audit logs, team RBAC and secret manager integrations require the Enterprise edition, which changes licensing scope and the setup work. | List the governance requirements before scoping, confirm which need an Enterprise license with LiteLLM, and decide early whether the migration waits for them. |
| Evidence needed for internal audit | Internal audit usually asks for the key inventory, access approvals, budget decisions, configuration history and test results from the cutover, and collecting these after the fact adds effort. | Agree the evidence list at kickoff, capture test and approval records as each wave completes, and store configuration in version control with review history. |
Migration phases and outputs
The sequence moves applications from scattered provider keys and direct SDK calls onto one governed LiteLLM endpoint, with each phase closing on a named artifact the client keeps.
Inventory every direct SDK call and key
Discovery maps each application, direct provider SDK call, environment variable, and provider key in use, including calls made by scripts, batch jobs and third-party tools. The closing output is a call-site inventory with owners, providers, models and key locations, reviewed and signed off by the client's application owners before design begins.
Design teams, keys, budgets and aliases
Target design defines teams, virtual keys, per-key and per-team budgets, TPM and RPM limits, model aliases that map application requests to approved models, and default routing and fallback order. The output is an alias map and access design document that the client approves, so each application calls a stable alias rather than a provider model name.
Build the proxy in a non-production environment
DeployOpen builds the LiteLLM proxy in the client's non-production environment with PostgreSQL for keys, budgets and spend, and Redis when more than one proxy instance runs, since the production guide requires Redis 7.0 or newer for shared rate limits and router state. Completion is shown by a working deployment, a secret-managed master key, and test evidence for authentication, budget enforcement and routing.
Pilot one low-risk application
One application with low business impact and clear owners moves to the gateway first. Its calls are repointed to the LiteLLM endpoint with a newly issued virtual key, and the team compares responses, latency, errors and logged spend against the baseline. The pilot closes with a pilot report listing defects, fixes and the decision to proceed.
Cut over in stages with a rollback path
Remaining applications move in stages by application or team, each with a documented rollback step that restores the previous configuration if errors exceed agreed thresholds. Each stage produces a cutover log recording the change, the key issued, the test result and the rollback tested. Thresholds are agreed with the client before the first stage, not during it.
Revoke old keys and remove direct paths
Once every application reports healthy traffic through the gateway, the old provider keys are revoked and direct SDK paths are removed from code, configuration and secret stores. The output is a revocation record listing each key, its owner and the date revoked, plus a scan showing no remaining direct provider calls in the repositories in scope.
Hand over the runbook
DeployOpen hands the environment to the client's operations team with a runbook covering key issuance, budget changes, alias updates, upgrades, backup and restore, and incident steps. Handover is complete when the client's team has run the runbook steps under observation and signed off the document. Ongoing support, if wanted, is scoped separately as a retainer.
What to keep in-house and what to outsource
This table compares each LiteLLM migration work area with whether it is usually kept in-house or outsourced and the skills it needs, so you can see where your team and a partner typically divide the work.
| Work area | Usually kept in-house or outsourced | Skills it needs |
|---|---|---|
| Application code changes | Kept in-house: application owners change their own SDK calls, base URLs and key handling | Application language expertise, familiarity with OpenAI-compatible request formats, code review |
| Key and budget ownership | Kept in-house: business teams decide who gets virtual keys, budgets and model access | Cost center knowledge, product ownership, agreement on limits and approval paths |
| Provider account and contract management | Kept in-house: your procurement and provider administrators hold accounts, contracts and credentials | Vendor management, billing administration, data handling terms review |
| Inventory, migration design and cutover runbook | Usually outsourced under the agreed scope: partner maps callers, aliases and keys, then drafts the staged plan | Architecture design, LiteLLM configuration, change management, dependency mapping |
| Gateway hosting, PostgreSQL and Redis operations | Usually outsourced under the agreed scope or shared: partner builds, you confirm hosting and on-call ownership | Kubernetes, PostgreSQL administration, Redis operations, secret manager integration |
| Test design for streaming and error paths | Usually outsourced under the agreed scope: partner designs tests, application owners run them against their services | Streaming protocol knowledge, failure injection, fallback and retry testing |
| Cutover coordination | Usually outsourced under the agreed scope: partner coordinates timing, rollback steps and go/no-go checks | Release management, incident coordination, rollback planning |
| Post-cutover monitoring | Kept in-house after handover, with partner support if a retainer is agreed | Metrics and log analysis, alert tuning, on-call response, spend reporting |
What goes wrong in LiteLLM migrations
Most LiteLLM migration problems come from credentials, model names and limits that were moved without review. Each risk below has a prevention step to complete before application traffic moves to the gateway.
Old provider keys stay active
Direct SDK calls and scattered keys often keep working after traffic moves to the gateway, so original provider credentials remain valid outside LiteLLM's control. Build an inventory of every application, script and CI job that holds a provider key, then revoke each one through a checklist tied to its cutover date. Confirm revocation in the provider console, not only in code.
Model names break callers
Callers that send a provider model name break when the gateway exposes a different name or a mapping changes during the move. Write an alias map from every model name in use to its model group, then test each alias with the calling application's exact request payloads before cutover. Keep old names resolving until every caller is confirmed on the new ones.
Budgets set wrong or missing
A budget set below normal usage blocks production traffic on the first busy day, while a missing budget leaves a team's spend uncapped. Set each virtual key's budget from measured spend in the source systems for the same period, using provider billing exports. Confirm who approves temporary budget increases and who receives budget alerts.
Shared keys carried over
Migrating one shared key into LiteLLM keeps the same accountability gap, because spend, rate limits and model access cannot be attributed to a team or application. Issue virtual keys per team and application, restrict each to the models it needs, and map every old shared key to its replacements. Track which callers still use the shared key until it is removed.
Single proxy without Redis
A single proxy instance is simple to test but offers no failover during restarts, and several instances without shared state cannot enforce common rate limits or router state. LiteLLM's production guidance calls for Redis 7.0 or newer once more than one proxy instance runs. Load test the cutover topology, confirm PostgreSQL is in place for virtual keys, and verify that limits hold across instances.
Master key and logs exposed
A master key written into a config file gets copied into repositories, backups and tickets, and prompt content sent to a shared log destination exposes user data. Store LITELLM_MASTER_KEY, which must start with sk-, in a secret manager. Before go-live, review which callback or log sink receives request bodies and confirm access controls on each destination.
How the work changes by starting point
Applications that call provider SDKs directly need changes at each call site. The gateway exposes an OpenAI-compatible endpoint, so many changes are limited to client setup and model names, but streaming, tool calls and provider-specific parameters still need checking.
- Inventory: find every provider SDK import, base URL and hard-coded model string across repositories, batch jobs and notebooks.
- Key replacement: issue a virtual key per application or team and move provider credentials into the proxy, so applications stop holding provider keys.
- Alias design: map model names in code to stable aliases for each task, so a provider model change happens in one configuration file.
- Testing and cutover: send the same prompts through the direct and gateway paths, compare outputs and errors, then move one service at a time.
- Confirm: which SDK features your code depends on and whether the gateway passes each one through for the providers you use.
Keys spread across config files, CI variables and secret stores make discovery and ownership the main workload. Code changes are often smaller than the work of finding every place a key is used.
- Inventory: search repositories, CI settings, container images, config maps and secret stores, and record an owner and last rotation date for each key.
- Key replacement: after the proxy holds provider credentials, retire the old keys and issue virtual keys with budgets and model restrictions per team.
- Alias design: agree a naming scheme for aliases before migrating, since key-level model restrictions depend on those names.
- Testing and cutover: review proxy and provider logs for calls that still use old keys before revoking them, so unexpected breakage is found early.
- Confirm: where the master key will be stored, which secret manager you use, and whether that integration requires LiteLLM Enterprise.
Moving from another gateway is a feature mapping exercise. Each capability in the old gateway, such as routing, fallbacks, quotas, caching, logging and authentication, needs an explicit match or a documented decision to drop it. Do not assume equivalent behavior from feature names alone.
- Inventory: list every gateway feature in use, the configuration that enables it, and the applications that depend on it.
- Key replacement: map existing gateway keys to virtual keys and confirm how each budget or rate limit is enforced in both systems.
- Alias design: compare existing route names with LiteLLM model aliases and remove any that exist only as workarounds for the old gateway.
- Testing and cutover: run parallel traffic, compare routing decisions and spend reports, and keep the old gateway available for rollback until results match.
- Confirm: feature mapping for each capability in your own environment with a test, because support depends on the feature and the version you deploy.
The target environment changes the dependency and delivery work more than the application changes. Application teams see the same endpoint, but platform teams own the database, cache, image delivery and network paths.
- Kubernetes: confirm namespaces, ingress and how the master key is synced from your secret manager. Follow the production guidance on one Uvicorn worker per pod and scale horizontally, and verify this in staging.
- VM-based: plan process supervision, restart behavior and how PostgreSQL and Redis are reached from each host.
- Air-gapped: plan how images are mirrored and how signatures are verified after transfer, and how the master key is generated and stored without internet-based secret services.
- Private models: confirm that every model endpoint, including self-hosted ones, is reachable from inside the network the proxy runs in.
- Confirm: whether your team operates PostgreSQL and Redis, how images are delivered into the environment, and which provider endpoints are reachable from it.
What you need to provide
Most of the migration plan depends on inputs that only your organization can supply. The list below covers what we need from your side before the cutover sequence can be fixed, and why each item changes the plan.
Code and configuration access. Read access to the repositories, deployment manifests and configuration where application code calls providers and where keys are stored today, including environment files, CI variables and secret stores. This tells us how many call sites exist, which ones use direct provider SDKs, and which ones read keys from places that must be retired. Without it, the effort estimate for key replacement stays open.
Providers, private models and spend. A list of every provider account and private or self-hosted model in use, with current spend and request volume broken down by team or application. Usage data sets the initial virtual key budgets, rate limits and routing weights. Spend history also lets finance confirm that the gateway's cost reporting matches existing invoices before anyone relies on it.
Named owners and alias decisions. A named owner for each application and team who can approve their cutover window and sign off on test results. You also need to decide model alias naming (for example, whether applications request a stable alias such as a generic chat model, or a provider model name) and the default routing behavior for each alias. Aliases are hard to rename once applications depend on them, so this decision should be made early.
Frequently asked questions
Can our team migrate applications to LiteLLM without outside help?
Yes, many teams can, if they already run Python or container workloads and have a clear inventory of the applications and keys involved. The main drivers are the number of applications, how many provider SDKs and direct keys they use, and whether an older gateway must be retired. The skills needed include PostgreSQL and Redis operation, Kubernetes or container deployment, identity integration, secrets handling and staged cutover testing. Teams without these skills often spend their time learning the platform while application owners wait for their cutover.
How is the cost of a LiteLLM migration estimated?
Cost is set from a written scope, and DeployOpen does not publish a price list. A quote depends on the number of applications to move, the providers and private models to connect, the teams and keys to design, the state of existing keys and gateways, and the target environment, including Kubernetes, PostgreSQL and Redis. Ask the quote to separate one-time migration work from recurring costs such as infrastructure, any required licenses and ongoing support. Also ask for the deliverables and acceptance criteria for each phase.
What does a LiteLLM migration require from our team?
Your team supplies decisions, access and named owners. Expect to provide an inventory of applications, the providers and keys in use, the identity provider for admin sign-in, target infrastructure and access to it, and application owners who can test each stage. Security and platform staff usually approve key scopes, secret storage and network exposure. Ask the quote to state which tasks DeployOpen performs, which stay with your staff and when each input is needed.
Do our applications need code changes to use LiteLLM?
Usually applications change configuration more than logic. Applications that send requests in the OpenAI format typically change the base URL to the LiteLLM endpoint, replace the provider key with a LiteLLM virtual key and update model names to the aliases you define. Prompts, response handling and business logic often stay the same. Applications built on provider SDKs with different request shapes may need adapter changes. Confirm this by testing each application in a staging environment before cutover.
Does migrating to LiteLLM require an Enterprise license?
Moving applications onto the proxy does not by itself require an Enterprise license, but some governance features you may need do. Admin UI SSO supports Google, Microsoft Entra ID, Okta and generic OIDC, and from v1.76.0 it is free for up to 5 users. Beyond that limit, SSO and SCIM, JWT authentication, audit logs, organization and team RBAC and secret manager integrations for AWS, Azure, Google, HashiCorp Vault and CyberArk are Enterprise features. List your user count and required integrations, then confirm license terms with LiteLLM before the design is final.
Does DeployOpen certify our LiteLLM deployment for compliance?
No, DeployOpen does not issue audit reports, attestations or certifications. It prepares systems, controls and evidence for compliance assessments, and independent auditors or certification bodies perform the assessment. For a LiteLLM gateway, evidence can include key scoping, access logs, configuration records and change history, which DeployOpen can help organize. Before the work starts, confirm with your auditor which frameworks apply and which evidence they expect.
How DeployOpen runs LiteLLM migrations
DeployOpen moves applications from direct provider SDKs, scattered API keys or another gateway onto a self-hosted LiteLLM proxy in infrastructure you control. The work follows a written scope, a staged build and a cutover that each application team can verify.
How we scope
- Inventory every application calling a model provider directly, including SDK versions, endpoints and where keys are stored today.
- Map each current model call to a LiteLLM model alias that applications will request through the OpenAI-compatible endpoint.
- Agree in writing which teams, keys, budgets and approved models are in scope before any application changes begin.
- Confirm the migration path for applications already routed through another gateway, including request formats and logging dependencies.
- Define the cutover order, rollback criteria and sign-off owners so each application has a named accountable contact.
How we build
- Deploy the LiteLLM proxy with PostgreSQL as the primary database and Redis where more than one proxy instance runs.
- Configure provider connections and model aliases, keeping provider credentials out of application code and out of config files in plain text.
- Issue virtual keys per team or application, with budgets, TPM and RPM limits and model restrictions set from the agreed scope.
- Test each migrated application against staging aliases, comparing responses, error handling and timeouts with the original direct integration.
- Run a staged cutover one application or team at a time, keeping the previous direct path available until the agreed rollback window closes.
How we hand over
- Document the alias map, key inventory, budget settings and routing configuration as delivered in your environment.
- Provide runbooks for key replacement, adding a provider, rotating the master key and reading spend and error reports.
- Walk your platform and application teams through the cutover results, open issues and the remaining direct-SDK call sites.
- Record the secrets locations, deployment manifests and configuration sources so your own team can operate the gateway.
- Confirm with your team which items are complete and which remain open before the engagement is closed.
How we support
- Ongoing support, upgrades and continuous testing are available as a DeployOpen retainer after go-live.
- Plan version upgrades around LiteLLM's four supported minor versions, testing routing and budgets before each rollout.
- Confirm with LiteLLM whether your enterprise or open-source edition needs SSO, SCIM or audit log features for your teams.
- Review key scopes, budgets and logging settings on a schedule agreed in the retainer or the written scope.
- Ask your quote to state which support items are covered by the retainer and which are billed separately.
