Dify and LiteLLM proxy integration: routing all model traffic through one gateway
Most teams running several Dify apps against several model providers have no single place to see spend, cap usage or rotate keys, because each provider credential sits in Dify's model settings. The official LiteLLM plugin connects Dify to a LiteLLM Proxy as one OpenAI-compatible endpoint, so every app can call a single gateway that fans out to the providers behind it. The plugin is published by langgenius and verified by Dify on the Dify Marketplace. In this model the proxy, not Dify, holds the provider credentials, budgets and rate limits, and Dify only needs a key for the proxy.
The plugin supports LLMs for chat and completion, including streaming and tool calls, and text embedding models. You configure it with three inputs: the proxy base URL, a *virtual key* or master key, and the *model alias* defined in the proxy configuration. If the proxy runs without authentication, the key field stays empty, which is acceptable only on a private network segment. Confirm before go-live that every model type your apps need, especially embeddings used by knowledge bases and rerank steps, is exposed through the proxy's model list, because a missing alias fails at runtime rather than at setup.
Decide early which key scope each Dify app or workspace receives. Virtual keys can be limited to specific models and carry budgets and request limits, so a reporting app can be held to a cheaper alias than a production agent. Treat the master key as an administrative secret that stays out of Dify entirely, and plan how aliases change when you swap a provider, since app configurations reference the alias and not the underlying model. Ask your team to confirm how the proxy is backed by PostgreSQL and Redis before relying on virtual keys for spend control, because those features depend on that infrastructure.
What you need in place before connecting Dify to LiteLLM
The gateway sits between your Dify api and plugin daemon and every model provider, so its database, cache, network path and credentials need to exist before the Dify plugin is configured. Confirm each item below in your own environment, and confirm the Dify and LiteLLM versions against their release pages on the day you install.
A self-hosted Dify release
Dify runs in your infrastructure with the Docker Compose layout, which needs at least 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later. Pick the Dify release on the GitHub releases page at install time, since minor versions arrive roughly monthly. Check the upgrade guide for that version and take a backup first, because the plugin daemon that carries the LiteLLM plugin is part of that stack.
LiteLLM proxy with PostgreSQL
Virtual keys, per-key budgets and spend tracking require a PostgreSQL database, so a proxy without one can only use a master key. Run the proxy on a version you can upgrade to, and confirm the release you pick on the LiteLLM releases page. Keep LITELLM_MASTER_KEY in a secret manager, since it must start with sk- and grants full administrative access.
Redis 7.0 or newer for multiple instances
A single proxy instance can run without Redis, but the moment you run more than one, rate limits, router state and cache must be shared. LiteLLM's production guide asks for Redis 7.0 or newer in that case. Size the proxy at roughly 1 vCPU and 4 GiB of memory per worker as a starting point, then measure it under the traffic your Dify apps generate.
Provider accounts behind the proxy
The gateway holds the upstream keys, so each provider account (OpenAI, Anthropic, Gemini, Bedrock, Azure and others you approve) must exist before you define model aliases. Decide which accounts the proxy owns, who can rotate them, and which spend limits they carry. Dify will only see the aliases, so a provider change should not require Dify edits.
Network path from Dify to the proxy
The Dify api and plugin daemon must reach the proxy base URL over the network, and the proxy must reach each provider endpoint. Dify's SSRF proxy blocks internal and private IP ranges by default, so a gateway on a private address needs a deliberate allowance rather than an open route. Test reachability from inside the Dify containers, not from your workstation.
TLS and secret handling
Virtual keys and the master key travel in request headers, so the Dify-to-proxy and proxy-to-provider paths should use TLS with certificates your team controls. Confirm the Dify SECRET_KEY is set before first use, because changing it later logs out all users. Store the plugin credential only in Dify's model provider settings, and keep the proxy's own configuration out of shared repositories.
How to configure the LiteLLM plugin in Dify, step by step
This sequence moves from proxy configuration to verified Dify model calls, so every model alias Dify uses is defined in the gateway first and every step closes with a check you can see.
Define model aliases in the proxy configuration
Before touching Dify, decide which models your teams may use and give each one a stable alias in the LiteLLM proxy configuration, mapping the alias to a provider model you have approved. Keep aliases stable and descriptive, because Dify stores the alias name, not the provider model, so renaming one later breaks every app that uses it. Your platform owner and the model owner should sign off on this list. You are done when the proxy starts cleanly and a direct request to the proxy using one alias returns a completion from the intended provider.
Create a test virtual key scoped to the aliases
Create a virtual key in the LiteLLM proxy that is restricted to the aliases from the previous step and nothing else. Virtual keys require a PostgreSQL database, so confirm the proxy is backed by one before you start. Set a conservative budget and rate limit for this test key, since it is only for validating the integration. Store the key in your secret manager rather than in a notes file. You are done when a request using this key succeeds for an approved alias and is rejected for an alias outside its list.
Install the LiteLLM plugin from the Dify Marketplace
In the Dify console, open the Marketplace and install the LiteLLM plugin published by langgenius, which Dify verifies. Dify runs plugins through its plugin daemon, so confirm your self-hosted instance can reach the marketplace or that you have an approved offline plugin process for your environment. Install it into the workspace that will use the models. You are done when the plugin appears in the workspace's installed plugins with the expected publisher.
Enter the proxy base URL and key
Open the LiteLLM plugin configuration and enter the base URL of your proxy, using an address the Dify api and worker containers can reach on your network. Enter the test virtual key in the key field. Leave the key field empty only if the proxy runs without authentication, which is rarely appropriate for a shared gateway. Save the configuration and confirm the plugin reports a successful connection. You are done when the save succeeds without a connection error and the proxy access log shows the connection attempt.
Register each alias as a Dify model
Add each approved alias as a model under the LiteLLM provider, selecting the model type that matches its role: chat or completion for language models, and text embedding for embedding models. Use the alias exactly as defined in the proxy configuration, since a mismatch produces failed calls that look like provider errors. Enter any context and output limits you have confirmed for that model. You are done when each alias appears in the model list with the correct type and a test prompt in the Dify model test panel returns output.
Set workspace default models
Choose default chat and embedding models for the workspace from the aliases you registered, so new apps and knowledge bases inherit gateway-routed models instead of a direct provider key. Review whether any existing app still points at a direct provider credential and reconfigure it to use an alias. Record which alias is the default and who approved it. You are done when a newly created test app uses a gateway alias by default and an existing app's model panel shows the alias rather than a direct provider.
Confirm calls appear in proxy logs
Run a test workflow or chat in Dify that calls each registered alias, including one streaming request and, if your use case needs them, one tool-calling request, since the plugin supports both. Then open the LiteLLM proxy logs or spend view and find each call under the test virtual key with the expected alias and model. Check that a call outside the key's allowed aliases is refused. You are done when every alias produces a visible proxy record and the scope restriction behaves as configured.
Replace the test key with a planned key structure
Once the integration is verified, replace the test key with keys issued per application or team, each with its own budget, rate limits and alias list, and store the master key in a secret manager. Remove the test key from the plugin configuration and revoke it at the proxy. Ask your LiteLLM admin about the budgets and limits you will need, and confirm which features of your chosen LiteLLM edition you rely on. You are done when Dify uses only planned keys, the test key returns an authorization failure, and the key inventory is recorded.
How the gateway changes spend tracking, budgets and failover
A LiteLLM proxy changes who decides how much a Dify app can spend and where its requests go. Dify only sees the model alias you expose through the LiteLLM plugin. Every budget, rate limit, model restriction and routing rule lives in the LiteLLM proxy, so your team has to configure each of them there. Nothing in the Dify console enforces them by default. A direct provider key gives you one shared bill and no per-application limits, while a virtual key issued by the proxy can be scoped to a single Dify workspace or application, with its own budget and model list. Virtual keys require a PostgreSQL database in the proxy, so plan that database before you issue keys.
The proxy controls several limits that Dify cannot set on its own. Per-key, per-user and per-team budgets cap spend, while TPM and RPM limits cap token and request throughput. Temporary budget increases let an owner raise a limit for a set period without reissuing the key, and restricting a key to specific models stops a Dify app from calling a model it was never approved to use. Confirm these settings in the LiteLLM virtual keys documentation for your proxy version, and test each one with a deliberately low limit before production. Decide who may approve budget increases and record that decision outside the proxy.
Failover is also a LiteLLM setting. The router can use simple-shuffle (the recommended strategy), usage-based, latency-based, least-busy or cost-based selection, and it can retry failed calls, cool down a deployment after repeated failures, order fallbacks by priority and keep sessions on the same deployment through session affinity. Pick the strategy with your cost and latency targets in mind, then rehearse a provider outage by disabling one deployment and checking that Dify answers still arrive. Dify does not know which provider served a request, so confirm that your logs and the LiteLLM routing documentation give you the evidence you need.
Single proxy or multi-instance: which layout fits your team
A single LiteLLM proxy instance is the simplest layout for a pilot, a proof of concept or one Dify environment serving a small group of builders. The proxy still needs a PostgreSQL database if you want virtual keys, since LiteLLM's virtual keys docs require it for per-key, per-user and per-team budgets and rate limits. A single instance keeps router state and cache in memory, so Redis is optional at this size. Use the pilot to confirm how your models, aliases and budgets behave before you add redundancy.
- Database choice: Run PostgreSQL as the state store from the start. Do not plan around SQLite for anything that holds virtual keys or spend records.
- Redis not yet required: With one proxy instance, shared rate limits and router state across replicas do not apply, so Redis can wait until a second instance exists.
- Narrow key scope: Issue virtual keys restricted to the model aliases the pilot needs, so a test app cannot reach every provider.
- Documented upgrade path: Record the configuration and database schema version so moving to several instances later is a planned change.
Before you rely on a pilot's numbers, confirm with your LiteLLM version which features you depend on, using the LiteLLM production best practices as the reference for the single-instance baseline.
Production usually runs more than one proxy instance so that a single failure does not stop every Dify app. Once more than one instance runs, the proxy needs shared state. LiteLLM's production guide requires Redis 7.0 or newer for shared rate limits, router state and cache when more than one instance runs. Without it, each replica would count limits and routing health on its own, so a key could exceed a limit it should have been stopped at.
- Shared Redis: Provision Redis 7.0 or newer as a managed or self-operated service, with the same availability target as the proxy.
- PostgreSQL as primary store: Keep PostgreSQL as the primary supported database for keys, teams, budgets and spend logs across all instances.
- Consistent configuration: Deploy the same model list, aliases and routing settings to every instance, and roll changes out in one controlled step.
- Load balancer health checks: Route Dify traffic through an internal load balancer that removes unhealthy instances from rotation.
- Sizing per worker: The production guide suggests 1 vCPU and 4 GiB memory per worker as a starting point, which you should validate against your traffic.
Confirm with your platform team how Redis failover affects rate limits and routing before go-live, and check the LiteLLM routing docs for the router behavior your chosen strategy depends on.
On Kubernetes, the LiteLLM production guide recommends running one Uvicorn worker per pod and scaling horizontally by adding pods, instead of raising the worker count inside each pod. This keeps resource requests and limits predictable and lets the orchestrator replace a failed pod without affecting its neighbors. It also means every pod must share the same Redis and PostgreSQL endpoints, so scaling is only safe once that shared state is in place.
- One worker per pod: Set the Uvicorn worker count to one in the container command and size CPU and memory requests per pod.
- Horizontal scaling: Use a horizontal pod autoscaler or manual replica counts, and test that new pods join with the correct configuration.
- Readiness checks: Gate traffic on readiness so a pod that has not connected to the database and Redis receives no Dify requests.
- Shared endpoints: Point every replica at the same database and Redis services, and confirm network policies allow those connections.
- Rolling changes: Test that config reloads and rolling updates do not drop in-flight streaming responses for Dify chat apps.
Confirm with your cluster team how the proxy handles pod termination during a streamed model response, since that depends on your ingress and termination settings.
The master key is the most privileged credential in a LiteLLM deployment. LiteLLM's production guide says LITELLM_MASTER_KEY must start with sk- and should be kept in a secret manager rather than in a plain configuration file or image. Anyone holding it can create virtual keys, change budgets and read spend data, so its handling matters as much in a single-instance pilot as it does at scale. Dify itself should only hold virtual keys scoped to the models each app needs.
- Secret manager storage: Store the master key in your organization's secret manager and inject it at runtime into each pod or container.
- Virtual keys for Dify: Create one virtual key per Dify environment or team, instead of putting the master key into the Dify LiteLLM plugin settings.
- Rotation plan: Define who rotates the master key, how often, and how running instances pick up the new value without a long outage.
- Access logging: Restrict who can read the secret, and record access through your secret manager's audit trail.
- Admin access: Limit who can use the admin UI, and review SSO options with the LiteLLM admin UI SSO docs before exposing it.
Confirm with your security team which secret manager integrations your LiteLLM version and edition support, because some secret manager integrations are listed under LiteLLM's Enterprise features.
Every model call through the proxy can generate a spend update in PostgreSQL. At low volume this is negligible, but under sustained Dify traffic the write load grows with request count. LiteLLM's production guidance describes batching spend updates to the database every 60 seconds at scale, which trades a short delay in spend visibility for lower database write pressure. Budgets and spend reports therefore reflect activity with a lag, and that lag should be part of how your finance and security teams read the numbers.
- Batching window: Treat the 60-second batching interval as the default to validate, since spend shown in dashboards may trail real calls by up to that window.
- Budget headroom: Set per-key budgets with margin, so a batching delay cannot let a team overrun a hard limit unnoticed.
- Database capacity: Size PostgreSQL connections and storage for spend log growth, and plan retention for those logs.
- Reporting baseline: Reconcile proxy spend records with provider invoices on a regular schedule, not only when a budget alert fires.
- Load testing: Run a representative Dify workload against the proxy before go-live to see how write volume behaves in your environment.
Ask your LiteLLM contact or check the LiteLLM releases page which spend batching behavior applies to your version before you set budget alerts.
Which Dify and LiteLLM settings map to each other
This table matches each field in the Dify LiteLLM plugin to its counterpart in the LiteLLM proxy, with the owner of each value and what changes when it moves. Read it before you create keys or rename aliases, because a change on the proxy side must be mirrored in Dify.
| Dify plugin field | LiteLLM counterpart | Who owns the value | Effect of rotation or renaming |
|---|---|---|---|
| Base URL | Address of the LiteLLM proxy server, including its OpenAI-compatible path | Platform team that runs the proxy | Changes only when the proxy moves or gets a new hostname; update the plugin configuration in each workspace that uses it |
| Virtual or master key | Virtual key issued per app, team or user, or the master key set through LITELLM_MASTER_KEY | Platform team issues keys; the master key should stay in a secret manager | Rotating a key breaks every Dify model that uses it until the new value is entered; master keys should not be shared with app builders |
| Model alias | model_name entry in the proxy configuration that routes to one or more provider models | Platform team defines aliases; app builders select them in Dify | Renaming an alias on the proxy makes the Dify model fail until the alias is reselected; keep old aliases during a cutover |
| Model type (LLM or text embedding) | Chat or completion model entry versus embedding model entry in the proxy configuration | Dify builder selects, proxy team confirms the entry is an embedding model | Embedding models change vector dimensions; switching an embedding alias to a different model usually means re-indexing knowledge bases |
| Authentication disabled | Proxy started without a master key or virtual key requirement | Platform team decides, usually only for isolated test setups | Enabling authentication later means every Dify model entry must receive a key, or calls will be rejected |
| Fallbacks and routing | Router settings such as priority ordering and retries in the proxy | Platform team configures; Dify sees only the alias | Changing fallback order changes which provider answers, so re-check output behavior and data handling for each provider |
Securing the Dify to LiteLLM path
Every Dify model call that passes through LiteLLM carries credentials, prompts and spend data across two services. The cards below cover the decisions that keep that path scoped, encrypted and reachable only by the services that need it. Each card names a setting to review or a question to put to the vendor before go-live.
Scope keys per Dify use
Give Dify a LiteLLM virtual key rather than the master key. A virtual key can carry its own budget, request limits and a list of allowed models, so a misconfigured or compromised Dify workspace cannot reach every provider behind the proxy. Create one key per Dify environment or workspace where you need separate spend tracking. Confirm in your LiteLLM version which limits apply to embedding models, since the Dify plugin uses both LLM and embedding calls.
Protect the master key
LITELLM_MASTER_KEY is the administrative credential for the proxy and must start with sk-. Store it in a secret manager and inject it at runtime, not in the Dify .env file or in a Compose file under version control. Limit who can read it, because it can create virtual keys and change spend limits. Rotate it on a schedule you can document, and confirm how rotation affects existing virtual keys in your proxy version.
Limit outbound reach with SSRF_PROXY
Dify's SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges. A LiteLLM proxy on your internal network is a private address, so it will not be reachable through that filter by default. Add a narrow allowance for the proxy host and port only, and confirm the exact configuration against the Dify environment reference for your release. Keep every other internal host blocked so a workflow HTTP node cannot reach the same network.
Encrypt traffic between Dify and the proxy
Prompts, completions, tool arguments and virtual keys cross the link between Dify and LiteLLM in each request. Terminate TLS at the proxy or an internal load balancer and set the Dify base URL to the https endpoint. Use a certificate issued by an internal CA that the Dify containers trust, and test the trust chain from inside the api and worker containers, since a failure there shows up only as failed model calls.
Understand the SECRET_KEY dependency
Dify uses SECRET_KEY to encrypt OAuth credentials and to sign sessions. Provider API keys and virtual keys saved in Dify are stored encrypted, so the LiteLLM key is only as safe as the Dify secret that protects it. Changing SECRET_KEY after deployment logs out all users, and you should confirm with your Dify version whether stored model provider credentials can still be decrypted after a change. Back up the value with your database backup procedure.
Review LiteLLM Enterprise controls
The free admin UI SSO covers up to 5 users from v1.76.0, and audit logs, IP allowlisting, SCIM and secret manager integrations with AWS, Azure, Google, HashiCorp Vault and CyberArk sit in the Enterprise tier. If your security team needs named admin accountability, IP restriction on the proxy or managed secret retrieval, ask LiteLLM for a current quote and confirm which features your deployment model includes. Treat these as options to verify, not as settled controls.
How to test that every model call goes through the gateway
The test you care about is whether any Dify call can reach a model provider without passing through the LiteLLM proxy. Start by inventory: open each Dify app and workflow, and confirm that every LLM node, agent node and knowledge base embedding setting points to a model alias defined in the proxy configuration rather than a provider the workspace configured directly. Repeat the check for the workspace default models, because a default that still points at a direct provider will quietly bypass the gateway for new apps. Record the alias list as the expected state, then compare it with what the proxy's request logs show after a short test session.
Then run functional tests through the gateway. Send a streaming chat request and a request that triggers a tool call through an agent or a workflow with a tool node, and confirm both complete and appear in the proxy logs. Run an embedding test by indexing a small test document into a knowledge base that uses a proxy embedding alias, then run a retrieval query against it. Each test should produce a spend entry tied to the virtual key you assigned to that Dify workspace or app, so you can confirm attribution as well as delivery.
Next test the controls. Deliberately exceed a small test budget on a non-production virtual key, or restrict a key to a single model and request a different one, and confirm the proxy rejects the call with a clear error that Dify surfaces to the user. Simulate a provider outage by disabling or misconfiguring one provider behind a model group, then confirm the router falls back to the next deployment in the priority order you defined. Finally, remove or restrict the provider API keys that Dify or other systems could still use directly, and attempt a call with them to prove the bypass path is closed. Confirm spend totals per key and per team in the admin views before you sign off.
Common LiteLLM and Dify failures and their fixes
This table pairs the failures teams most often hit when Dify calls a LiteLLM proxy with the cause behind each one and the fix to try first. Read it by symptom: match the error you see in Dify or the proxy logs to a row, then check the cause before changing configuration. Most failures trace back to a mismatch between Dify's plugin settings and the proxy's own configuration.
| Symptom | Likely cause | Fix |
|---|---|---|
| Plugin connection refused | Dify cannot reach the proxy base URL from its api or worker containers, often because the URL uses localhost or a wrong port, or TLS is misconfigured | Use an address the Dify containers can resolve and reach, such as the proxy service name on the shared Docker network or an internal DNS name. Confirm the scheme, port and certificate trust chain match what the proxy serves |
| Authentication error from the proxy | The virtual key in the plugin is wrong, expired, revoked or copied with extra whitespace, or a master key is expected but not entered | Generate or confirm the key in the proxy, paste it into the plugin without spaces, and check that the key is allowed to use the model alias in use. Leave the key empty only if the proxy has no authentication enabled |
| Model not found | The alias entered in Dify does not match the model_name defined in the proxy configuration, so the request names a model the proxy does not route | Copy the model alias exactly from the proxy configuration into the Dify model entry, including case and any prefix. Test the alias with a direct request to the proxy before retrying in Dify |
| Streaming responses cut off | A reverse proxy, load balancer or the LiteLLM side has a read or idle timeout shorter than a long generation, so the connection closes mid-answer | Raise the read and idle timeouts on every hop between Dify and the proxy, and confirm that buffering is disabled for streamed responses. Test a long streamed completion end to end |
| Embedding call fails | An alias configured for chat is used as the embedding model in a knowledge base, or the proxy route does not point to an embedding-capable model | Add a separate alias for the embedding model, register it under the text embedding type in the plugin, and confirm the proxy route resolves to an embedding endpoint |
| Budget exceeded error | The virtual key, user or team has reached its budget or its rate limit, which the proxy enforces before the provider is called | Check the budget and limits on that key in the proxy, then either raise the budget through the proxy's temporary increase option or move the workload to a key with capacity. Set alerts so teams see the limit before users do |
| Spend or usage data missing | PostgreSQL is not configured for the proxy, and virtual keys need a database for spend tracking, budgets and per-key records | Configure a PostgreSQL database for the proxy and confirm the proxy starts with it connected. Note that calls made before the database was attached are not recovered, so record the change date for audit purposes |
Questions teams ask before connecting Dify to LiteLLM
What drives the effort of connecting Dify to a LiteLLM proxy?
The effort is driven mostly by how many providers sit behind the proxy, how many model aliases you define, and how much policy you attach to each key. Dify's LiteLLM plugin itself is configured with a base URL, a key and a model alias, so the Dify side is small. The larger share of work is on the gateway: provider accounts, alias naming, virtual key design, budgets, rate limits, fallback order and logging. Setting up PostgreSQL is also required once you use virtual keys, and Redis becomes necessary once more than one proxy instance runs. Most of this is one-time design and build. Recurring work includes rotating keys, reviewing spend, adjusting budgets and testing fallbacks after provider or model changes.
Can our team connect Dify to LiteLLM alone using the documentation?
A team with container and database experience can get a basic connection working from the Dify Marketplace plugin page and the LiteLLM docs. The harder part is the production layer: a proxy that stays up, a database and Redis configured for shared state, a secret manager holding the master key, virtual keys mapped to real owners, and routing that fails over correctly. Those choices need someone who owns the budget and the security review, not only someone who can paste a base URL. Ask yourself whether you have engineers who will maintain the gateway after launch, because the connection is only as reliable as the proxy behind it.
What does our organization need to provide before the work starts?
The work needs several inputs from your side before anyone configures a key. You need accounts and contracts with each model provider you plan to route to, along with a decision on which providers are approved for which data. Your network team must confirm that the Dify api and worker services can reach the proxy, and that the proxy can reach each provider endpoint or private model server. Name an owner for each virtual key and team, and a finance or platform owner who sets budgets and approves increases. Security should decide where the master key lives and who may read it.
How long does a LiteLLM connection take to set up?
No duration is fixed for this work, and the timeline depends mainly on the number of providers, the number of model aliases and the policy depth you require. A proxy with one provider and a few aliases is a much smaller job than one with several providers, private models, per-team budgets and SSO for the admin UI. Ask for the timeline in a written scope that lists each provider, each alias, the key and budget model, the fallback rules and the verification steps. Scope-based timelines are agreed in writing before work starts.
Does LiteLLM have to be an Enterprise license for this setup?
No. The core proxy is open source under the MIT license outside the enterprise directory, and the virtual keys, budgets, rate limits and routing that this integration depends on are part of the open-source proxy, with a PostgreSQL database for virtual keys. An Enterprise license becomes relevant for specific features. The admin UI SSO is free for up to 5 users from v1.76.0, and an Enterprise license is needed beyond that. SCIM, audit logs, organization and team RBAC, IP allowlisting and secret manager integrations are Enterprise features. Confirm the current feature split against the LiteLLM Enterprise documentation before you commit to a design.
Does Dify Enterprise change how LiteLLM is connected?
The connection method stays the same. The Dify LiteLLM plugin is configured with the proxy base URL, a key and a model alias whether you run Dify Community or Dify Enterprise. What changes is the platform around it. Dify Enterprise adds SSO, multiple workspaces and Kubernetes deployment for high availability, and its resources checklist lists Kubernetes, PostgreSQL, Redis, a Qdrant vector database and object storage as requirements. Those choices affect where the Dify services reach the proxy from and how workspaces receive model access, so plan network paths and workspace model settings together.
What should we ask for in a quote for this integration?
Ask for a written scope that lists each provider and model alias to be routed, the deployment target for the proxy and its database and Redis, and the key model with owners for master, team and virtual keys. Request the budget and rate limit design, the fallback and retry rules, the logging and spend tracking approach, the secret storage method, and the verification steps for each provider. Also ask which items are one-time and which recur, such as retainer support, upgrades and key reviews. Confirm who handles provider account setup and whether any Enterprise license cost falls outside the quote.
How DeployOpen scopes and delivers a Dify LiteLLM integration
DeployOpen begins every Dify and LiteLLM engagement with a written scope agreed before any work starts. That scope names the Dify workspaces and apps that will call models through the gateway, the LiteLLM proxy that will sit between them and the providers, the Dify LiteLLM plugin version that will be installed, and the environment where each piece runs. Because most engagements are fixed-scope, the deliverables and the acceptance checks are written down at this stage, so the client knows what is included before a quote is accepted. Confirm with DeployOpen which pieces are in scope, such as the proxy's database and Redis, the admin UI sign-in, and the Dify side configuration for each workspace.
The design work happens with the client before anything is deployed. DeployOpen plans the model alias map so that Dify apps reference stable names rather than provider model IDs, decides which team owns each virtual key, and agrees the budgets, TPM and RPM limits and model restrictions for each key, user or team. It also settles where the LITELLM_MASTER_KEY is stored and who may rotate it. These decisions shape how spend is reported and who is notified when a budget is reached, so they are better made once, in writing, than adjusted after the first invoice from a provider.
DeployOpen then deploys the proxy and configures the plugin inside the client's own cloud accounts, data centers or air-gapped network, and tests the result against the agreed checks, including streaming, tool calls, embedding models and fallback behavior when a provider is unavailable. The handover includes documentation of the alias map, key ownership and runbook steps. Ongoing work such as proxy and plugin upgrades, provider changes and budget reviews can be covered by a retainer. Ask for the upgrade and support terms in writing, and confirm them against your own change control process.
