How Dify connects to vLLM through the OpenAI-API-compatible plugin
A regulated team that wants Dify workflows calling models it hosts should connect them through the official OpenAI-API-compatible plugin from the Dify Marketplace, published by langgenius and verified by Dify. The plugin is configured with a model name, an API key and an API base URL, and it covers LLM, rerank, text embedding, speech-to-text and text-to-speech models. Each model type is added as its own entry in the workspace, so a single served model can be used for chat, retrieval or transcription, depending on what the endpoint actually provides. Settings such as completion mode, context size, token limits, streaming and vision are set per model, and they must match what the serving engine really supports, not what the model card claims.
vLLM is an open-source inference and serving engine, and its vllm serve command exposes OpenAI-compatible endpoints, including /v1/chat/completions, /v1/completions, /v1/embeddings and audio transcription and translation. That is why the plugin is the connector here and not a vLLM-specific one. The same configuration pattern works for any server that speaks the OpenAI API, so the reader should decide the endpoint contract first and then map each Dify model type to a specific route. Confirm in your own environment which routes your vLLM version serves, since rerank and text-to-speech support depend on the server and model you run. The vLLM online serving documentation lists the endpoints each release provides.
Two decisions shape the rest of this integration. The first is which base URL Dify uses: it should point at a internal address that only the Dify components can reach, not at a public listener, because the vLLM --api-key option does not protect every route. The second is which models become workspace defaults, since Dify uses those defaults in new apps and knowledge bases unless someone overrides them. Record the model name exactly as the server reports it, test each model type with one request before you set it as a default, and confirm that the API key is stored only as a Dify credential and never copied into exported app files.
What you need before connecting Dify to a model server
Connecting Dify to a model server touches four layers: the Dify deployment, the model server itself, the network path between them, and the model names and settings Dify will request. Confirm each layer before you open the plugin configuration, because most connection failures trace back to one of these foundations.
A supported Dify deployment
Run Dify with Docker Compose 2.24.0 or later on a host that meets the documented minimum of 2 CPU cores and 4 GB of RAM. The stack includes the api, worker and plugin daemon services, which make the outbound model calls, so the host must have capacity for them. Plan to track the 1.17.x release line, and confirm the exact version against the GitHub releases page before you install.
A running OpenAI-compatible server
Confirm that vLLM, or another server, is serving the models you plan to expose through the OpenAI-compatible endpoints before you touch Dify. vLLM started with vllm serve exposes chat completions, completions, embeddings and audio transcription routes. Test each route with a direct request from the Dify host, not from your laptop, so the result reflects the path Dify will use.
API key and base URL scheme
The plugin asks for an API key and an API base URL. Use the base URL that includes the version prefix the server expects, typically ending in /v1, and confirm the path with a test request. If you enable the server's --api-key option, store the key in Dify's credential settings only, and remember that this option does not protect every route, so a reverse proxy in front of the server is still needed.
Network routes from Dify services
The request originates from the Dify api and plugin daemon containers, not from the browser. Verify that those containers can reach the model host and port, including any firewall rules, VPN routes or container network settings. If Dify routes outbound requests through its SSRF proxy, check that the model host is allowed through it, since that proxy blocks internal and private IP ranges by default.
TLS and certificate trust
If the model endpoint uses HTTPS with an internal certificate authority, confirm that the Dify containers trust that CA before configuration. A certificate failure looks like a generic connection error in the plugin, so test with the same trust store Dify uses. Where traffic crosses networks, prefer encrypted transport, since the vLLM security docs state that node-to-node traffic in multi-node deployments is unauthenticated and unencrypted by default.
Served model names and capability support
Dify requests a model by the exact name the server reports, so list the models from the server's model route and copy the name character for character. Then confirm, for each capability you plan to use, that the endpoint is actually supported and enabled: chat, completion, embedding, rerank, speech-to-text or text-to-speech. A model that serves chat does not automatically serve embeddings, and the team should verify this per model before assigning it as a workspace default.
Configuring the plugin, models and workspace defaults in order
This sequence connects a self-hosted Dify workspace to models your team serves, from plugin installation through per-model settings to workspace defaults. Each step closes with a check so that a failure is found at the model it belongs to, not later in a production workflow.
Install the OpenAI-API-compatible plugin
An administrator opens the Dify Marketplace from the workspace plugin area and installs the OpenAI-API-compatible plugin published by langgenius and verified by Dify. Confirm the plugin is listed as installed in the workspace, and check that the plugin daemon is running on your deployment before continuing, since model providers depend on it. If the install fails, check outbound access from the plugin daemon to the Marketplace, because an offline or proxied network may need a separate plan agreed with your platform team.
Add the provider credentials
Still in the plugin settings, add the provider with the API base URL of your model server and the API key that your server expects. For vLLM, the base URL points at the OpenAI-compatible server path your deployment exposes, and the key must match the value set with the server's API key option. Store the key only in Dify's credential fields, not in workflow prompts or exported DSL files. The check is a successful credential validation message; if validation fails, confirm the base URL is reachable from the Dify API and worker containers, not only from your laptop.
Add the LLM model with its parameters
Add each chat model by its exact model name as your server reports it, then set the context size, maximum token limits, completion mode, streaming and vision options to match what the model actually supports. Completion mode should follow the endpoint type your server exposes for that model, and vision should stay off unless the served model accepts images. Confirm these values with the model card and your server's documentation rather than defaults. The check is a test chat in the model settings that returns a response with streaming behaving as configured.
Add the embedding, rerank, speech-to-text and text-to-speech models
Add each remaining model type the same way, one model at a time, using the model name, context limit and any vector size or language settings the plugin exposes for that type. Embedding models must be the ones your knowledge bases will be indexed with, because switching later means re-indexing the content. Confirm the dimension and context behavior with your model owner. The check for each type is a test call from its settings screen, and for embeddings, a small test indexing run on a sample document.
Set the workspace default models
In the workspace model settings, choose the default system model for each model type so new apps and knowledge bases start from approved models instead of whichever model was added first. Record the choices in your change log, because a default changed later affects apps that rely on it. The check is that creating a new blank chat app selects the intended default model without manual changes, and that the embedding default matches the one used for existing knowledge bases.
Run a single-node workflow to verify end to end
Build a minimal workflow with one LLM node that uses the new default model, run it with a short test input, and inspect the node output, token counts and latency in the run log. Then repeat the run with a knowledge retrieval node if your design uses one, to confirm the embedding and rerank paths. This check exercises the same credentials, network route and model parameters that production apps will use. Only after both runs succeed should teams build production workflows on these models.
Mapping LLM, embedding, rerank and speech endpoints to Dify model types
Dify's LLM model type covers the models that generate text for workflows, chatflows and agents. When you add an OpenAI-compatible LLM through the OpenAI-API-compatible plugin, Dify sends requests to the endpoint that matches the completion mode you select. The vLLM server exposes /v1/chat/completions for chat-style models and /v1/completions for completion-style models, so the mode must match what the served model is built to answer.
- Completion mode: choose chat for instruction-tuned models and completion for base models. A mismatch usually produces malformed or empty answers rather than a clear error.
- Context size: set the context window to the limit the served model and its vLLM startup settings actually support. Setting it higher than the server allows leads to rejected requests on long prompts.
- Token limits: the maximum output tokens should leave room for the prompt inside the context window. Retrieved knowledge base chunks and workflow variables both consume that space.
- Streaming: enable streaming only if your serving path and any proxy in front of vLLM pass server-sent events without buffering. Otherwise users see responses arrive all at once or time out.
- Vision: turn it on only for models that accept image input. A text-only model given image parts will fail the request.
Confirm with each model's documentation and a test call in your own serving stack which endpoints it supports, its maximum context length and whether it handles streaming and tool calls correctly before you set these values in Dify.
Embedding models turn document chunks and user queries into vectors for knowledge bases. Dify calls the OpenAI-compatible embeddings endpoint, which vLLM exposes as /v1/embeddings. Embedding models are configured as their own model type, so a workspace can use a different embedding model from its chat model.
- Endpoint support: confirm that the served embedding model works with vLLM's embeddings route in your version. A model that serves only chat or completions will return errors when Dify tries to index a knowledge base.
- Dimension consistency: the vector size is fixed by the model. Changing the embedding model after documents are indexed means re-indexing the knowledge base, because vectors from different models cannot be compared.
- Batch and input limits: indexing sends many chunks at once. Confirm the server's batch behavior and maximum input length, then tune chunk size in the knowledge base settings so chunks fit.
- Query and document consistency: use the same embedding model for indexing and retrieval. Mixing models silently degrades retrieval quality without raising an error.
Before indexing production content, run a small knowledge base test and check that retrieval returns the expected chunks for known questions. Ask the model's provider which embedding dimension and input length apply to your exact checkpoint.
Rerank models score retrieved chunks against a query so Dify can order them before they reach the LLM. Rerank is a separate model type in Dify and depends on a rerank-capable model being served behind an OpenAI-compatible interface. The topic facts list the endpoints vLLM exposes, and rerank is not among the core chat, completion, embedding and audio routes named there, so the path your rerank model uses must be verified rather than assumed.
- Endpoint verification: confirm the exact route and request format your rerank model exposes in your vLLM version or serving layer before entering it in Dify.
- Model choice: a model trained for reranking behaves differently from a general chat model used as a scorer. Test rank order on your own queries.
- Top K and score thresholds: these are set in the knowledge base retrieval configuration. Rerank output only helps if the candidate count passed to it is large enough to contain the right chunks.
- Latency: rerank adds a call per retrieval. Measure response time under realistic query volume on your accelerators.
- Fallback behavior: if the rerank endpoint is unreachable, check how your Dify version handles the failure before relying on it in a workflow.
What breaks when the endpoint is missing is retrieval, not the chat model: answers may still generate, but from less relevant context. Confirm the rerank route with your serving stack and check results on sample questions.
Speech-to-text and text-to-speech are separate model types in Dify that the OpenAI-API-compatible plugin can address when the served model implements OpenAI-style audio routes. The topic facts name audio transcription and translation as routes vLLM exposes, so speech-to-text maps to those routes. Text-to-speech is not listed among the vLLM routes in the supplied facts, so you must confirm it with your serving stack before configuring it.
- Speech-to-text route: confirm that the served model accepts audio uploads on the transcription route and check the supported audio formats and maximum file length.
- Language and quality: test transcription on recordings that match your users' accents, languages and recording conditions.
- Text-to-speech availability: if your stack has no OpenAI-style speech synthesis route, leave this model type unconfigured. Dify features that depend on it will not work until a compatible endpoint exists.
- Data handling: audio contains personal data in many workflows. Keep the serving host and any proxy inside your own network and confirm where uploaded audio is stored and for how long.
- Upload limits: the reverse proxy in front of the server, along with Dify's own limits, can reject large files before the model sees them.
If a speech route is missing, the failure appears when a user triggers voice input or spoken output, not at startup. Test each audio path end to end after configuration and confirm the endpoint with the model's documentation.
Which Dify settings map to which model server values
This table pairs each field in Dify's OpenAI-API-compatible plugin with the vLLM or other server value it has to agree with. Read each row as a check to run before saving the provider and again after any change to the server launch settings. The model name, base URL and endpoint path are the values most likely to cause a failed connection, so confirm those first.
| Dify setting | Server-side value it must match | What to check |
|---|---|---|
| Model name | Model identifier the server exposes for that model | Match it exactly against the server's model list before testing the connection |
| API key | Value set for the server's --api-key option | Covers /v1 endpoints only, so restrict other paths at a reverse proxy as well |
| API base URL | Server address that includes the /v1 prefix | Confirm Dify can reach the address and that the proxy forwards /v1 paths |
| Completion mode | Endpoint path: chat or completion | Chat models use /v1/chat/completions; completion mode uses /v1/completions |
| Model type (LLM, embedding, rerank, speech) | Endpoint family the server actually serves | Embedding uses /v1/embeddings; speech-to-text uses audio transcription endpoints |
| Context size | Context length configured on the server for that model | Enter no more than the limit in the server's launch settings |
| Max tokens | Output limit the server accepts per request | Keep it within the context window left after the prompt, or requests fail |
| Streaming and vision | Streaming on chat endpoints; image input support in the served model | Enable only when a test request confirms the server streams or accepts images |
| Workspace default model | Provider instance selected for each model type in workspace settings | Set it after the instance validates; apps without their own model choice will use it |
Protecting the model endpoint and Dify credentials
A vLLM server and a Dify instance each hold credentials and trust settings that the other depends on. The controls below cover what the vLLM API key does and does not protect, how traffic between nodes and from Dify is contained, and how Dify stores and signs its own secrets.
Limits of the vLLM API key
The --api-key option protects the endpoints under /v1, /v2, /inference and /cohere. It does not protect /invocations, /pause, /resume, /abort_requests, /metrics or /health, and vLLM's documentation warns against relying on the key alone. Check each of these paths from outside the network before you call the setup protected, and confirm the behavior against the vLLM version you run.
Reverse proxy allowlist for user endpoints
The vLLM documentation recommends a reverse proxy in front of the server that forwards only the endpoints meant for users, such as the chat, completion and embedding routes that Dify calls. Everything else, including metrics and health paths, stays on the internal network. Ask whoever runs the proxy to show the allowlist rules and test that blocked paths return denials.
Isolated network for multi-node traffic
In a multi-node vLLM deployment, communication between nodes is insecure by default, with no authorization or encryption. The vLLM security documentation says to protect it by placing the nodes on an isolated network. Keep the node-to-node path off the network Dify and users reach, and record who can route to it.
Dify SECRET_KEY and stored credentials
In Dify, SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials. Changing it after deployment logs out all users, so set it once, store it in your secrets manager, and back it up with the rest of the configuration. Model API keys entered in Dify are stored by the platform, so confirm where your deployment keeps the workspace encryption keys.
SSRF proxy and private addresses
Dify routes outbound requests through the SSRF proxy configured by the SSRF_PROXY settings, which blocks internal and private IP ranges. A self-hosted vLLM server usually sits on a private address, so the endpoint may be refused until you add a specific allowlist entry. Keep that entry as narrow as possible, limited to the one model host and port, and review it when the host changes.
Rotating model server keys
Rotation touches two places: the key passed to vLLM with --api-key and the matching key stored in the Dify model provider configuration. Plan a window where both are updated, then confirm that chat, embedding and rerank calls still succeed from Dify. Record the rotation date and owner, and confirm whether your process requires a restart of the vLLM server to apply a new key.
What drives the cost and effort of this integration
The cost of connecting Dify to vLLM or another OpenAI-compatible server splits across four parties: the Dify edition, the plugin work, the model server infrastructure and the people who run both. The Dify edition is the first decision. The self-hosted Community edition is free and limited to a single workspace, while the self-hosted Enterprise edition is custom priced and adds SSO, multiple workspaces and professional support, according to the Dify pricing page. If several teams need separate workspaces with different default models, that choice alone changes the quote. The OpenAI-API-compatible plugin itself is a marketplace install, so the recurring cost there is mostly the time to keep its model entries correct as endpoints change.
The larger cost sits with the model server. vLLM has no paid edition, so the spend is accelerator capacity plus the surrounding infrastructure, implementation and operations. Ask a partner to price the GPU or other accelerator footprint separately from the integration work, and confirm whether that capacity is already owned, reserved or to be provisioned. Multi-node layouts add interconnect and isolation work, because vLLM's security documentation says node-to-node traffic is insecure by default and must be protected by an isolated network. Embedding, rerank, speech-to-text and text-to-speech endpoints each need their own test pass against Dify, so the testing effort grows with the number of model types you expose.
The recurring cost is the upgrade cycle. vLLM ships new minor releases about every two weeks, so a deployment that tracks upstream will need a repeatable check each time: confirm the OpenAI-compatible endpoints still respond as the plugin expects, re-run the model-type tests and review the API key and reverse proxy rules. Dify releases also arrive roughly monthly. In a quote, request a one-time figure for the initial build and tests, a separate recurring figure for release tracking and regression runs, and a statement of which of these are fixed scope and which are retainer work. Also ask who pays for accelerator changes, since those usually come from your budget rather than the integration contract.
Common failures when Dify calls a self-hosted model server
This table pairs each failure a team typically sees when Dify calls a vLLM or other OpenAI-compatible server with its likely cause and fix. Read it while testing each model in Dify, because most failures trace back to a name, key, URL or capability flag that differs between the server and the plugin settings.
| Failure | Likely cause | Fix |
|---|---|---|
| Model not found | The model name entered in Dify differs from the name the server exposes, often because a served alias or a path prefix was used. | Query the server's model list and copy the exact served name into the Dify model configuration. |
| Authentication error (401 or 403) | The API key in Dify does not match the --api-key value the vLLM server was started with, or the key was rotated on one side only. | Align both values, restart the server after changes, and confirm the endpoint you call is under the /v1 path that the key protects. |
| Request blocked or timeout from Dify | The SSRF proxy blocks private and internal IP ranges, so an API base URL on an internal host never reaches the server. | Review SSRF proxy settings in the environment reference and confirm, with your network team, that only the intended model host is allowed. |
| Embedding or rerank call fails | The server does not expose the endpoint the plugin needs for that model type, or the model was served for chat only. | Confirm the server exposes the embeddings endpoint or a rerank endpoint, then add the model under the matching model type in Dify. |
| Context or token errors | The context size set in Dify is larger than the maximum sequence length the server was started with. | Set the context size in Dify to no more than the server's configured limit and keep max tokens within the remaining window. |
| Streaming or vision errors | The streaming or vision switch is enabled for a model that does not support it, so responses fail or images are rejected. | Turn off the flag, test with a plain prompt, and enable vision only for a model you have verified accepts image input. |
| Empty or malformed responses | The completion mode in Dify does not match the endpoint, such as chat set for a completions-only model. | Match completion mode to the endpoint the model serves, then retest a short prompt before building workflows on it. |
Questions buyers ask before connecting Dify to their own model server
How long does it take to connect Dify to a model server we run?
No supplied fact gives a duration for this connection, so treat any number you hear as unverified until it is tied to a written scope. The connection itself is a small configuration step in Dify: you add the OpenAI-API-compatible plugin, then enter a model name, API key and API base URL for each model type. The real duration depends on how much of the surrounding work is already done. The drivers are the number of model types (LLM, embedding, rerank, speech-to-text, text-to-speech), whether the endpoint is already reachable from the Dify network, how many model owners must approve settings, and how much testing you want before users depend on the models. Ask for a written breakdown that separates endpoint preparation, Dify configuration, and verification.
Can our own team connect Dify to vLLM without a partner?
Yes, a capable platform team can make this connection alone, because the Dify side is a plugin configuration and the vLLM side exposes OpenAI-compatible endpoints such as /v1/chat/completions and /v1/embeddings. The harder questions sit outside the plugin: how the endpoint is protected, which endpoints users may reach, how GPU capacity is scaled, and how model upgrades are tested. vLLM's security documentation notes that its --api-key option does not cover every route, so a reverse proxy that allowlists only user-facing endpoints is usually part of the design. A partner is most useful when you want those decisions documented, tested and handed over in writing under a fixed-scope agreement.
What does our side need to provide for the connection?
You need to provide four things: a reachable model endpoint with a stable base URL, an API key policy, network paths between the Dify host and that endpoint, and named owners for each model. The endpoint owner confirms the served model names, context lengths and whether the server supports streaming or vision for the models you plan to use. The network owner confirms which subnets and ports may reach the endpoint and whether traffic must stay inside the data center. The model owner decides which models are approved for which teams. Also check how Dify's SSRF proxy treats internal addresses, because outbound requests to private IP ranges are blocked by default and may need an explicit allowlist entry.
Does Dify Community support the OpenAI-API-compatible plugin?
The supplied facts describe the OpenAI-API-compatible plugin as published by langgenius and verified by Dify, and the Dify pricing page lists the self-hosted Community edition as free and limited to a single workspace. The facts do not state that Community and Enterprise differ in plugin support, so confirm the current plugin install path on your Dify version in the Dify Marketplace before you plan around it. What Community does not give you is multiple workspaces, SSO or the other Enterprise features listed on the pricing page. If several business units need separate model defaults, that limit matters more than the plugin itself.
How do vLLM model upgrades affect our Dify settings?
A vLLM upgrade can change the served model name, context limits, supported features or endpoint behavior, and Dify stores those values in each model entry. vLLM ships new minor releases about every two weeks according to its release history, so a schedule that re-checks settings after each upgrade is more useful than a one-time setup. After every change, confirm the model name returned by the server matches the name in Dify, re-test context size and token limits, and verify streaming and embedding calls. Keep the previous model entry until the new one passes your tests, and check the vLLM release notes for changed routes before you update the proxy allowlist.
What does a fixed-scope engagement for this connection cover?
A fixed-scope engagement covers only what the written scope lists, and most engagements of this type are agreed in writing before work starts. A typical scope for this topic would list the models to connect by type, the endpoint and proxy design, the Dify plugin configuration and workspace default models, a verification plan with test cases for each model type, and the handover documentation. It would also state what is excluded, such as GPU procurement, model training, or changes to the vLLM cluster itself. Ask for deliverables, acceptance tests, and the assumptions the quote depends on, because those assumptions determine the price and the schedule more than the plugin does.
Official sources for the plugin and vLLM endpoints
Before you enter a base URL or key in Dify, confirm the endpoint paths and settings yourself against the sources that define them. Start with the Dify Marketplace page for the OpenAI-API-compatible plugin, which lists the model types the plugin covers and the fields you must supply: model name, API key, API base URL, completion mode, context size, token limits, streaming and vision. Then check those fields against the server version you actually run, because the plugin only sends requests and cannot tell you whether a given vLLM build serves a given route. Record the vLLM version, the model name it reports and the Dify plugin version in your change record so a later upgrade on either side can be traced.
Next, read the vLLM online serving documentation to confirm which OpenAI-compatible routes your server exposes, including chat completions, completions, embeddings and audio transcription. Do not assume that a route for one model type exists just because another one does. Then read the vLLM security documentation before you expose the server to Dify. It explains that the --api-key option does not cover every endpoint, such as health, metrics and control routes, and recommends a reverse proxy that allowlists only the routes Dify needs. Plan that proxy and its allowlist as part of the deployment, not as an afterthought.
Finally, use the vLLM releases page to track the minor versions you run, since the project ships new minor releases about every two weeks. Each upgrade is a moment to re-test the endpoints Dify depends on, including embedding and rerank routes if you serve them, because response shapes and supported parameters can change between releases. Ask the team that operates your inference servers to confirm which vLLM version is in production and who owns the regression check before each upgrade.
How DeployOpen scopes and delivers a Dify model server integration
DeployOpen plans and implements self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Dify with vLLM or another OpenAI-compatible model server, we start from the models, endpoints and network paths your teams need, then deliver a configured integration that you can operate and verify yourself.
How we scope
- Engagements are fixed-scope, with the model endpoints, capability types and Dify workspaces agreed in writing before work starts
- The written scope lists which LLM, embedding, rerank, speech-to-text and text-to-speech endpoints are connected and which are excluded
- We confirm the Dify version and the vLLM or other server versions, since model compatibility depends on both sides
- Scope records the network path from Dify's API and plugin daemon to each model server, including any SSRF proxy rules needed
- Ongoing support and upgrades can be added as a retainer once the initial integration is accepted
How we build
- We install the OpenAI-API-compatible plugin from the Dify Marketplace and register each model with its name, API base URL and key
- We set completion mode, context size, token limits, streaming and vision flags per model from the model server's documented capabilities
- We place model servers behind a reverse proxy that exposes only the endpoints users need, since the vLLM --api-key option alone is not enough
- We configure workspace default models for each model type and test that fallback behavior matches the agreed design
- We keep API keys in the Dify credential store and secrets outside exported DSL files
How we hand over
- Test results for each model type, covering a sample completion, embedding, rerank and audio call where those endpoints are in scope
- A configuration record listing each model, its parameters, its network route and the owner of each key
- Runbook steps for adding a model, rotating a key, and checking the vLLM /metrics endpoint from your monitoring system
- A note on which settings were tuned for your models and which should be retested after a Dify or model server upgrade
- Walkthrough with your platform and security teams so they can run the integration without us
How we support
- Retainer support covers Dify upgrade checks, including the upgrade guide, a backup before each upgrade and review of new .env.example variables
- We review model server release notes before changing the connected version, since vLLM publishes minor releases about every two weeks
- Periodic re-tests confirm that streaming, context limits and tool calls still behave as documented after changes
- Support requests are handled inside the agreed retainer scope, with changes to model endpoints reflected in writing
- Your team keeps administrative control of the infrastructure, keys and data throughout
