vLLM as the model provider for Dify

The request path starts in Dify's Model Provider settings, where the vLLM provider is installed and given the base URL of your vLLM server, ending in /v1, along with the model name the server reports. When a workflow LLM node, agent node or knowledge base step runs, the Dify api or worker container builds an HTTP request against that base URL and sends it over the network the Docker Compose stack can reach. Trace this path end to end before go-live. Confirm that the vLLM host is reachable from the Dify containers, not only from an administrator workstation. Then check the outbound proxy. Dify's SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges, so a vLLM host on a private address may be refused. Confirm in your Dify version whether that proxy applies to model provider calls, and if it does, plan how to allow the single vLLM address without opening other internal ranges.

vLLM started with vllm serve exposes OpenAI-compatible routes under /v1: /v1/completions, /v1/chat/completions, /v1/responses, /v1/embeddings, and audio transcription and translation. Dify does not need all of them for every feature. Chat and agent style features are expected to use the chat completions route, text generation with completion-style models may use the completions route, knowledge base indexing depends on the embeddings route, and speech features depend on the audio routes. Treat this mapping as something to confirm, not assume, because the provider code and feature set change between Dify releases. Write down each feature you plan to use and the route it calls in your Dify version, then send a direct request to each route before any workflow depends on it.

The chat completions route needs a chat template. The vLLM online serving documentation states that a model must provide one, or be given one with the --chat-template option, before it can serve that API. Many instruction-tuned models ship a template, but some fine-tuned or community checkpoints do not, and the failure can appear only when Dify sends its first chat message. Add a chat request with a messages array to your acceptance test and confirm the rendered turn format matches what the model expects. Record the template source in your deployment configuration, so that a model swap or vLLM upgrade does not change chat behavior without anyone noticing.

Dify and vLLM should sit inside the same network boundary, with the vLLM server reachable only from the Dify services that need it. The --api-key option protects the /v1 routes that Dify calls, but the vLLM security documentation warns against relying on it alone and recommends a reverse proxy that allowlists only the endpoints meant for users. Endpoints such as /metrics, /health and /pause sit outside that key's protection, so keep them behind the proxy or on an internal interface that Dify users cannot reach. Store the API key in Dify's secret handling, and rotate it on both sides in the same change window so the provider does not fail mid-rollout.

How the model is served, including parallelism, batching, quantization and accelerator choice, belongs to the vLLM engine and inference scope and is not decided on this page. This page covers only the calls Dify makes once that server is running. Designing agents, prompts and tool use inside Dify, and training or fine-tuning the models themselves, are outside this page and are handled separately.

The first decision is where traffic is allowed to flow. Place the Dify containers, the vLLM host and the reverse proxy in one controlled network segment, and give vLLM no route from outside that segment. Dify's Docker Compose deployment starts the api, worker, web, plugin daemon, code sandbox and SSRF proxy alongside PostgreSQL, Redis and Weaviate, so the Dify side already spans several containers that must reach each other. The only inbound path should be the one your users need, which is the Dify web and API entry point. The vLLM server should accept connections only from Dify's outbound path and your monitoring network. Confirm that the vLLM host has no public address and that no security group or firewall rule exposes the vLLM listening port or its management paths to the internet. Write the segment layout and firewall rules down before the first model call, because a reviewed diagram is far easier to defend in an audit than a reconstruction made afterward.

The SSRF proxy is where most teams meet an unexpected failure. Dify routes outbound requests through the proxy configured by its SSRF_PROXY settings, and that proxy blocks internal and private IP ranges. Self-hosted vLLM usually sits on a private address, so the proxy can refuse Dify's calls even though both services share a segment. Do not resolve this by disabling the proxy or adding a broad bypass for private ranges, because that opens the same path to every other internal host Dify can reach. Instead, read the environment variable reference for your exact Dify version to confirm how model provider traffic is routed and whether an allowlist entry is required. If one is required, scope it to the single vLLM host and port. Then test two things: a model call from Dify must succeed, and a request from Dify to an unrelated internal address must still be blocked.

The vLLM --api-key option does not cover every path. According to the vLLM security documentation, the key protects /v1, /v2, /inference and /cohere, but not /invocations, /pause, /resume, /abort_requests, /metrics or /health. The documentation warns against relying on the key alone and recommends a reverse proxy that allowlists only the endpoints users should reach. For Dify, that means forwarding the /v1 paths Dify calls and rejecting everything else at the proxy. Metrics need separate handling. vLLM exposes Prometheus-format metrics at /metrics, covering request counts, token throughput, latency histograms and KV cache use. Let Prometheus scrape that endpoint from a dedicated monitoring network, keep /metrics off the route Dify and users reach, and confirm the scrape source addresses in your firewall rules. Restrict who can query the Prometheus server as well, because usage and latency data reveals how the model is being used.

If vLLM spans more than one node, the network boundary becomes a cluster boundary. The vLLM security documentation states that communication between nodes in a multi-node deployment is insecure by default, with no authorization or encryption, and that nodes must be protected by placing them on an isolated network. Multi-node runs use Ray or vLLM's own multiprocessing backend for this traffic, so the node-to-node network should be a dedicated segment that Dify, end users and the monitoring network cannot reach. Include that segment in your threat model as part of the trust boundary. Agree with the platform team, before go-live, on who can add nodes and how those changes are reviewed. The KubeRay sibling page covers the Ray-specific isolation details, so this section sets the requirement only. A single-node deployment avoids this node-to-node path, but it still needs the proxy and metrics controls described above.