Dify with GPUStack GPU clusters: what the integration does
A team that wants Dify apps to call models running on its own GPU servers, instead of an external API, can connect the two through the GPUStack plugin listed on the Dify Marketplace. The plugin's listing describes GPUStack as an open-source GPU cluster manager for running AI models, and that plugin is the supported path to confirm before any build starts. The division of work is simple to state: GPUStack decides where each model runs across the GPU servers in the pool, and Dify consumes the resulting endpoints. Dify then assigns those endpoints to individual workspaces and apps, so the platform team controls capacity while application teams choose from the models they have been given.
The details that matter for planning are not verified in this section. Confirm the exact plugin settings, the model types the plugin supports (chat, embedding, rerank or others), and the endpoint format against the plugin listing and the GPUStack release you intend to run, since those can change between releases. Ask the vendor or your integration lead to show a working connection in a test workspace before you commit to a model layout. Also confirm where API credentials for GPUStack are stored and who can view or rotate them inside Dify, because those keys will sit alongside keys for cloud providers in the same console.
Treat GPUStack as a scheduling layer that sits between your hardware and Dify, and plan for the failure modes that come with that position. A model that GPUStack moves or unloads can change latency for every app that depends on it, and a shared GPU pool means one team's batch job can slow another team's chat workflow. Before rollout, decide which models are reserved for production apps, which are available for experiments, and how a model change is announced to the workspaces that use it. Keep the Dify release you run in step with the GPUStack release you support, and test both together after each upgrade.
What must be in place before connecting Dify to GPUStack
Connecting Dify to GPUStack touches four systems: the Dify containers, the GPUStack cluster, the GPU servers beneath it, and the network path between them. Each group below is a condition to verify before you add the provider, so that a failed connection can be traced to one layer rather than guessed at.
A working self-hosted Dify
Start from a Docker Compose deployment that already passes its own health checks. The self-hosting docs set a floor of 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later, and the stack includes the api, worker and plugin daemon that will call GPUStack. Confirm those services are running before you change anything, and take a backup of the database and volumes so you can roll back a failed plugin install.
Version pairing for Dify and the plugin
Confirm that the GPUStack plugin listing supports your installed Dify release before you install it. Dify minor releases arrive roughly monthly, so a plugin verified against one version may behave differently on another. Check the plugin's Marketplace page and the Dify releases page, and record both version numbers in your change ticket so the pairing can be reproduced during an upgrade.
A reachable GPUStack cluster
Decide which GPUStack endpoint Dify will call and confirm it resolves and responds from inside the Dify api and worker containers, not only from your workstation. Containers often have different DNS resolvers and routing than the host. Test from inside each container with the same hostname and port you plan to enter in the provider form, and confirm the cluster's API is up before you blame the plugin.
GPU server driver and runtime readiness
Each GPU server in the pool needs a working driver and container runtime configuration before GPUStack can schedule models onto it. Confirm that the GPUs are visible to the host and that the runtime exposes them to containers on each node, and that the driver versions match what your chosen inference backend expects. A node that fails this check will still appear in the cluster but may not run models, so verify every node, not just the first one.
An admin account for API access
Create a dedicated administrator or API-capable account in GPUStack for the Dify connection, rather than reusing a personal login. Name it for the integration, restrict it to the models and projects Dify needs if your GPUStack version supports that scoping, and record who owns it. The API key or credential you generate is the secret Dify will store, so plan where it is kept and how it is rotated when staff change. Confirm the exact role options in the GPUStack documentation for your version.
Firewall rules and TLS trust
Open only the ports the Dify containers need to reach the GPUStack endpoint, and document the rule in both directions if your network policy requires it. Then decide how the endpoint is secured. If GPUStack uses HTTPS with a private certificate authority, the CA must be trusted inside the Dify containers, not only on the host, or the provider connection will fail certificate validation. Also check that Dify's outbound SSRF proxy settings do not block the private address of your GPUStack server.
Configuration sequence for the GPUStack provider
This sequence connects a GPUStack cluster to Dify in a controlled order, so each model is verified in a test workspace before it reaches production apps. The result is a provider configuration with scoped credentials, registered model types, and a logged test run that your team can repeat.
Confirm the GPUStack cluster and models are healthy
Before touching Dify, the infrastructure team checks that the GPUStack cluster is up, that each worker GPU server is reporting, and that every model you plan to expose is in a running state. Confirm the endpoint is reachable from the network segment where Dify's api and worker containers run, not only from an administrator workstation. Done means you can list the deployed models through GPUStack's own interface and get a response from one of them over the endpoint Dify will use.
Install the GPUStack plugin in a test workspace
A Dify administrator installs the GPUStack plugin from the Dify Marketplace into a test workspace rather than a production one. The Marketplace listing describes GPUStack as an open-source GPU cluster manager for running AI models. Confirm the plugin version against your Dify release and check it appears in the plugin list for that workspace. Done means the GPUStack provider card is visible in the model provider settings of the test workspace only.
Create a scoped GPUStack API key
The GPUStack administrator creates a dedicated API key for Dify rather than reusing a personal or administrator key. Record which models, users or access scope the key is meant to cover, and ask the vendor or your own configuration how keys are revoked and rotated. Store the key in your secrets manager, not in a shared document. Done means the key is recorded with an owner and a rotation plan, and a call with that key to the model list endpoint returns only the models Dify should use.
Add the provider in Dify with endpoint and key
In the test workspace, the Dify administrator adds GPUStack as a model provider, entering the cluster endpoint and the scoped API key. Use the internal address your network team has approved, and keep the connection on a path that stays inside your environment. Confirm the field values match the endpoint used in the check from step one. Done means Dify reports the provider as configured without a credential error.
Register each model with its type
Add each GPUStack model to Dify and assign its type: chat, embedding or rerank, as the model is offered by the cluster. A model used for embeddings registered as a chat model will fail or return poor results, so check the type against what GPUStack reports for that model. Name models so that a reader can tell the serving host, size and purpose apart. Done means every registered model appears under the correct type in Dify's model list.
Run a test prompt from a Dify app
Create a test app in the test workspace, select a registered GPUStack chat model, and send a representative prompt drawn from the job the model is meant for. For embedding or rerank models, run a small knowledge base indexing or retrieval test instead. Confirm the response is returned and that the run appears in the app's logs. Done means a successful test chat and a log entry showing the GPUStack model was used and token usage was recorded, which you can compare against your GPUStack capacity view.
Set workspace defaults after testing
Only after the test runs succeed, set the workspace default models for reasoning, embedding and rerank, since apps fall back to these defaults. Record which GPUStack model is the default and why, because changing an embedding default later affects existing indexes. Done means each default is documented and a fresh test app picks up the expected model without manual selection.
Promote the configuration to production
Repeat the provider setup in the production workspace using the same model registrations and a separate production key issued under the same scoping rules. Rerun the test prompt and confirm the log entry appears in production. Keep the test workspace configuration for future model changes. Done means production shows the GPUStack provider, the registered models and a successful production test run with a logged entry.
How GPUStack models and endpoints map to Dify settings
This table matches each GPUStack object, from cluster down to endpoint and API key, with the Dify setting it feeds, where that value is entered and what breaks when it changes. Read it as a checklist before wiring a model, and confirm exact field names against the GPUStack plugin version you install, since labels can differ between releases.
| GPUStack concept | Dify concept | Where the value is set | What breaks if it changes |
|---|---|---|---|
| Cluster | Provider instance (one per GPUStack cluster) | Dify model provider settings for the GPUStack plugin, using the cluster address entered during setup | Every model under that provider fails to respond if the address, network route or TLS trust changes |
| GPU server | No direct Dify setting | Managed in GPUStack only; Dify never sees individual servers | Removing or draining a server can reduce capacity and slow responses, but Dify settings stay valid |
| Model (deployed name) | Model entry under the provider, with model type set to chat, embedding or rerank | Model name field in the Dify provider configuration; must match the deployed name in GPUStack | A renamed or unpublished model causes errors in every app and workflow that references the old name |
| Model instance (replica) | Not exposed to Dify; Dify calls the model through the endpoint | GPUStack scheduling and replica count | Scaling replicas changes throughput, not configuration; confirm the endpoint stays the same when replicas move |
| API key | Credential stored on the GPUStack provider in Dify | Dify provider credential field; keep one key per environment where possible | A rotated or deleted key stops all calls through that provider until the Dify credential is updated |
| Endpoint (base URL) | Provider endpoint used by every model entry | Provider base URL field in Dify, including the path GPUStack exposes for the model API | Changing host, port or path breaks calls, and a mismatched path can return errors that look like model failures |
| Model type and capability | Model type (LLM, text embedding, rerank) and workspace default | Model type selection in Dify; workspace default model settings for each type | Assigning an embedding model as a chat default, or changing the embedding model later, affects knowledge base retrieval and may require re-indexing |
Choosing a GPUStack model for each job
Conversational apps send the most frequent and latency-sensitive requests to a model, so the chat model usually sets how responsive the application feels to users. The model must follow instructions reliably, hold context across the conversation window you allow, and respond in the language and tone your users expect. Larger chat models generally need more GPU memory and produce tokens more slowly on the same hardware, while smaller ones free capacity for other models but may handle complex instructions less consistently.
- Memory footprint: The weights and the key-value cache for concurrent sessions both draw from GPU memory, so concurrency limits often arrive before raw speed does.
- Latency under load: Response time climbs when several users share one GPU, so test with the number of simultaneous sessions you expect, not one user.
- Swapping later: Changing the chat model usually changes answer style and prompt behavior, so prompts and evaluation sets need retesting, but existing knowledge bases stay valid.
- Context length: Longer context windows cost more memory per session, so set limits in Dify to match what the app actually needs.
- Data handling: Confirm that the model runs entirely on your GPU servers and that no request falls back to an external provider.
Before you set this model as a workspace default, run your own prompts and sample conversations through the candidates and compare answer quality and response times at your expected concurrency.
An embedding model converts document chunks and user questions into vectors so that Dify's vector store can find related passages. It runs during ingestion for every file you upload and again for each query. Its quality determines whether the right passages are found at all, and its output dimensions determine how much storage the vector index needs. Embedding models are usually smaller than chat models, but ingestion of a large document set can keep a shared GPU busy for a long time.
- Ingestion load: Bulk uploads queue many embedding calls, so schedule large imports away from peak conversational use.
- Vector dimensions: Output size affects index storage and query cost in the vector store you choose.
- Re-indexing on change: Vectors from different embedding models are not comparable, so changing the model means re-embedding every existing knowledge base, and that work recurs with each change.
- Language coverage: Confirm the model handles the languages in your documents, including tables and technical terms.
- Data residency: Chunks of confidential documents pass through this model, so keep it on your own GPUs.
Before you choose an embedding model, benchmark retrieval on a sample of your own documents and typical questions, and record the re-indexing effort you would accept if the model later changes.
A rerank model scores a short list of candidate passages against the question and reorders them before the chat model reads them. It improves precision when the first retrieval pass returns many loosely related chunks. Because it runs on each query, its latency adds directly to response time, but it processes only a limited number of candidates, so its memory use is usually modest compared with a chat model.
- Per-query cost: Every retrieval call triggers scoring, so latency grows with the number of candidates you ask it to rerank.
- Candidate count: Reranking more candidates improves recall at the cost of time, so tune this setting against your test questions.
- Swapping later: Changing the rerank model does not require re-indexing, which makes it the easiest retrieval component to change.
- Compatibility: Confirm that GPUStack exposes the rerank model type in the form Dify's provider configuration expects.
- Shared capacity: Reranking often shares a GPU with embedding, so check that both fit together under peak query load.
Before you set a rerank default, compare answers with and without reranking on your own questions, and measure the added response time at the candidate count you plan to use.
Agent workflows call a model several times per task, deciding which tool to use, reading results and writing a final answer. The model must produce structured output reliably and recover when a tool returns an error. Because one user request can trigger a chain of model calls, agent workflows multiply GPU load and make latency harder to predict than a single chat turn.
- Tool calling reliability: Confirm that the chosen model supports function or tool calling in the way your Dify nodes require, and test malformed output handling.
- Call multiplication: Each step adds a model call, so a workflow with several steps can need far more capacity than a plain chat app with the same user count.
- Timeouts and queues: Set time limits for workflow runs so a slow GPU does not leave requests hanging.
- Swapping later: A different model may change which tools it calls and how it formats results, so retest every workflow after a change.
- Separate pools: Consider dedicating GPU capacity to agent workflows so batch automation does not slow interactive users.
Before you assign an agent model, run each workflow end to end on representative inputs, including failure cases, and confirm the load it places on your GPU pool.
Capacity planning when GPUs are shared across apps
A shared GPU pool behaves differently under Dify than under a single application, because several kinds of load arrive at the same accelerators at once. Interactive chat sessions want short, fast responses. Workflow runs fire model calls in bursts as nodes execute. Agent loops can hold a model busy for many sequential calls before a user sees an answer. Knowledge base indexing sends large batches of chunks to an embedding model on a schedule or when someone uploads documents. Each of these competes for the same model slots and GPU memory, so a batch job that starts during business hours can lengthen every chat response on the same model. Plan for that contention before anyone configures a production app.
A capacity plan needs concrete inputs from the business and from the Dify apps, not from the hardware. Collect the expected peak number of concurrent sessions per app, the typical prompt and output length for each workflow, the indexing schedule and volume for each knowledge base, the number of apps that share each model deployment in GPUStack, and how long users will tolerate queueing before they abandon a task. Ask the application owners to confirm these figures in writing. The plan is only as good as those assumptions, so record them with the model and version used. Confirm with the GPUStack documentation how its scheduler queues and prioritizes requests before you rely on any queueing behavior in the design.
Where the GPUStack scheduler and deployment settings allow it, give indexing and interactive traffic separate model deployments or separate GPU placements, so a large embedding job cannot starve chat. Keep the Dify worker, which runs asynchronous tasks such as indexing, and the plugin daemon, which runs model plugins, in mind as well. Both consume CPU, memory and network on the Dify host, and they can become the bottleneck before the GPUs do. Size those hosts separately and watch them during tests. Validate the whole plan with load tests on the actual GPU servers, using a realistic mix of chat, workflow, agent and indexing traffic, and repeat the tests after any model, Dify or GPUStack upgrade.
Security controls for the GPUStack connection
A Dify connection to GPUStack carries prompts, retrieved documents and API credentials between two systems that each hold sensitive data. The controls below cover where credentials live, how traffic is encrypted and routed, how outbound requests are filtered, and how teams and calls stay separated and traceable. Dify documents its side of most of these; the GPUStack side must be confirmed in your own cluster.
Scoped, rotatable GPUStack API keys
Create a dedicated GPUStack key for the Dify connection rather than reusing an administrator token, and store it only in the Dify model provider settings. Dify's environment reference says SECRET_KEY encrypts OAuth credentials, so keep the Dify secrets configuration under the same change control as the key itself. App exports exclude third-party API keys, so every environment that imports an app must have its own key entered by hand. Confirm in GPUStack which permissions a key carries, whether it can be revoked without restarting the cluster, and how rotation is recorded.
TLS and certificate trust
Serve the GPUStack endpoint over HTTPS and confirm that the Dify api and worker containers trust the certificate chain. A private certificate authority must be installed in the container trust store, or the connection fails in ways that look like model errors. Check the certificate expiry date and the renewal owner, because an expired certificate stops every workflow that calls a GPU-hosted model. Do not disable verification to get a connection working in testing and forget to restore it.
Network allowlisting to GPUStack only
Restrict outbound traffic from the Dify host to the GPUStack server addresses and ports that the connection requires. Confirm that the GPU servers do not accept inbound connections from other Dify workspaces, the sandbox, or general user networks. Dify 1.16.1 added sandbox network isolation through a Squid proxy, which helps constrain agent code, but it does not replace host-level firewall rules. Record the allowed flows in your network design so that a change to the GPU pool is reviewed as a change to the Dify path.
SSRF proxy and private address handling
Dify routes outbound requests through the SSRF proxy configured by the SSRF_PROXY settings, and that proxy blocks internal and private IP ranges. A GPUStack server on a private address may therefore be unreachable from Dify until you decide how that address is handled. Do not open the whole private range to solve one endpoint. Decide whether the GPUStack address must be an explicitly allowed internal target, document that exception, and test it from the Dify container, not from your workstation.
Workspace separation for models and keys
Model providers and their keys are configured per workspace, so a GPUStack connection added for one team should not be assumed visible to another. The community edition is limited to a single workspace, so separation between teams on one GPU pool depends on either Dify Enterprise multi-workspace features or separate Dify instances. Confirm which model is assigned to which workspace, and confirm in GPUStack whether separate user or API key scopes can keep one team's models from being called by another team's key.
Logging model calls for audit
Dify's LLMOps monitoring records application activity, but you should confirm which fields reach your audit store: the calling app, workspace, model name, timestamp and token counts. Dify's exports do not include usage logs, so logs must be collected from the running system rather than from app files. Check whether GPUStack keeps its own request log, how long it retains entries, and whether those entries contain prompt content. Align retention and access with your data classification before a regulated workload goes live.
Common GPUStack connection failures and fixes
This table maps the symptoms a team usually sees when Dify cannot use models served by GPUStack to their likely causes and a check that confirms each fix. Read it from the top when a provider test fails, then move down the rows as the symptom changes. The main takeaway is that most failures sit in the network path, the key scope, or the state of the model on its GPU server, and each can be verified before anyone changes Dify settings.
| Symptom | Likely cause | Fix |
|---|---|---|
| Provider test fails with a connection error | Dify containers cannot reach the GPUStack address, or the SSRF proxy blocks a private IP range | From the api container, run a request to the GPUStack endpoint with curl. If it fails, correct routing or DNS first. If it succeeds only outside Dify, review the SSRF_PROXY settings and the proxy allow path for that host |
| Provider test fails with an authentication error | The API key was revoked, copied with extra characters, or created without access to the model | Create a fresh key in GPUStack, confirm it can list models with a direct API call, then replace the key in Dify and retest the provider |
| Model is listed in Dify but calls fail | The model is stopped, not loaded, or running on a GPU server that went offline | In GPUStack, confirm the model instance shows as running and healthy on its worker. Send one short test request to that model name and compare it with the name Dify is using |
| Responses are slow or queued | GPU contention, or many Dify apps sharing one model instance with a large context | Check GPU memory and utilization on the serving host during a slow call. Count the apps and workflows pointed at that instance, then add a replica or move heavy jobs to a separate model |
| Embedding results changed after a model swap | Vectors from the old embedding model no longer match the new one in existing knowledge bases | Record the embedding model name in each knowledge base. Re-index every affected knowledge base after the swap, then run a fixed set of test queries and compare the top results |
| Certificate errors when calling an HTTPS GPUStack endpoint | The private certificate authority is not trusted inside the Dify containers | Confirm the CA certificate is mounted into the api and worker containers and added to their trust store. Repeat the curl test from inside each container and check that the handshake completes without bypassing verification |
Questions teams ask before connecting Dify to GPUStack
What does it cost to connect Dify to GPUStack?
The Dify Community edition is free to self-host, and the pricing page lists it as limited to a single workspace, so the software line is zero. The costs that remain are the GPU servers themselves, the time to run GPUStack and Dify, storage and networking, and staff time for setup and testing. Hardware is usually the largest item and it recurs through power, hosting and refresh cycles. A DeployOpen quote follows from a written scope that lists the number of GPU hosts, models, Dify workspaces and test cases, so ask for line items that separate one-time build work from recurring operations.
How long does a GPUStack connection take to set up?
The duration depends on how ready the cluster already is. A pool of GPU hosts with working drivers, container runtime, stable addressing and reachable networking takes far less time to connect than one that still needs driver work or network changes. The number of models to register, the number of Dify workspaces that will call them, and how much testing you require (latency, output quality and failover) also move the schedule. Ask for a timeline that is tied to the written scope and to the readiness items you must complete first.
What does our team need to provide before the work starts?
Most delays come from access and decisions, not from configuration. Plan to provide:
- Access to the GPUStack server and the GPU hosts, including the accounts and jump paths your security team approves
- Host details: GPU models, driver versions, memory, operating system and network interfaces
- Network approvals for traffic between Dify, GPUStack and the model endpoints, along with any proxy rules
- Test documents such as acceptance criteria, sample prompts and expected outputs for each model
- A named owner who decides which models each job may use and who approves changes
Can our own platform team connect Dify to GPUStack without a partner?
Yes, if your team already runs Docker, understands GPU drivers and container runtimes, and can manage network routes and firewall rules between the Dify host and the GPU pool. The Dify Marketplace GPUStack plugin is the documented starting point, and the Dify Docker Compose docs describe the self-hosted base. Gaps usually appear in capacity testing, secrets handling for model endpoints, and runbooks for when a GPU host fails. A written checklist can show which of these your team already covers, so you can decide which parts to keep in-house.
Which parts of the work change when we need Enterprise SSO or multiple workspaces?
Enterprise SSO and multi-workspace management change who can reach Dify and which workspace they see. They do not change how Dify calls GPUStack. The model connection, the provider setup and the GPU capacity plan stay the same. The access design does change: you will need to decide which workspaces may use which GPUStack models, and whether shared capacity is split by workspace. The Dify Enterprise SSO documentation separates workspace access from web app access, so confirm which scope your identity design needs before finalizing the model routing.
How do we plan capacity when several teams share the same GPUs?
Capacity planning depends on the models you run, how many requests they receive at the same time, and how long each response takes to generate. Shared pools also need a rule for what happens when two workloads compete for memory. Start by listing which models each workspace needs, then measure load under realistic concurrency on a test host before deciding how many hosts to add. Ask the vendor or your own engineers to confirm model memory requirements, since they differ by model and quantization.
What breaks most often after a GPUStack connection goes live?
The most common failures are model endpoints that became unreachable after a host restart, driver or runtime updates that changed GPU visibility, and Dify upgrades that required a new plugin version. Outbound requests from Dify may also be blocked by the SSRF proxy if the GPUStack address falls into a range it filters, so check that path during testing. Plan monitoring for GPU memory, endpoint health and queue depth, and keep a rollback path for both the GPUStack and Dify versions.
How DeployOpen scopes and delivers the GPUStack integration
DeployOpen plans, implements, documents and supports self-hosted Dify inside your own infrastructure, including its connection to GPUStack GPU clusters. We start from a written scope that names the models, GPU pool, Dify workspaces and access rules involved, then build and verify the connection before handing it over.
How we scope
- Inventory the GPU servers, GPUStack version, and the models your jobs need before any Dify configuration starts.
- Define which Dify workspaces and apps may call which GPUStack models, and who may manage provider credentials.
- Agree the written scope, deliverables and timeline in writing before work begins, since most engagements are fixed-scope.
How we build
- Configure the GPUStack plugin in Dify and register each model with its intended role, such as chat, embedding or rerank.
- Test model routing per job, confirming that each Dify app reaches only the models its scope allows.
- Measure latency and concurrency on the shared GPUs under realistic Dify workflow load and record the results.
How we hand over
- Provide documentation covering provider setup, model assignments, network paths and the capacity assumptions behind them.
- Walk your platform and security teams through credential handling, outbound network rules and the upgrade steps for both products.
- Deliver runbooks for adding a model, rotating keys and recovering a failed GPU node, tailored to your environment.
How we support
- Offer ongoing support, version upgrades and continuous testing as a retainer, scoped to your Dify and GPUStack releases.
- Review Dify minor release notes and GPUStack changes for impacts on your model connections and plan updates with you.
- Revisit capacity planning as model sizes, user counts and shared GPU demand change, and document each adjustment.
