How Dify connects to Xinference for on-premises models
Dify reaches Xinference through the langgenius model provider plugin for Xorbits Inference, which is listed on the Dify Marketplace, and the connection stays inside the network your team controls. Xinference is the serving layer: it loads open-weight models onto your hardware and exposes them over an API, while Dify consumes those endpoints as a model provider. Dify itself does not run the models, so the GPU capacity, model storage and serving behavior all sit with the Xinference side of the design.
Before you build, confirm the plugin's configuration fields and the Xinference API behavior against the plugin listing and the Xinference documentation for the exact versions you run. Check which fields the plugin asks for when you register a model, how it handles the base URL and model identifiers, and whether the chat, embedding and rerank model types are each supported as you expect. Record the versions of Dify, the plugin and Xinference together, because a plugin or serving update can change how a model is registered and called. Ask the plugin's maintainers or your own test environment to confirm any behavior you intend to rely on rather than assuming it from the listing.
Plan the network path as a deliberate design decision. Dify's outbound requests pass through the SSRF proxy controls described in its environment reference, and that proxy blocks internal and private IP ranges by design. A Xinference endpoint on a private address may therefore need an explicit allowance or a reachable path that your security team has reviewed, so test the route from the Dify api and worker containers, not only from your workstation. Keeping this traffic on private links and documenting the allowed destinations gives auditors a clear account of where prompts and retrieved content travel.
What you need before connecting Xinference to Dify
Four to six prerequisites have to be in place before Dify can call a Xinference model. Each one sits on a different side of the connection: the Dify platform, the inference server, the network path between them, and the model and plugin choices. Check them in this order so that a failed connection can be traced to one layer.
A self-hosted Dify deployment
Dify's Docker Compose deployment needs at least 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later, and it starts the api, worker and plugin daemon that make outbound model calls. Confirm that the host meets these minimums with headroom for your expected load. If you need high availability, Dify Enterprise on Kubernetes is the other supported layout, and its resource checklist sets different requirements. Record which layout you use, because the network rules in the next card depend on it.
A running Xinference server
Xinference must be installed and serving the models you plan to use before you configure anything in Dify. Confirm the Xinference version you intend to run against the requirements published by the Xinference project, along with the GPU driver and CUDA levels it expects on your hardware. Load each model file on the server first and confirm it responds locally, so that a later failure points to the connection rather than the model.
Network path from Dify to Xinference
The Dify api and worker containers, not your browser, open the connection to the Xinference endpoint. Confirm that those containers can reach the Xinference host and port, and that any firewall, proxy or Docker network rule permits it. Check how the SSRF proxy treats private IP ranges, since an on-premises endpoint is usually a private address, and plan the allow rule deliberately instead of loosening the proxy broadly.
A model inventory by job
List every chat, embedding and rerank model you intend to register, with its Xinference model UID and the knowledge base or app that will use it. Chat, embedding and rerank models behave differently in Dify, so a single model rarely covers all three jobs. Record the context length, the accelerator each model needs and who approved its weights for your data classification. This list also drives GPU placement and the embedding choice you cannot easily change later.
A compatible Xinference plugin
Dify connects to Xinference through the langgenius Xinference model provider plugin listed in the Dify Marketplace. Install the version that matches your Dify release, and treat 1.17.1 as the latest Dify release in the facts available here, so confirm the plugin's compatibility range on its Marketplace page before you install it. Pin the plugin version in your change records, because plugin updates can alter how model parameters are passed.
A named owner for model credentials
Xinference endpoints may run without authentication on an internal network, so decide who owns the endpoint address and any access control in front of it before you store anything in Dify. Name the person who approves changes to the model provider configuration in each workspace. Document where the endpoint URL is recorded so that a moved or re-IPed Xinference server can be updated without rebuilding apps.
Configuring the Xinference provider in Dify, step by step
This sequence takes a self-hosted Dify workspace from an empty model list to Xinference-served chat, embedding and rerank models that appear in app model selectors. Each step ends with a check you can run before moving on.
Confirm the workspace can accept the Xinference plugin
Before any model is added, a workspace administrator confirms that plugin installation is allowed for the workspace and that the Dify instance can reach the Dify Marketplace or your approved plugin source. Plugin installation rules are governed by your administrator settings, so confirm them in your own environment rather than assuming the default. Done means the plugin installation option is available to the administrator and the Xinference entry can be opened from the Marketplace listing, which you can review at the Dify Marketplace page for the langgenius Xinference plugin.
Install the Xinference plugin from the Marketplace
A workspace administrator installs the langgenius model provider plugin for Xorbits Inference (Xinference) from the Dify Marketplace listing into the workspace. Record the plugin version the installer shows, since plugin versions and the supported Dify version range should be checked against the release notes before production use. Done means the plugin appears in the installed plugin list with a status that shows it is active, and no installation error remains in the plugin panel.
Start Xinference and load the chosen models
The infrastructure or ML team starts the Xinference server on a host with the GPU or CPU resources planned for the workload, then launches each chosen model with its model type, format and size settings. Keep the server on a network segment that Dify's api and worker containers can reach, and confirm which exact model files and quantizations your team has approved. Done means every model shows as running in the Xinference dashboard or its model list, and a direct request from a machine on the Dify network returns a response.
Record each model UID exactly as Xinference reports it
For each running model, copy the model UID from Xinference into a shared configuration record, including case, hyphens and any version suffix, because Dify matches on this string when it calls the model. Note the model type reported for each UID (chat, embedding or rerank) so the registration in the next steps matches the server. Done means you have one line per model with its UID, type and host, and a second person has checked the entries against the Xinference model list.
Add the Xinference provider with endpoint and authentication
In the Dify model provider settings, the workspace administrator opens the Xinference provider and enters the server base URL that Dify's containers can resolve, plus any API key or authentication header your Xinference deployment requires. Keep the credential in the provider form rather than in environment files, and confirm which network path the traffic takes if the server sits in another segment. Done means the provider card shows as configured, and saving the settings returns without a connection error.
Register each model under its correct model type
The administrator adds each recorded model to the Xinference provider, entering the UID exactly as recorded and selecting the matching model type: chat or LLM for generation, text embedding for embedding models, and rerank for reranking models. A mismatched type is a common cause of models that save but never appear in the right selector. Done means each model is listed under the provider with its type, and the list matches the record from the previous step with no missing or duplicate UIDs.
Run a test call for each model type
For every registered model, the administrator runs the provider's built-in test or sends a short prompt that exercises the model type: a short chat request, one embedding of a sample string, and one rerank call over two or three sample passages. Use only non-sensitive sample text at this stage. Done means each test returns a valid response, the embedding returns a vector of consistent length, and the rerank returns an ordering, with any error recorded against the UID for troubleshooting.
Confirm models appear in app model selectors
A builder opens a test chat app and a knowledge base in the same workspace and checks that the chat model appears in the app's model selector, the embedding model appears in the knowledge base indexing settings, and the rerank model appears in the retrieval settings. Run one short conversation to confirm the full path works end to end. Done means each model is selectable in the right place, and a test app answers a question through the Xinference-served model.
Mapping Xinference model UIDs to Dify model types
This table shows how a planned Xinference inventory maps each model UID to a Dify model type, hardware placement and approving owner. Read each row as a template: the UID, type and approval columns must agree before a model is registered. The main takeaway is that a model is only usable in Dify when its UID, function and type all match what the server reports.
| Xinference model UID | Model family and function | Dify model type | Hardware | Approving owner |
|---|---|---|---|---|
| qwen2.5-instruct-illustrative | Open-weight chat model for drafting and summarization | Chat (LLM) | Two GPUs on inference node A | Head of AI platform |
| bge-m3-illustrative | Multilingual text embedding model for knowledge base indexing | Text embedding | One GPU on inference node A | Data governance lead |
| bge-reranker-illustrative | Cross-encoder reranker for retrieval results | Rerank | One GPU on inference node B | Data governance lead |
| llama-3-8b-illustrative | Smaller chat model for internal classification tasks | Chat (LLM) | One GPU on inference node B | Security architect |
| e5-large-illustrative | English text embedding model for legacy document sets | Text embedding | CPU pool on node C | Records management owner |
| gte-reranker-test-illustrative | Candidate reranker held in staging for evaluation | Rerank (staging only) | One GPU on staging node | Model risk committee |
Placing models on GPUs and sizing a shared Xinference cluster
GPU placement decides latency, concurrency and how far a failure reaches, so it should be designed before any model is registered in Dify. Each model loaded into Xinference occupies GPU memory on one or more devices, and a chat model, an embedding model and a rerank model competing for the same card will slow each other down under load. Decide which models share hardware, which get dedicated devices, and how many replicas of each high-traffic model exist. A single GPU server holding every model is cheap to start with, but one host failure then takes out chat, retrieval and rerank together for every Dify workspace that depends on the cluster.
Cost sits in several places. The GPU servers are the largest one-time and recurring item. Storage for model files is smaller but grows with every model version kept for rollback. Operator time to download, load, test and update models recurs with each model release. On the Dify side, retrieval and workflow calls add load to the Xinference endpoint even when no one is chatting, because knowledge base indexing and agent steps call the model as well. Ask the partner or your own team to estimate these four items separately so that a later change to one does not hide in the others. Capacity rises with model size, context length, the number of concurrent apps and the volume of embedding batches, and falls when models are quantized, batched well or scheduled away from peak hours.
Sizing must be measured in your environment with your own workloads. No capacity figures are supplied for Xinference or the models named in your plan, so confirm throughput, memory use and time to first token by running representative prompts, document sizes and batch volumes against the candidate GPU layout. A quote should include a written list of models with their UIDs and versions, the GPU layout per server and per model, the failover expectation (for example, whether a replica must take over automatically or an operator restores service), and acceptance tests that pass or fail on measured latency and error rates.
Securing the Xinference endpoint and model credentials
Xinference exposes model inference over HTTP, and Dify holds the credentials that call it. The controls below work together: they limit who can reach the endpoint, keep outbound calls filtered, protect the stored credentials and record what the model was asked to do.
Restrict the endpoint to the Dify network segment
Place the Xinference server on the same private segment as the Dify api and worker containers, or allow inbound traffic to its port only from those hosts. Confirm the exact source addresses in your own environment, including any NAT or load balancer in between, because the rule must match what Dify actually sends. Test from a host outside the segment to confirm the connection is refused.
Keep the endpoint off public routes
Do not publish the Xinference port through a public load balancer, reverse proxy or cloud address. Model endpoints are often deployed quickly for testing and left reachable, so review security groups, firewall rules and ingress objects after every change. Confirm with your platform team that no DNS record or gateway route exposes the service outside your internal network.
Store any Xinference access credential as a secret
If your Xinference deployment requires an access token, treat it like any other provider key: keep it in your secrets manager, avoid pasting it into shared notes, and rotate it on a schedule your security policy sets. Dify encrypts the provider credentials it stores, but the source value still lives somewhere else, so confirm where it is kept and who can read it.
Filter outbound calls with SSRF_PROXY
The SSRF_PROXY settings route Dify's outbound requests through a proxy that blocks internal and private IP ranges. Xinference usually sits inside those ranges, so you must allow the single approved Xinference address through the proxy's rules. Keep the allowance as narrow as possible and test that other internal hosts stay blocked after any change.
Plan SECRET_KEY before storing credentials
SECRET_KEY encrypts stored provider credentials and signs session cookies and JWTs. Changing it after deployment logs out all users, and the credentials it protects depend on it, so set the value during installation, keep it in your secrets manager and record it in your recovery documentation. Plan any rotation as a scheduled change with users notified in advance.
Limit who can manage model providers
Review which workspace members can add, edit or delete model providers, since those changes decide which endpoint and credential every app uses. Keep the list short, assign the role deliberately, and review it when people change teams. Record each provider change in your change process so a new Xinference address or key can be traced to an approved request.
Log model calls at both ends
Enable logging on the Dify side to see which app and workspace made each request, and on the Xinference side to see which model UID served it and from which source address. Matching the two records lets you investigate unexpected calls and confirm that only the Dify host reaches the endpoint. Confirm retention and access to these logs with your compliance owner before go-live.
Why Xinference models fail to appear or respond in Dify
This table pairs common Xinference symptoms in Dify with their likely causes, the checks that confirm each cause, and the fix to apply. Work down the rows in order, since a missing plugin or wrong model type hides every other problem. Before changing anything on a Dify host, take a backup and read the upgrade guide for the target version, as the Docker Compose docs advise.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| Plugin installed but no models listed | The Xinference model provider plugin is installed but no model was added with a valid server URL and model UID, or the Xinference server is not running the model. | Confirm the model is loaded on the Xinference server and that the server URL in the Dify provider settings responds from the Dify API container. | Launch the model on Xinference, then add it in Dify with the exact server URL and model UID shown by the server. |
| Model UID does not match the server | The UID typed in Dify differs from the UID Xinference assigned when the model was launched, including case or a custom name. | Compare the UID in Dify with the running model list on the Xinference server. | Re-enter the UID exactly as the server reports it, or relaunch with the intended UID and update Dify. |
| Model registered under the wrong type | A chat model was added as an embedding or rerank model, so Dify calls the wrong API and the model appears only in some selectors. | Check the model type recorded for the entry in Dify against the type the model was launched as in Xinference. | Delete the entry and add it again under the matching type, then retest it in a workflow that uses that type. |
| Timeouts on large context or cold model loads | The first request waits while weights load into GPU memory, or long prompts exceed the time Dify allows for a response. | Send a short test request to the model directly on Xinference, then time the same request through Dify and note whether only the first call fails. | Keep frequently used models loaded, reduce context length in workflows, and confirm the timeout settings in the Dify environment reference before raising them. |
| Requests blocked by the SSRF proxy | The Xinference address is in a private or internal IP range, and the SSRF_PROXY settings block outbound requests to internal ranges. | Check the Dify API and worker logs for proxy denial messages, and confirm the Xinference host address falls in a private range. | Review the SSRF_PROXY settings in the Dify environment reference and allow only the specific Xinference address your security team approves, rather than widening the whole range. |
| Plugin incompatibility after a Dify upgrade | The Xinference plugin version does not match the Dify version, or the upgrade changed variables or plugin behavior. | Compare the installed plugin version with the Dify release notes and the upgrade guide for the target version, and review .env.example for new variables. | Follow the upgrade guide for the target version, update the plugin to a compatible version, and roll back from the backup if models stop responding. |
| Out-of-memory errors on the GPU host | The model, its context size or a concurrent request exceeds available GPU or system memory on the Xinference host. | Review Xinference and host logs for memory errors at the time of failure, and check GPU memory use while the model serves requests. | Use a smaller or more heavily quantized model, lower the context length, limit concurrent requests, or place the model on a host with more memory. |
Questions teams ask before connecting Xinference to Dify
How much does connecting Xinference to Dify cost, and what is one-time versus recurring?
The cost is mostly labor and hardware, and no price is set by the software itself: Dify Community is free to self-host and Xinference is open source, so your spend goes to GPU hosts, the people who install and test the integration, and the staff who run it afterward. One-time work covers installing the Xinference model provider plugin in Dify, loading and registering each model, and acceptance testing against your own prompts and documents. Recurring costs include GPU hosting and power, time for model updates, and upgrades to Dify and Xinference. Ask any partner for a written scope and a quote that lists these items separately, and check how it handles each model you add later.
How long does the Xinference integration take, and what makes it longer?
Duration depends on your environment more than on the connection itself, so ask for a timeline in the written scope rather than relying on a general estimate. The biggest drivers are the number of models you serve, how GPUs are laid out across hosts, whether network changes need approval from security or network teams, and how many Dify workspaces must be connected. Each added model needs its own UID, placement check and test pass. Approvals for new network paths often take longer than the technical steps, so start those requests early.
What must our team provide before a partner can start?
Your team needs to provide the GPU hosts or capacity that will run Xinference, a list of approved open-weight models with their licenses reviewed, the network rules that allow Dify to reach the Xinference endpoints, and a named owner who can make decisions during the work. You should also provide access to a test Dify instance, representative sample data for acceptance testing, and the change window for production. Confirm with your security team how model weights are downloaded and stored, since that step often needs its own approval.
Can an internal engineer connect Xinference to Dify without outside help?
Yes, a capable platform engineer can do the connection alone if they have time, GPU experience and access to both systems. The work involves Docker or Kubernetes deployment knowledge, GPU driver and placement checks, model registration in Dify, and testing retrieval and generation quality. The hardest part for most internal teams is not the plugin itself but keeping the setup current after model and platform releases, which is an ongoing operations task. Decide who will own it after go-live before you start, whether that is your team alone or a partner under a retainer.
Can one Xinference cluster serve several Dify workspaces?
Technically you can point several workspaces at one Xinference cluster, but licensing decides whether you may run Dify that way. The Dify Community edition is limited to a single workspace, and the Dify Open Source License forbids using the source code to operate a multi-tenant environment without written authorization. If you need several workspaces, the Enterprise edition, which is custom priced, adds multiple workspace management. Confirm the license position with Dify before you design a shared cluster, and separate model credentials and access per workspace either way.
Are models served by Xinference as capable as hosted models?
Not necessarily, and the answer depends on the specific model, its size and quantization, and your task. Open-weight models you host can meet many internal use cases, but they may trail the largest hosted models on complex reasoning or long context, and they need GPU capacity that scales with your load. Test each candidate on your own prompts, documents and acceptance criteria before you choose it for production. Embedding and rerank models affect retrieval quality as much as the chat model, so evaluate them as well.
Does keeping models on Xinference keep our data inside our network?
Running Xinference inside your infrastructure keeps prompts and retrieved content off third-party model APIs, but that is only part of the picture. Dify itself still stores app data, knowledge bases and logs in the metadata database, file storage and vector store you configure, and outbound requests pass through the SSRF proxy settings you define. Check that Xinference endpoints are reachable only from Dify and that model download sources are approved. Your data protection review should cover each of these components, not just the model server.
How DeployOpen scopes and delivers a Dify and Xinference integration
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure. For Dify with Xinference, we start by agreeing which models run on-premises, where they sit, and which Dify workspaces will use them, then build and hand over the connection in writing before work starts.
How we scope
- Inventory the chat, embedding and rerank models your teams need and confirm which must stay on-premises.
- Map the Xinference hosts, GPU capacity and network paths that Dify's API and worker containers will reach.
- Agree in writing which Dify workspaces share the cluster and how each one is isolated and credentialed.
How we build
- Install and configure the Xinference cluster and register each model with its UID and endpoint settings.
- Connect the Xinference model provider plugin in Dify and test each model type in a non-production workspace.
- Verify GPU placement, outbound proxy rules and internal-only routing against the scoped design before cutover.
How we hand over
- Deliver a runbook covering model UIDs, placement decisions, credential locations and the steps to add a model.
- Document the Dify configuration per workspace, including which defaults point to on-premises models.
- Walk your operations and security teams through the tests performed and the open items still to confirm.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after the initial handover.
- Review Xinference and Dify version changes against your deployed configuration before you upgrade either one.
- Keep the documentation current as models, GPU hosts and workspaces are added or retired over time.
