Self-hosted embedding and rerank models for Dify

Self-hosted embedding and rerank models for Dify: what the integration does

When a knowledge base is indexed in Dify, every document chunk is sent to an embedding model, and at query time candidate passages may be sent to a rerank model. If that model is a hosted API, the content of your documents leaves your network on every indexing run and every retrieval. Serving the embedding and rerank models yourself is the way to keep knowledge base content inside your infrastructure, and the Dify Marketplace lists a Text Embedding Inference plugin for self-served embedding models, described as a fast inference solution for text embedding models (Dify Marketplace: Text Embedding Inference). In Dify, model providers are configured per provider, so the endpoint, credentials and the models it exposes are set in the model provider settings rather than in the knowledge base itself.

This page answers two questions: how to keep document content inside your own network when embeddings and reranking run on your hardware, and what it costs to change the embedding model after the knowledge base is populated. The second question matters because vectors produced by one embedding model generally cannot be compared with vectors from another, so a model change usually means re-embedding the existing content. Confirm rerank support before you plan around it: check the plugin listing and the capabilities of your chosen inference server for each model type you need, including whether the server exposes a rerank endpoint that the plugin can call. Also confirm which embedding models the plugin and server support, their maximum input lengths, and whether they run on your hardware without an external download step.

What you need before connecting embedding and rerank models to Dify

Connecting a self-hosted embedding or rerank model to Dify depends on several pieces lining up: a supported Dify and Docker Compose version, an inference host sized for the model, network paths the Dify containers can actually use, and a vector store that will hold the output. Confirm each one before you change any workspace defaults, because the vector store and embedding model choices are the hardest to undo later.

Dify and Docker Compose versions

The Docker Compose deployment requires at least 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later, and the stack includes the api, worker, plugin daemon, PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and an SSRF proxy. Pick the Dify release you will run before testing, since minor releases arrive roughly monthly. Check the upgrade guide for that version so the model setup you build survives the next upgrade.

Inference host capacity

Confirm the GPU or CPU capacity of the host serving the embedding and rerank models before you choose a model size. Embedding throughput affects how long a large knowledge base takes to index, and rerank latency affects every retrieval call a user makes. Ask your platform team for memory headroom for the model weights, batch size limits and whether the host is shared with other inference workloads.

Network path from Dify containers

The api and worker containers must reach the inference endpoint over the network, which often means a route across Docker networks, firewalls or a Kubernetes namespace boundary. Test the path from inside the containers, not from your workstation. Also confirm that the SSRF proxy settings do not block the internal address range where the endpoint sits, since the proxy is designed to block internal and private IP ranges by default.

TLS certificates for internal endpoints

Traffic between Dify and an internal inference server carries knowledge base content, so plan TLS for that hop rather than relying on the network alone. Decide which certificate authority issues the endpoint certificate, make sure the Dify containers trust that authority, and set a renewal owner. Confirm how the chosen inference server handles certificate reload, because a missed renewal will break indexing and retrieval together.

Vector store that holds the vectors

The vector store defaults to Weaviate, and the environment reference also lists Qdrant, Milvus, pgvector, OpenSearch and Elasticsearch as options. The store you choose will hold every vector your embedding model produces, so its dimensions and index settings must match the model. Confirm the store is provisioned, backed up and sized before you index content, because switching later means re-indexing.

Marketplace plugin and Enterprise prerequisites

Install the Text Embedding Inference plugin from the Dify Marketplace in the environment where you will test, and record the plugin version alongside the Dify release. On Dify Enterprise running on Kubernetes, the resources checklist requires Kubernetes 1.24 or later, Helm 3.14 or later, PostgreSQL 14 or later, Redis 6 or later, a Qdrant vector database and S3-compatible or cloud object storage. Confirm these exist before planning the vector store choice.

How to configure a self-hosted embedding model in Dify step by step

This sequence takes a team from an inference server inside its own network to a Dify workspace whose default embedding and rerank models keep knowledge base content on infrastructure it controls. Each step ends with a check so problems surface before documents are indexed.

Confirm the target Dify edition and plugin path
The platform team confirms which Dify edition runs the workspace and that the Text Embedding Inference plugin is available for it. The Dify Marketplace lists a Text Embedding Inference plugin described as a fast inference solution for text embedding models, so the team checks the plugin listing against its installed Dify version before anything else. Community edition is limited to a single workspace, which affects where defaults are set. The step is done when the plugin's compatibility with the running version is written down and the owner of the workspace is named.
Install the Text Embedding Inference plugin
A Dify administrator installs the Text Embedding Inference plugin from the Marketplace into the workspace where the knowledge bases will live. Installation runs through the plugin daemon, so the team should confirm that the plugin daemon container in the Docker Compose stack is healthy and that outbound access for plugin installation follows the organization's egress policy. The step closes when the plugin appears in the installed plugin list with its version recorded in the change log.
Deploy the embedding inference server inside your network
The infrastructure team deploys the Hugging Face Text Embeddings Inference server on hardware or a cluster the organization operates, choosing the embedding model and the hardware it needs. Ask the server's documentation which model architectures and hardware targets it supports before selecting a model. Place the server on a private network segment that the Dify api and worker containers can reach, and do not expose it to the internet. The step closes when the server reports healthy and responds to a direct request from an administrator's workstation on the same segment.
Register the embedding endpoint and model name in Dify
The Dify administrator adds the embedding model under the model provider settings, entering the internal endpoint URL of the inference server and the exact model name the server serves. Store any access token in the credential store rather than in plain notes, and confirm the endpoint is reachable from the Dify api container, since the worker processes indexing jobs on the same network path. The step closes when a test call from the Dify api container returns an embedding vector, and the team records the vector dimension the model reports.
Add a rerank model if the server supports it
If the chosen inference server offers a rerank model, the team registers it the same way as the embedding model, using its own endpoint and model name. Not every inference server provides reranking, so the team confirms this with the server's documentation or vendor before planning the retrieval design around it. Rerank calls add latency to every query, so the team measures a sample request before making it a default. The step closes when a test rerank request from the Dify api container returns scores ordered by relevance for a small set of candidate passages.
Set workspace default embedding and rerank models
A workspace owner sets the embedding and rerank models as workspace defaults in the model settings, so new knowledge bases and apps fall back to the self-hosted models rather than an external provider. Confirm that existing knowledge bases still point at the model they were indexed with, because defaults affect only new work. The step closes when a newly created test knowledge base shows the self-hosted embedding model selected without manual override.
Run a test ingestion on a sample knowledge base
The team creates a sample knowledge base with a small set of non-sensitive test documents and indexes it using the self-hosted embedding model. It checks that the vectors produced for each test document have identical dimensions, which confirms the same model produced all of them. Indexing failures usually show up in the worker logs, so the team watches those during the run. The step closes when every test document reaches an indexed status with the same vector dimension recorded.
Verify retrieval returns the expected chunks
The team runs a fixed set of questions against the sample knowledge base, each with a known source passage, and reviews which chunks come back in the top results. If rerank is enabled, it compares results with and without rerank to confirm the reordering is useful for its content. Record the query set and results as the baseline for later model changes. The step closes when the expected source passage appears in the retrieved chunks for each test question, and the baseline is stored with the configuration.

Why the embedding model choice sets your re-indexing cost

Vectors produced by one embedding model cannot be compared with vectors produced by another, so the embedding model you choose for a knowledge base is effectively permanent. If you change it later, every document in every knowledge base that uses it must be chunked and embedded again, and the old vectors must be replaced. Rerank models do not have this problem because they score query and passage pairs at query time, so the decision that carries re-indexing cost is the embedding model and its output dimension. Record both in a written decision before the first document is ingested, along with the workspace and knowledge bases they apply to.

The re-indexing bill is driven by factors you control. Document volume and the number of knowledge bases set the total amount of text to embed. Chunking settings multiply that volume, because smaller chunks or more overlap produce more vectors from the same files. Inference capacity determines how long the re-embedding run takes on your own servers, and downtime tolerance determines whether you can run it in place or need a parallel index. Vector store size also grows with dimension and chunk count, so a larger model can raise storage and query costs as well as the one-time run. Ask any partner to estimate these four inputs against your actual corpus, and to state which of them they have measured rather than assumed.

Before any upgrade that touches the Dify version or its configuration, follow the upgrade guide for the target release on the Dify releases page, take a backup, and check .env.example for new variables, as the Docker Compose documentation advises. Confirm with the vendor whether a given Dify release changes how it stores or reads vectors before you plan any model migration on top of it. Keep a tested rollback path, because a half-finished re-index against a changed embedding setting is harder to recover from than a failed upgrade.

Choosing where the embedding and rerank endpoints run

The Community edition runs the api, worker, plugin daemon and other core services on one host with Docker Compose 2.24.0 or later, alongside PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and an SSRF proxy. The usual placement is an embedding or rerank server on a separate machine in the same private network, reached by a private address. That address is where the proxy matters. The environment reference says SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges, so an endpoint on a private address may be refused before it reaches your inference server. Confirm which Dify container makes the embedding and rerank calls for your chosen plugin, and whether those calls pass through the proxy at all. The Docker Compose deployment guide is the starting point for that check.

Host sizing: The stated minimum of 2 CPU cores and 4 GB of RAM covers Dify's own services only, so the inference host needs capacity sized for the model you select.
Proxy path: Confirm whether model calls from the plugin pass through the SSRF proxy in your version, and whether an internal address can be permitted without weakening the block for other ranges.
Port exposure: Keep the inference port on the Compose network or a dedicated network segment, and do not publish it on a public interface.
Marketplace plugin: Confirm the outbound requirements of the Text Embedding Inference plugin at install time, since the plugin's own network needs can differ from the core services.
Vector store: The default Weaviate store is in place here, so any later change of embedding model means re-indexing into that store.

Before pointing production workspaces at the endpoint, run a test knowledge base through the proxy and record which addresses succeed.

Mapping Dify model settings to your inference server

This table pairs each Dify model settings field with what your inference server must expose for it to work. Read each row as a question to answer from the plugin and server documentation before you enter values in the console. Items marked as confirm are details that vary by plugin version and serving stack, so verify them in a test workspace first.

Dify field	What the server must provide	What to confirm before saving
Provider	The plugin that speaks the server's API, such as the Text Embedding Inference plugin from the Dify Marketplace	Plugin version compatibility with your Dify release and whether it is installed in the target workspace
Model type	A text embedding endpoint or a rerank endpoint, which are configured separately in Dify	Whether the served model is an embedding or rerank model, since the two types cannot be swapped in the console
Model name	The model identifier the server loads, as shown in its model listing or startup logs	Exact string match, including organization prefix and casing, because a mismatch often fails only at first indexing
Endpoint URL	A reachable HTTP address on the private network, with the port and path the server publishes	Routing from the Dify api and worker containers, not only from your workstation, and whether TLS is terminated internally
API key or auth token	Only if the server is configured to require one; some internal deployments accept unauthenticated calls on a restricted network	Whether the key is stored in Dify's encrypted credential store and scoped to one environment
Embedding dimensions	The output vector size of the model, which the vector store index must match	The dimension value for your exact model, and whether the vector store index was created with that size
Max input length	The token limit per input the server accepts before truncating or rejecting	Chunk size settings in the knowledge base must stay below this limit, or long chunks may fail
Default embedding selection per workspace	The model Dify uses for new knowledge bases when no model is picked explicitly	Changing this default does not re-index existing knowledge bases, so document the date and the affected datasets
Default rerank selection per workspace	The rerank model applied when retrieval settings enable reranking	Whether the rerank endpoint returns scores in the format Dify expects for your plugin version

Keeping knowledge base content inside your network

Serving embedding and rerank models yourself only keeps knowledge base content inside your network if the endpoint, the Dify containers that call it, and the credentials that reach it are all controlled. The cards below cover the placement, access, transport, logging and key decisions that connect those layers.

Isolate the inference endpoint

Place the embedding or rerank server on a network segment that only Dify's outbound path can reach, such as a dedicated subnet or GPU host. Dify's SSRF proxy blocks internal and private IP ranges by default, so the endpoint address must be allowed deliberately through the proxy settings rather than opened broadly. Confirm the exact allow-list mechanism for your Dify version in the environment reference before you rely on it. Record every exception so reviewers can see which internal address Dify is permitted to call.

Restrict which containers reach it

In the Docker Compose layout, the api and worker services are the components that call model providers, so they are the only ones that should hold a route to the inference host. Test this from the web, plugin daemon and sandbox containers as well, because a broader route than needed is a common gap. Use host firewall rules or network policies, and verify with a connection test from each container rather than from the host.

Authenticate calls to the endpoint

Do not leave the inference server open on the internal network. Require a token or API key on every request, and store that value in Dify's model provider configuration instead of in a shared file. Confirm whether your chosen serving stack supports per-client keys, so that a rotated key for one workspace does not break others. Document who can read and rotate the key.

Terminate TLS on the model path

Traffic from Dify to the embedding server carries document chunks and queries, so encrypt it even inside the data center. Use certificates issued by your internal CA, and confirm that Dify is configured to trust that CA. Plan certificate renewal before the endpoint goes live, since an expired certificate will stop indexing and retrieval together.

Log metadata, not request content

Inference servers often log full request bodies by default, which means raw document text ends up in plain-text logs. Set logging to record latency, status, model name and token counts, and keep request content out of routine logs. Decide separately how long any content-level debugging logs may be kept and who may read them, and confirm the server's logging options before go-live.

Protect SECRET_KEY and encryption keys

SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials, and changing it after deployment logs out all users. Store it in a secrets manager and back it up with the database, because losing it breaks access to stored credentials. Workspace encryption keys can be kept locally or in Azure Key Vault, so choose the location before the first model provider is saved and record who holds the recovery path.

Common embedding and rerank failures and how to fix them

This table pairs each failure operators commonly hit with embedding or rerank models in Dify with its likely cause and the check that confirms it. Read it from the symptom column inward, and run the checks in order before changing configuration. Most failures trace back to network reachability, proxy filtering, or a mismatch between the model that wrote the index and the model that now queries it.

Failure	Likely cause	Check to confirm it
Dify container cannot reach the endpoint	The model server listens on localhost or a host address that the api and worker containers cannot resolve, so Dify cannot connect from inside Docker	Run a connection test from inside the api container to the endpoint URL, and confirm the server binds to an address the container network can route to
SSRF proxy blocks the internal address	SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges, so a self-hosted endpoint on a private address is refused	Check the proxy logs for denied requests to that host, then confirm the endpoint's address and the proxy policy against the environment variables reference
Dimension mismatch after a model change	The new embedding model returns vectors of a different length than the index was built with, so writes or queries fail	Compare the output dimension of the new model with the dimension stored for the knowledge base index, and check the vector store error log for dimension errors
Documents ingested with the old model still in use	Chunks were embedded with one model, but queries now use another, so similarity scores are unreliable even when no error appears	Check which embedding model each knowledge base is configured with and when documents were indexed, then compare a known query against both indexes
Timeouts on large batches	Ingestion sends more text per request than the inference server can process within the client timeout, often during initial bulk upload	Reduce the batch size in the provider settings, watch GPU or CPU load during indexing, and inspect worker logs for timeout messages
Rerank selected but not applied	The rerank model is chosen in one place but the retrieval setting that invokes it is off, or the rerank provider is unavailable, so Dify falls back to the raw retrieval order	Open the retrieval settings for the app, confirm rerank is enabled with the expected model, and test a query while reviewing the retrieved results order in the trace
Rerank model unavailable after a restart	The inference container started before the model finished loading, or the model name registered in Dify no longer matches the served name	Check the inference server health endpoint and its loaded model list, then confirm the model name in Dify matches what the server reports
Workspace default points to the wrong model	A default set for the workspace is overridden or was never saved for a new app, so new knowledge bases use an external provider unexpectedly	Review the workspace default models and the embedding model recorded on a sample new knowledge base, and confirm no external provider is selected

Questions regulated teams ask before self-hosting embedding models for Dify

What does self-hosting embedding models for Dify cost, and what drives it up or down?

The cost has three parts: inference hardware (GPU or CPU capacity for the embedding and rerank servers), engineering time to deploy and wire them into Dify, and testing that shows retrieval quality holds up on your documents. Hardware is usually the largest recurring item and scales with document volume, query load and the size of the models you choose. Integration effort rises when you run several knowledge bases or need separate environments. Ask any partner for a quote that lists each part, states what is one-time and what recurs, and names the hardware assumed. DeployOpen does not publish prices. Quotes follow a written scope agreed before work starts.

How long does a self-hosted embedding and rerank setup take?

Duration depends on scope: how many models you serve, whether you need GPU capacity provisioned, how many knowledge bases are re-indexed, and how much validation your risk owners require. A single embedding model on existing hardware with a small document set is a much narrower job than a multi-environment rollout with formal test sets. Ask for a timeline tied to the written scope, with milestones for model serving, Dify registration, workspace defaults, re-indexing and acceptance testing. Any timeline should identify which steps wait on hardware procurement or data owner sign-off.

What must our team provide before the work starts?

Expect to provide:

Network access from the Dify api and worker services to the inference servers, and an agreed route that keeps document traffic inside your boundary
GPU or CPU capacity, or approval to procure it, sized to the models you select
Named data owners who can approve a representative test set of documents and queries
Access to your current Dify environment, including workspace owners who can change default models

Confirm these items in the scope document. Missing data owners are a common cause of delay because retrieval quality cannot be judged without them.

Can our own team deploy this without an outside partner?

Yes, if your team already operates containers, GPU nodes and Dify. The work involves serving an embedding model through an inference server, registering it in Dify through the Text Embedding Inference plugin listed on the Dify Marketplace, setting workspace defaults and validating retrieval. The parts teams most often underestimate are the evaluation design and the re-indexing plan, not the install. An outside partner adds value when you need a fixed-scope delivery with written acceptance criteria or extra hands for testing. Decide based on the skills you have in-house and the deadline you face.

Does Dify Community or Enterprise change how we approach this?

The model-serving approach is the same on both editions, because embedding and rerank endpoints are configured through model providers. The differences lie around it. The Community edition is free and limited to a single workspace according to the Dify pricing page. The Enterprise edition is custom priced and adds SSO, multiple workspaces, advanced security, a commercial license and professional support. If you plan several workspaces that should share models, or you need centralized credential management, factor Enterprise into the design. Confirm current edition features and licensing terms with Dify directly before you commit.

How do we avoid re-indexing our knowledge bases later?

Re-indexing is required when you change the embedding model, because vectors from different models are not comparable. Decide the embedding model before loading production documents, and record it as the workspace default and in your configuration standard. Keep the test set you used to choose it, so a future change can be measured against the same queries. Plan for a re-index window, its compute cost and a rollback path if the new model performs worse. Rerank models do not require re-indexing of stored vectors in the same way, so treat them as a separate, lower-risk change.

Does keeping embeddings in-house mean our documents never leave the network?

Not automatically. Dify also calls chat models, and those may still go to external APIs unless you route them elsewhere. Outbound requests from Dify can be filtered through the SSRF proxy, which the Dify environment reference says blocks internal and private IP ranges, so review how that proxy is configured for your inference endpoints. Map every component that touches document text, including chat, embedding, rerank, logging and vector storage, and confirm each one against your data classification. Your security team should verify this with network captures in a test environment.

How DeployOpen scopes the embedding and rerank work

DeployOpen scopes the embedding and rerank work in writing before any build starts, and the scope names the model servers, the Dify workspaces that will use them, the knowledge bases affected and the evidence the client will keep. The written scope covers planning, implementation inside your own cloud accounts, data centers or air-gapped network, testing of retrieval quality and outbound traffic, and documentation of the configuration. It also states what is out of scope, such as model selection decisions that belong to your data science or risk teams, and what DeployOpen will not certify. DeployOpen prepares systems, controls and evidence for compliance assessments, but independent auditors and certification bodies issue any attestation.

The scope should record the re-indexing plan as a deliverable, not an afterthought. Changing an embedding model means every stored vector in affected knowledge bases must be regenerated with the new model, so the scope needs a named owner for the cutover, a list of knowledge bases and their document counts, a rollback position, and a decision on whether old and new indexes run side by side during the switch. DeployOpen can prepare the runbook for that procedure and test it in a non-production Dify instance first. Ask for the runbook, the verification steps and the operator hand-off as named items in the statement of work so they can be accepted or rejected before work begins.

Before you sign, confirm three points with DeployOpen and your own teams: which Dify edition and deployment layout the work targets, who on your side approves model changes and which environment holds the test corpus. Fixed-scope agreements set deliverables and timelines in advance, and any ongoing model monitoring, upgrades or re-testing after Dify releases fall under a separate retainer that should be scoped alongside the build. Ask for a written statement of assumptions covering the Dify version, the hosting platform for the inference servers and the GPU or CPU capacity you have committed, because those assumptions drive both the effort estimate and the schedule.

How DeployOpen works on this

DeployOpen plans and builds self-hosted embedding and rerank models for Dify inside your own infrastructure, so knowledge base content is embedded and reranked on servers you control. We scope the model choice, the workspace defaults and the re-indexing plan in writing before any build starts, then hand over documented, tested systems your team can operate.

How we scope

Engagements are fixed-scope, with the embedding model, rerank model, hosting target and workspaces agreed in writing before work starts.
We inventory which knowledge bases hold sensitive content and confirm which data must never leave your network.
The written scope names the Dify version, the inference server and the default model settings each workspace will receive.

How we build

We deploy the inference server inside your cloud account, data center or air-gapped network and connect it to Dify through the supported plugin.
We configure workspace default embedding and rerank models, then test retrieval quality against sample documents your team approves.
We plan the re-indexing run, its duration and its downtime window before any embedding model is changed in production.

How we hand over

You receive documentation covering endpoints, model versions, credential locations, network rules and the workspace default settings.
We provide a runbook for changing embedding models, including how existing knowledge bases are re-indexed and verified.
Your team receives a walkthrough of monitoring, restart procedures and the checks to run after each Dify minor release.

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover.
Retainer work can cover Dify minor release reviews, inference server updates and retrieval regression checks.
Support scope and response expectations are written into the retainer agreement, so you can confirm them before signing.

Self-hosted embedding and rerank models for Dify