When Dify hybrid search without a rerank model is enough, and when it is not
Hybrid retrieval without a rerank model is often enough when the questions your users ask share vocabulary with the documents in the knowledge base. Dify's hybrid mode combines two signals: semantic similarity from embeddings, which matches meaning even when wording differs, and keyword matching, which catches exact terms such as product codes, policy numbers and error strings. The weight settings control how much each signal contributes to the combined ranking. A team whose knowledge base returns uneven answers should first find out which signal is failing, because shifting weights only helps when the right chunk is already among the candidates and is ranked too low.
The decision point is whether the ordering problem can be solved by weights alone. If the right passage appears in the results but below weaker ones, and the weight adjustments you test move the problem around without fixing it, a reranker is the next step. A rerank model reorders a candidate set by scoring each passage against the question directly, which costs an extra model call per query and a separately configured model. Confirm in your installed version whether TopK and score threshold apply when no rerank model is selected, because the topic brief indicates they act only in the rerank phase. If they do not appear to change results in weights-only mode, that is expected behavior to verify, not a fault to fix.
Before changing any setting, record the version you run, since retrieval options have changed across releases. The Dify FAQs and the knowledge base settings in your own console are the sources to check against your version. Then run a fixed set of real user questions through retrieval testing with weights only, note where the correct chunk ranks, and only then add a reranker and repeat the same test. Plan for the reranker's model choice, its provider or hosting, and the added latency and cost on every query as ongoing items, not one-time setup.
How the retrieval modes differ in Dify
Semantic retrieval converts the user's question into an embedding and returns the chunks whose embeddings sit closest to it in the vector store. The score measures similarity of meaning, so a paraphrased question can find a chunk that shares no exact words with it. This mode depends on the embedding model chosen at indexing time, because the question must be embedded with the same model family to be comparable.
- Scored signal: vector similarity between the query embedding and each chunk embedding.
- Visible settings: the retrieval method selector set to semantic search, the embedding model fixed by the knowledge base, and the TopK and score threshold fields as shown in your version.
- Where TopK applies: confirm in the Dify docs whether TopK and score threshold limit results when no rerank model is configured.
- Typical weak spot: exact identifiers, part numbers and error strings, which may rank below loosely related text.
- Index dependency: semantic search needs an embedding-based index, so confirm which index types your version offers for this mode.
Check this mode with test questions that use your product codes and policy numbers, since those expose the weak spot quickly.
Keyword retrieval scores chunks by how well their terms match the terms in the question. It does not need an embedding model to rank results, which keeps the retrieval step simpler to reason about and avoids a vector call per query. The trade-off is that the ranking follows words, so a question phrased with different vocabulary may return weak or empty results.
- Scored signal: a term-matching relevance score computed over the chunk text. Confirm the exact scoring function in the Dify docs for your version.
- Visible settings: the keyword retrieval option in the knowledge base retrieval settings, plus TopK and score threshold where your version exposes them.
- Where TopK applies: confirm whether the keyword results are capped by TopK before any reranking step.
- Strong case: identifiers, clause numbers, acronyms and product names that appear verbatim in source documents.
- Weak case: synonyms, translated content and conversational questions with few shared terms.
Before relying on this mode, compare its results with a semantic run on the same questions to confirm that keyword matches cover the vocabulary your users actually type.
Hybrid retrieval with weights runs both the semantic and the keyword search, then combines their scores using a weight that favors one signal over the other. The combined list returns without a second model pass, so the cost stays close to a single retrieval. Because the two scores come from different scales, the weight is a practical blend tuned against test questions, not a measure of how much each signal is correct.
- Scored signal: a weighted combination of the semantic score and the keyword score for each chunk.
- Visible settings: the hybrid search option, a weight control between semantic and keyword, and no rerank model selector until one is added.
- Where TopK applies: the Dify docs describe TopK and score threshold as part of the rerank phase, so confirm what limits the merged list in weighted mode for your version.
- Tuning approach: adjust the weight in small steps and rerun the same question set after each change.
- Scope of change: the weight is set per knowledge base, so one setting affects every app that uses that knowledge base.
Record the weight, the embedding model and the question set together, so a later change can be traced to its effect on results.
Hybrid retrieval with a rerank model adds a second stage. The system first gathers candidates from both semantic and keyword search, then a rerank model scores each question and chunk pair directly and reorders the candidates. This is the stage where TopK and score threshold take effect in Dify, so these two controls decide how many reranked chunks reach the model and which low-scoring chunks are dropped.
- Scored signal: a relevance score from the rerank model for each candidate chunk.
- Visible settings: the rerank model selector, which requires a configured rerank model in the model provider settings, along with TopK and score threshold.
- Where TopK and threshold apply: TopK caps the reranked list, and the score threshold removes chunks below the cut-off. Score scales differ between rerank models, so set the threshold from test results.
- Cost and latency: each query makes an extra model call, and the candidate chunks are sent to the rerank provider or inference server.
- Data flow: confirm where the rerank model runs, because retrieved text leaves the knowledge base for that call.
Compare results with and without the reranker on the same questions, and confirm the rerank model's data handling before enabling it.
Settings that change hybrid search results
This table compares the retrieval settings a builder touches in a Dify knowledge base and shows what each one changes when no rerank model is selected. Read the last column first, because it shows which settings are inactive in a weights-only setup. The main takeaway is that TopK and score threshold do nothing until a rerank model runs, so tune the weights first.
| Setting | When it fits | What it changes | What it does not affect without a rerank model |
|---|---|---|---|
| Semantic weight | Users paraphrase, ask in natural language or use terms that differ from the documents | Raises or lowers the share of embedding similarity in the blended score | Does not add a second scoring pass or reorder results by a reranker |
| Keyword weight | Queries depend on exact identifiers such as product codes, clause numbers or error strings | Raises or lowers the share of keyword matching in the blended score | Does not change the embedding vectors or the semantic side of the score |
| Rerank model selection | Top candidates contain relevant chunks but the best ones sit too low | Adds a second pass that rescores the candidate set with a dedicated model | Nothing runs in this phase, so the setting has no effect on the weighted order |
| TopK | You need to limit how many chunks reach the app after retrieval | In the rerank phase, sets how many reranked chunks are kept | Confirm in your version whether it applies to the weighted list; the topic premise is that it only applies in the rerank phase |
| Score threshold | You want low-relevance chunks dropped before they reach the model | In the rerank phase, discards chunks scored below the cut-off | Has no effect on the blended ranking when no rerank model is selected |
| Index method | You plan to combine semantic and keyword retrieval from the start | Determines whether embeddings exist for the semantic side of hybrid search | Confirm that hybrid retrieval requires a High Quality index; a keyword-only index has no semantic weight to tune |
| Embedding model | Your content or language is poorly covered by the default embedding model | Changes the vectors behind the semantic score for every chunk | Does not change keyword matching; confirm in the docs whether a change requires re-indexing |
Why TopK and score threshold apply only in the rerank phase
TopK and score threshold filter the output of the rerank step, so without a rerank model they do not govern the weighted merge of semantic and keyword results. In a hybrid setup that uses only weights, Dify combines the vector score and the keyword score according to the weight split and returns results from that merged list. The two controls are not applied to that merge, so changing them will not reorder or trim the weighted results. Teams that expect a higher threshold to remove weak matches, or a lower TopK to shorten the list, will see no change and may conclude the retrieval is broken when the setting is simply inactive.
The consequence for testers is that retrieval tuning has to be planned around which mode is active. If the knowledge base is set to weighted scoring, adjust the semantic and keyword weights and judge the effect on the returned chunks. If the team wants TopK or a score threshold to matter, they need to select a rerank model first, because the controls only act on that second phase. Confirm this in your own version by opening the retrieval test panel for the knowledge base, running the same question with rerank turned off, changing TopK and the threshold, and checking whether the returned chunks and their order change. Repeat the run with a rerank model selected and compare.
Record the results of both runs with the exact settings used, since the panel behavior is the evidence a reviewer will ask for before an app goes live. Confirm the control names, default values and panel layout against the Dify documentation for your release, because the retrieval settings have changed between versions and this page does not assume a specific one. Treat a control that appears to do nothing as a configuration question first, and only move to chunking or model questions after the mode is confirmed.
How to configure and test hybrid search in Dify
This sequence sets the index method, retrieval mode, weights, reranking, TopK and score threshold in a fixed order, then checks each setting with representative questions. The result is a knowledge base configuration the team can justify, repeat and revisit after content or model changes.
Confirm the index method first
The knowledge base owner checks that the index method supports the retrieval you plan. Hybrid search combines a semantic signal with a keyword signal, so the semantic side depends on embeddings being built for the documents. If the knowledge base uses a keyword-only index, decide whether to rebuild it before tuning anything else. Confirm in your Dify version whether the index method can be changed after documents are indexed. Done when the index method is recorded in your configuration notes and matches the retrieval you intend.
Set retrieval mode to hybrid
The knowledge base owner opens the retrieval settings and selects the hybrid retrieval mode. Keep semantic-only and keyword-only runs saved as comparison baselines, because the final test depends on comparing the same questions across modes. Save the setting, then reopen the knowledge base to confirm that hybrid is still selected. Done when the mode reads hybrid after a reload and the app that uses this knowledge base still points to it.
Set semantic and keyword weights
Set the semantic and keyword weights to the balance you want, starting near the middle and adjusting in small steps. Semantic weight helps with paraphrased questions, while keyword weight helps with exact product codes, policy numbers and names that appear verbatim in your documents. Change one value at a time and note what each change does to results. Done when you have two or three recorded weight settings, each with the questions it handled best.
Decide whether to add a reranker
The platform owner decides whether a rerank model justifies its extra model call. Without a reranker, the weights alone shape the ordering of hybrid results. Add a reranker when relevant chunks appear but rank poorly, or when near-duplicate chunks crowd out the best answer. Confirm in your model provider settings that a rerank model is available and supported in your version. Done when the team has a written yes or no decision tied to a specific test failure.
Set rerank model, TopK and threshold
If you chose a reranker, select it in the retrieval settings, then set TopK and the score threshold. Both settings apply in the rerank phase, so they do not control ordering on the weighted-only path. Start with a modest TopK and a threshold low enough that known good chunks survive the filter. If you skipped the reranker, record that TopK and threshold are not the controls for ordering in that configuration. Done when the saved values appear in the knowledge base settings and the team knows which phase each setting affects.
Run retrieval tests with representative questions
The knowledge base owner and a domain expert run a set of representative user questions through retrieval testing, saving the returned chunks and scores for each configuration. Include questions with exact identifiers, paraphrased questions and questions the knowledge base cannot answer. Compare results across weight settings, with and without the reranker. Check that the expected source chunk appears, that off-topic chunks are filtered and that unanswerable questions return little or nothing. Done when the team signs off one configuration with recorded results and a list of failures that point to chunking or to the retrieval settings.
Security and data handling for retrieval settings
Hybrid search settings look like tuning, but they also decide which services receive document text and which credentials and endpoints retrieval can reach. The cards below cover the risks to review before choosing weights or a rerank model, and the control to set for each.
Hosted rerank calls leave the stack
If the rerank model is a hosted provider, every candidate chunk returned by retrieval is sent to that provider with the user's query. Chunks can contain personal, contractual or regulated data, so the data path must be approved before the rerank step is enabled. Confirm the provider's data processing terms and whether a self-hosted rerank endpoint is an option for the data set in scope.
Embedding calls carry document content
Embedding models receive document text at indexing time and query text at retrieval time, so the same external-processing question applies to the embedding provider even when the rerank model stays local. Review the embedding and rerank providers together, since a hybrid setup can send content to two third parties. Record which provider handles which step in your data flow diagram.
Model provider credentials
Each model provider key used by a rerank or embedding model is stored in Dify and grants billable, data-bearing access to that provider. Scope keys per environment, restrict who in the workspace can view or change model settings, and rotate keys on staff changes. Confirm where Dify stores these credentials in your version and who can read them in the console.
SSRF proxy effects on model endpoints
Dify routes outbound requests through an SSRF proxy that blocks internal and private IP ranges. A self-hosted rerank or embedding server on a private address can fail if the proxy blocks it, so teams often widen the proxy rules, which weakens the protection. Add the specific endpoint to an approved allow list rather than opening private ranges, and test retrieval after each change.
Knowledge bases reachable by retrieval
Hybrid retrieval searches every knowledge base an app is bound to, so a broad binding can return sensitive chunks to users who should not see them, and reranking can rank them higher. Bind each app only to the knowledge bases its audience may read, and keep sensitive document sets in their own knowledge bases. Check the bindings after any app is copied or edited.
Secret key changes during tuning
The SECRET_KEY setting signs sessions and JWTs and encrypts OAuth credentials, and changing it after deployment logs all users out. Tuning work that touches environment files can lose this value if the file is regenerated. Keep the key in your secrets store, check it against .env before any restart, and include it in the backup plan.
Questions teams ask before tuning Dify hybrid search
How much does it cost to tune Dify hybrid search?
Most of the cost is labor, plus model usage that you pay separately. Tuning takes engineer time to build a question set, run retrieval tests and compare settings. The Dify Community edition that runs the tests is free, according to the Dify pricing page, but it is limited to a single workspace. Recurring costs come from the models. An embedding model and, if you add one, a rerank model are each billed by the provider per call, or they run on hardware you already own or buy. A rerank model adds cost to every query, so its bill grows with traffic. When you request a quote, ask for the labor split by phase (test set, tuning runs, final configuration and handover) and confirm which model fees you pay directly.
How long does tuning hybrid search take with a test set?
The effort depends on how many test questions you have, how many documents they touch and how many settings you compare. A small set of representative questions with known source documents keeps each tuning round short. Larger knowledge bases with mixed file types need more rounds, because weight changes affect different document sets in different ways. Ask any partner for a schedule tied to those counts rather than a single overall estimate. Also ask how many tuning rounds the plan includes and what happens when results need a second pass after the team reviews answers.
What must our team supply before tuning hybrid search starts?
Tuning needs inputs that only your organization can provide. Gather these before the work begins:
- Sample questions from real users, each with the document or passage that should answer it
- The knowledge base documents in their current form, including file types and any metadata already applied
- Your chosen embedding model and a decision on whether a rerank model is in scope
- A Dify test instance on the same version as production, so results carry over
- Named reviewers who can judge whether an answer is correct and complete
Missing reviewers or unlabeled questions are the most common cause of delay, because tuning cannot be judged without them.
Can our internal team tune Dify hybrid search without a partner?
Yes, if your team has administered Dify and can judge retrieval quality. The weight and rerank settings sit in the Dify console, so changing them is not the hard part. The skill lies in building a fair test set, reading why a document ranked where it did and telling a chunking problem from a retrieval setting problem. Teams with no one who owns evaluation often tune against a handful of favorite questions and miss failures in the rest of the knowledge base. A partner is useful when you need a fixed-scope tuning plan, a written test method and a handover your staff can repeat after a Dify upgrade.
When is a rerank model worth its ongoing cost?
A rerank model is worth testing when the right document is retrieved but ranks below less useful passages, and weight changes do not fix the order. Run your test set first with weights alone. If correct passages appear in the candidate results but fail to reach the answer, a reranker can reorder them, which can improve accuracy at the cost of added latency and model fees. If the correct passage never appears, a reranker cannot help, and the problem is usually chunking, indexing or the query. Compare both results on the same questions before you commit to the recurring cost.
Which rerank model should we choose for self-hosted Dify?
Choose the rerank model by four criteria: language coverage for your documents, the document types it handles well, latency your users will accept and where the model runs. A hosted rerank service bills per request and sends query and passage text to the provider. A self-hosted model keeps text inside your network but needs compute capacity, patching and an operator who monitors it. Confirm supported models and the provider configuration in the current Dify documentation for your version, because options change between releases. Then test two or three candidates on your own question set rather than relying on general benchmarks.
Does tuning hybrid search change our data handling or security?
Changing weights changes ranking only, so it does not move data by itself. Adding a rerank model does change data flow. Each query and the candidate passages are sent to whichever endpoint you configure, whether that is a provider in another jurisdiction or a model on your own hardware. Review that endpoint against your data classification before enabling it, and restrict who can change the setting in the console. Keep test questions and reviewer notes under the same access controls as the source documents, since they often quote sensitive content.
Where the official Dify documentation covers retrieval settings
Before you change hybrid search weights or add a rerank model, check the official Dify documentation for the version you actually run. Retrieval settings sit in the knowledge base configuration in the Dify console, and the behavior of weights, reranking, TopK and score threshold can change between minor releases. The Dify docs cover how apps and their knowledge connections are set up, and the langgenius/dify releases page lists what changed in each version. Minor releases arrive roughly monthly according to the release history, so a settings guide you read a few months ago may describe controls that have moved.
Start by confirming your running version. In a Docker Compose deployment, check the image tags in your deployment files and the version shown in the console, then open the release notes for every version between your current release and the target. Look for entries that mention knowledge retrieval, retrieval modes, rerank models, vector store changes or knowledge API behavior. Pay particular attention to security entries that affect how services talk to each other, because they can change what your retrieval test results mean after an upgrade.
Record the version and the release notes you reviewed alongside your retrieval test results. When you tune hybrid search, a result that holds on one release may not hold on another, so the evidence you keep should name the version it was measured on. Also read the upgrade guide for the target version and the self-hosting environment reference, since new variables may affect how your vector store is configured. Confirm any setting behavior described here against the docs for your exact release before you apply it to production.
How DeployOpen scopes and tunes Dify retrieval
DeployOpen treats hybrid search as a measured configuration decision rather than a default to accept. We scope the knowledge bases, test weights and reranking against real questions from your team, and document the settings so the choice can be reviewed later.
How we scope
- Inventory which knowledge bases feed each app and the question types they must answer well.
- Record the index method, retrieval strategy and whether a rerank model is in scope for each knowledge base.
- Confirm which embedding and rerank models are available inside your environment before the build starts.
How we build
- Configure semantic and keyword weights in the retrieval settings, then adjust them in small steps against a fixed question set.
- Add a rerank model only where testing shows the candidate ordering is the problem, and set TopK and score threshold in that phase.
- Connect the chosen settings to the apps through the Dify workflow so retrieval behaves the same in every environment.
How we hand over
- Deliver a written record of the chosen weights, rerank decision, TopK and threshold values, with the test questions used to justify them.
- Document how to repeat retrieval testing after a chunking change, a new model or a Dify upgrade.
- Walk your platform and application teams through the test results so they can make later changes with evidence.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, scoped in writing.
- Re-run retrieval tests when knowledge base content, models or Dify releases change the baseline.
- Confirm with your team, in the written scope, who approves changes to retrieval settings in production.
