Testing Dify retrieval quality before an app goes live
Start by separating retrieval from generation. When a knowledge base is loaded and the app answers oddly, the cause can sit in three places: how documents were split into chunks, how those chunks were indexed, or how the model used what it retrieved. Changing the model first is a common mistake, because it can hide the fault and adds cost to every test. Retrieval testing in the knowledge base lets your team send actual questions to the knowledge base and inspect which chunks come back, without generating an answer. Run it before an app goes live and again after any re-indexing, because a change to chunk settings rewrites the material the app searches.
Use the test to compare settings in a controlled way. Keep the question set and source documents fixed, then vary one setting at a time: chunk mode and maximum length, the index method, the retrieval strategy and whether a rerank model is applied. Record the top results for each variant. Confirm in the Dify version you run which of these settings the test panel can change and which must be set in the knowledge base or the app, since the console has changed between releases. Treat the result as evidence about retrieval only. Whether the live app follows the same retrieval path is a question for the docs at your version, so run a final check through the app itself before sign-off.
The pattern of results points to the cause. If the passage that answers a question is missing from the returned chunks, or its answer is split across two chunks, the problem is usually chunking or indexing, and a different model will not fix it. If the right passage comes back near the top and the model still answers wrongly or ignores it, the problem sits in the prompt or the model, and that is where to look next. Write down, for each test question, the passage that should be found, whether it appears and at what rank. That record turns a vague complaint into a decision, and it becomes the baseline your team re-runs after the next re-index or Dify upgrade.
Which retrieval setting to change first
This table compares the settings you can test in Dify retrieval, what each one changes, the symptom that points to it, and the cost or lock-in to check first. Read it top to bottom: fix chunking before retrieval strategy, and only add a rerank model once the candidate set is already close to right.
| Setting | What it changes | Symptom that points to it | Check before changing |
|---|---|---|---|
| Chunk mode and length | Size and boundaries of the text units that get embedded and returned | Answers cite a passage that is missing the sentence that answers the question, or pull in unrelated text around it | Chunk mode is fixed once indexed. Re-index the documents after a change. See the chunking topic for general versus parent-child mode. |
| Index method | Whether chunks are embedded (High Quality) or matched by keyword (Economical) | Paraphrased questions find nothing, or exact product codes and terms are missed | High Quality cannot revert to Economical, and embeddings add model cost at indexing. See the indexing topic. |
| Retrieval strategy | Semantic, full-text or hybrid search and the weights between them | Exact identifiers rank low while conceptually similar but wrong passages rank high, or the reverse | Changing strategy does not re-index, but results shift for every app using the knowledge base. See the hybrid search topic. |
| TopK | How many candidate chunks are passed to the model | The right passage appears in the logs but is not sent to the model, or too much noise dilutes the answer | Higher TopK raises prompt size and token cost per request. Check how it interacts with the rerank phase. |
| Score threshold | Minimum relevance a chunk needs to be kept | Relevant chunks are dropped on some questions, or the model receives low-relevance text | Thresholds depend on the scoring method, so set them only after testing with real questions. See the hybrid search topic. |
| Rerank model | Reorders the candidate chunks with a second model before they reach the answer model | The right chunk is retrieved but ranked below weaker chunks | Adds a model call per query and a provider or self-hosted endpoint to govern. Confirm the model's cost and data location. |
| Metadata filters | Narrows the search to documents with matching fields such as department or date | Answers mix content from the wrong product, region or period | Metadata naming and deletion rules affect every document. See the metadata filtering topic. |
Running a comparison in Dify step by step
This sequence turns a set of labelled questions into a comparison of chunking, index and rerank settings, so the team can see which configuration returns the right source passage and keep that configuration before the app goes live. Each step closes with a check so that no result is compared against a moving target.
Prepare a test knowledge base copy
Create a separate knowledge base in the same workspace and import the same source documents that the production app will use, keeping the file names, versions and metadata identical. Do not test on the live knowledge base, because changing its chunk settings would re-index content that other apps may already depend on. Record the document set, the Dify version and the embedding model in a test log so each later run can be traced. The step is complete when the document list in the copy matches the source inventory one for one and every document shows a completed indexing status.
Load the labelled question set
Build a list of real user questions, each paired with the document and passage that should answer it, and agree the list with the subject owner before testing starts. Include questions that users phrase badly, questions that need two passages and questions the knowledge base should refuse. Store the list in a version-controlled file so it can be reused unchanged for every configuration. The step is complete when every question has a named expected source and a second reviewer has checked a sample of the pairings.
Run the baseline retrieval test
Open the knowledge base retrieval testing view in Dify, enter each labelled question, and record the chunks returned with their rank and score under the baseline configuration. Use the same retrieval settings for the whole run, including the top K value and any score threshold, so that the baseline reflects only the chunking and index choices. The step is complete when every question has a recorded result row, including questions that returned nothing.
Record the expected chunk and its rank
For each question, note whether the expected passage appears in the returned results and at which rank, then mark the result as found, found low in the list, or missing. Keep a separate note of the chunk text that was returned when a wrong passage ranked first, because that text shows whether the problem lies in how the document was split or in how the question was matched. The step is complete when every question has a found, low or missing mark and the totals for each category are calculated for the baseline.
Change one setting at a time
Change a single variable, such as chunk length, overlap, the parent-child mode, the index method or the addition of a rerank model, then rebuild the test copy and repeat the full question run. Keep a change log that names the variable, its old value and its new value so that any improvement can be attributed to that one change. The step is complete when the new configuration has finished indexing and its results sit in a separate column beside the baseline, with the same question set.
Compare results against the same question set
Confirm before each comparison that the question list, document set and retrieval settings are identical to the baseline, except for the one variable being tested. If a run used a different question set or an extra document, discard it and rerun it. Compare the found and missing counts and the rank of the expected passage across configurations, and look for questions that moved from found to missing as well as the reverse. The step is complete when each configuration has a recorded result and any regression has been reviewed.
Freeze the chosen configuration
Select the configuration with the best recall on the labelled questions, weighing any regressions against the questions that matter most to the business, and write down every setting that differs from the defaults. Apply the same settings to the production knowledge base or to the app that uses it, and check that the indexing finished and the retrieval test reproduces the chosen results. Some settings, such as the index method and chunk mode, cannot be changed later without rebuilding the knowledge base, so confirm them with the owner before applying. The step is complete when the frozen configuration is saved with the test log and the production results match the test results.
How to tell a chunking problem from a model problem
Start by sorting each failed test question into one of three buckets, because each bucket points to a different owner and a different fix. The first bucket is the right text is never retrieved. Here the answer exists in the knowledge base, but the retrieval test does not return the passage that contains it in the results you would pass to the model. This is usually a chunking or indexing problem: the answer is split across boundaries, or the chunk contains the concept but not the terms the question uses. The second bucket is the right text is retrieved at low rank. The passage appears, but below the chunks that matter more, so a low TopK or a weak score threshold drops it. That points to ranking, hybrid weights or a rerank model. The third bucket is the right text is in the context, but the answer is wrong. Only here should you suspect the model or the prompt.
Chunk problems leave recognizable traces in the retrieved text itself, so open the retrieved chunks and read them before changing anything. Illustrative example: a policy answer whose condition sits at the end of one chunk and whose consequence starts the next, so neither chunk alone answers the question. Other common symptoms include a table cut mid-row, where the header is in one chunk and the values are in another, and a heading separated from the paragraphs it governs, so the chunk that matches the question has no context saying what it applies to. If the retrieved chunks are fragments like these, adjusting the prompt or switching models will not help. Check chunk mode and length in the knowledge base settings first, and confirm how your source documents were extracted. Parent-child settings and the document types they suit are covered in the chunking section, so use that section for the setting changes themselves.
Keep the test set fixed while you compare settings, so that a change in results can be traced to one change in configuration. Record for each failed question which bucket it fell into under each configuration, and treat a question that moves from bucket one to bucket three as progress, since the model now has the evidence it needed. Confirm the exact behavior of retrieval testing, the TopK and score threshold options and the model versions in use against your Dify version and the Dify docs, because option names and defaults change between releases. When a question still fails in bucket three after retrieval is sound, hand it to prompt and model review with the retrieved chunks attached, so the evidence travels with the ticket.
Signals that point to a chunking fault
When a Dify retrieval test returns the wrong passage, the cause is often how the document was split, not the language model that reads it. The cards below describe the patterns that point to a chunking fault, what each looks like in the test panel, and the first setting to check before changing anything else.
Answers split across chunks
The retrieval test returns two chunks that each hold half of the answer, or the correct sentence appears at the end of one chunk while its subject sits in the previous one. This usually means the maximum chunk length is too small for the content or the delimiter cuts at the wrong place. Check the maximum chunk length and the delimiter in General mode first, then compare against Parent-child mode, where a larger parent can hold the full procedure while a smaller child still matches the question.
Boundary truncation at chunk edges
Results begin mid-sentence, drop a condition from a rule, or end before a table row is complete. The text is intact inside Dify, but the boundary falls between related words. Check the chunk overlap setting in General mode, since overlap carries trailing context into the next chunk. Then open the document preview to confirm where each boundary falls before re-indexing.
Duplicated boilerplate chunks
The same header, footer, legal notice or navigation block appears in many results, pushing useful passages out of the TopK window. Repeated text also makes the index compete with itself on generic wording. Check the text preprocessing rules for removing repeated headers and footers, and review the source files for content that should be excluded before upload.
FAQ questions failing against paragraphs
A user question matches the topic of an FAQ entry but the answer ranks low, because the question text and the stored paragraph use different wording. The relevant passage is present, yet paragraph retrieval scores it poorly. Test the same questions in a knowledge base set to Q and A mode before concluding that the embedding model is weak, since that mode indexes questions against questions.
Right chunk, weak ranking
The correct chunk appears in the results but below unrelated passages. This points to the retrieval strategy or the absence of a reranker rather than to the split itself. Only after the chunk boundaries check out should you adjust hybrid weights or add a rerank model, and then retest the same question set so the change is measured.
Protecting test data and access during tuning
Retrieval tests run against the same documents and keys your production app will use, so the test environment carries the same data risk. The cards cover who may run tests, where keys live, and the operational items that must survive a tuning cycle intact.
Real customer questions in test logs
Retrieval tests use real customer questions to be representative, and those questions can contain personal data, account details or internal case references. Decide before testing whether questions are copied from production, anonymized, or written by the team. Check where Dify stores the test inputs and results in your deployment, and who can open them through the console. Confirm the retention period for those records in your own environment, since the docs here do not set one.
Who may run sensitive retrievals
Creating a knowledge base and using it in a test are separate rights, so a tester may have access to a sensitive document set without rights to edit it. Use one knowledge base per sensitive set and limit the members who can run retrieval tests against it. Document each named tester and remove access when the tuning cycle ends. Confirm in your deployment how workspace roles map to knowledge base access before you grant testers any sensitive data.
Dataset-scoped API keys stay server side
Scripted retrieval tests and ingestion often call the Knowledge API with a dataset-scoped key. The key grants access to a specific knowledge base, so it belongs in a server-side secret store or CI variable, never in a notebook, front-end bundle or shared test script. Rotate any key that was pasted into a chat tool or ticket during tuning. Confirm the key limits and binding behavior for your Dify version before you rely on them.
SECRET_KEY must survive the test cycle
The SECRET_KEY setting signs session cookies and JWTs and encrypts OAuth credentials, and the Dify environment reference says changing it after deployment logs out all users. Test and staging stacks often get rebuilt during tuning, so a regenerated key can silently break saved sign-ins. Record the key in your secret manager when the environment is created and restore the same value on any rebuilt stack. Confirm the impact on stored credentials before changing it.
Back up before changing chunk settings
Changing chunk mode, delimiters or the index method typically means reprocessing documents, and a failed run can leave a knowledge base in a state you did not plan. Take a backup of the metadata database and the file storage before each tuning change so you can return to a known configuration. The Dify Docker Compose docs recommend a backup before every upgrade, so apply the same habit to tuning. Confirm the restore steps in your own environment before the first test cycle.
Recheck tests after upgrades
Dify ships minor releases roughly monthly, and the Docker Compose docs say to follow the upgrade guide for the target version and to check .env.example for new variables. A retrieval baseline recorded on one version may not match results after an upgrade. Keep your test question set and expected results under version control, and rerun them after each upgrade before tuning further. Read the release notes for any retrieval or security changes in the target version.
Questions teams ask before testing retrieval
How much effort does a Dify retrieval test take?
The effort depends mainly on three things: the size of the question set, the number of documents in the knowledge base, and the number of configurations you compare. Each configuration needs the same questions run against it, and each result needs a person to judge whether the retrieved chunks actually contain the answer. A small pilot with a few dozen questions, one knowledge base and two or three settings is a different job from a full evaluation across several collections. Ask any partner or internal lead to size the work from the written question set and the list of settings to compare, and to state what the review of results will cost in reviewer hours.
Can our internal team test Dify retrieval without outside help?
Yes, a capable internal team can run retrieval testing in Dify using the built-in retrieval testing feature, provided someone owns the question set and the judgments. The hard part is rarely the clicks. It is writing realistic questions, agreeing what counts as a correct retrieved chunk, and keeping results comparable when settings change. Teams without an evaluation habit often test only questions they already know will work. An outside partner is most useful for designing the method, running the comparison and documenting the decision, but the team still needs to be able to rerun the test after its documents change.
What do we need from the business before the test starts?
The test needs labelled answers and named owners. Business owners should supply real questions from the users the app will serve, along with the source document or passage that answers each one. Subject matter experts should confirm which answers are acceptable and which documents are current, since retrieval tests against outdated material measure the wrong thing. You also need one decision owner who can accept or reject a configuration, and a reviewer who will score retrieved chunks consistently. Without these people, the results will show which settings return text, not which settings return the right text.
How long does retrieval testing take before a go-live decision?
The duration is set by how long it takes to assemble the question set, label expected sources, run each configuration and review results, plus any reruns after a change. None of these steps has a fixed length in Dify, so the honest answer depends on the size of your collection and how quickly your subject matter experts can respond. Plan the test as a gate in your release schedule, not as a single afternoon. Ask for a written schedule that separates the question-collection phase, the comparison runs and the sign-off review, and confirm who attends each one.
How do we tell a chunking problem from a model problem?
A chunking problem usually shows up at the retrieval step. The right passage exists in the document but is split across chunks, buried inside a chunk dominated by unrelated text, or missing from the top results even when the question matches its wording. A model problem usually shows up after retrieval: the correct chunk is returned, but the generated answer ignores it, invents details or misreads a table. Check retrieved chunks before you change the prompt or model. If the expected passage is absent from the returned set, adjust chunking, indexing or retrieval settings first.
When is a partner worth bringing in for retrieval testing?
A partner is worth considering when the knowledge base holds regulated, sensitive or high-volume content, when several teams will depend on one configuration, or when your team lacks someone to own evaluation over time. DeployOpen plans, implements, tunes and documents self-hosted platforms inside your own infrastructure, and can write the test method, run the comparison and hand over a repeatable procedure. Engagements are fixed-scope, with the deliverables and timeline agreed in writing before work starts. Ask for the written scope to show which questions, documents and settings are covered.
What should a written retrieval test scope include?
A useful scope lists the knowledge bases in scope, the number of questions and where they come from, the expected source passages, the configurations to compare, the scoring method and the acceptance criteria. It should also name who reviews results, how reruns are handled after document changes, and what is delivered at the end, such as the chosen settings and the test set for future regression runs. Ask for these items explicitly. A quote that cannot describe them is describing a different job.
Where the Dify documentation describes retrieval settings
Before you change chunking, index or rerank settings, read the current Dify documentation for the version you run. The knowledge base and retrieval behavior changes between releases, and a setting that works in one version may be labeled, defaulted or limited differently in another. The official Dify docs on app management describe how apps and their knowledge base connections are configured, which is the starting point for any retrieval test you plan to repeat later.
Dify ships minor releases roughly monthly, so the retrieval options you tested last quarter may have moved. Check the Dify releases on GitHub for the version you run, and read the release notes for changes to chunk modes, index methods, retrieval settings or the knowledge API. Record the exact version alongside each test result. A score comparison is only meaningful if the same build produced both sides of it.
Where the docs leave a behavior unclear, such as default values for TopK or score thresholds in your release, confirm it in the console of your own instance rather than relying on a memory of an earlier version. Ask your vendor or the Dify community channels to confirm anything the public docs do not state for your release, and keep those answers with your test plan so the next upgrade review can reuse them.
How DeployOpen scopes and delivers retrieval tuning
DeployOpen treats retrieval tuning as a written, fixed-scope piece of work: we agree which knowledge bases, questions and settings are in play before any changes are made. The result is a documented comparison of chunking, index and rerank choices that your team can repeat and own after handover.
How we scope
- Agree in writing which knowledge bases, document sets and apps are included in the retrieval test
- Collect a representative set of real user questions with expected source passages from your content owners
- Define the settings to compare, such as chunk mode, index method and rerank configuration, and what each run will record
- Confirm who signs off on acceptance criteria and which Dify version and environment the tests will run against
How we build
- Load the agreed question set into a repeatable test plan and run it against each candidate configuration
- Record retrieved chunks, ranking position and misses per question so every change can be traced to a result
- Separate chunking failures, such as split answers or missing context, from model failures, such as correct chunks answered poorly
- Check that test data stays inside your infrastructure and that API keys used for testing are scoped and kept server side
How we hand over
- Deliver a written report of configurations tested, results per question and the settings recommended for go-live
- Document the final knowledge base settings and the reasons behind each choice so future changes can be reviewed
- Provide the test question set and run procedure so your team can rerun the comparison after content or version changes
- Walk your platform and content teams through the results and the open items that still need a decision
How we support
- Offer ongoing support and upgrade-aware retesting as a retainer when new documents, Dify releases or models arrive
- Retest affected knowledge bases after each planned upgrade before settings are changed in production
- Agree the retainer scope, response expectations and review cadence in writing alongside the original engagement
- Ask us to confirm current retainer terms and availability during scoping, since these are set per engagement
