When Dify chunk summaries and cleaning rules improve recall
Chunk summaries and cleaning rules help when a knowledge base returns the wrong chunks because the question and the document use different words, or because boilerplate, repeated whitespace, URLs and email addresses dilute what each chunk says. Cleaning rules change the text before it is chunked and embedded, collapsing runs of whitespace and stripping URLs and email addresses. A chunk summary is generated text describing a chunk, which can be indexed alongside it so that a differently worded question still finds it. Both are preprocessing choices made at ingestion, not retrieval tuning such as weights or score thresholds. Confirm in your Dify version which cleaning options and summary settings the knowledge base exposes before you plan around them.
Decide per knowledge base, and test before you commit. Summaries address vocabulary mismatch, while cleaning addresses noise, so identify which problem your failing questions show. Run retrieval tests with real questions first, because a chunk that is wrong for the question will not be fixed by a summary. Stripping URLs and emails also removes content that some answers need, such as a support address or a policy link, so decide which fields are noise and which are facts your app must return. Ask whether a settings change applies to documents already indexed or only to documents processed afterward, since you may need to re-index to compare results.
The summary model is a cost you pay at indexing time. It runs for the chunks you produce, so the cost recurs whenever documents are added, changed or re-indexed, and it is billed by whichever model provider you connect. The cost rises with document volume, chunk count and how often you rebuild the index. Chunk text is also sent to that model, so choose a provider that your data classification allows. In a quote or internal plan, ask for the model you will use, the expected number of chunks, the re-indexing schedule and where the summary calls will be processed.
What each preprocessing setting changes
Dify offers several preprocessing options that alter the text stored for retrieval, and each one works differently. Reviewing them separately shows which change what the retriever can match and which leave the source document untouched.
Auto-generated chunk summaries
When summary generation is enabled, a model writes a short summary for each chunk and that summary is indexed alongside the chunk text, so questions phrased differently from the source can still match. The summary model runs at indexing time, so every document you ingest or re-index adds model calls and their cost. Check which model generates summaries, whether it is approved for your data, and how summaries are regenerated after you edit a chunk.
Whitespace collapsing
Collapsing whitespace replaces runs of spaces, tabs and blank lines in the stored text with single separators. It changes only the indexed copy, which reduces noise in chunk length and embedding input, but it can merge paragraph boundaries that a reader relied on. Test with tables and code blocks, where layout carries meaning, before enabling it across a whole knowledge base.
URL removal
URL removal strips web addresses from the indexed text so that long links and tracking parameters do not consume chunk space or dominate similarity scores. Removed links are gone from retrieval results, so an answer that cites a documentation link will not return it from the chunk. Keep the link in the source file and, if citations matter, confirm the application can retrieve the source document name instead.
Email address removal
Email removal deletes addresses from the indexed text, which reduces personal data held in the vector index and in retrieval logs. It does not remove names, phone numbers or identifiers that appear in other forms, so it is not a complete redaction control. Confirm with your privacy team what counts as personal data in your documents and whether a separate redaction step is needed before indexing.
Source files stay the source of truth
Cleaning and summary settings apply to the text Dify stores for retrieval, and the uploaded document remains the authoritative copy. Editing or deleting a document in the knowledge base changes the indexed chunks, while the file in your own storage is unchanged. Confirm this behavior against the Dify knowledge base documentation for the version you run, because preprocessing details can change between releases.
Which preprocessing options fit which content
This table compares chunk summaries, whitespace collapsing, URL stripping and email stripping against common content types in a Dify knowledge base. Read each row for the benefit, the risk to watch and the signal to check in retrieval testing before turning an option on across a whole dataset.
| Content type | Option | Benefit | Risk and decision signal |
|---|---|---|---|
| Policy manuals | Chunk summaries on | Summaries can name the policy area, which helps questions phrased in business terms match the right section | Summary model cost is paid at indexing time; keep it only if retrieval tests show better recall on paraphrased questions |
| Policy manuals | Whitespace collapsing on | Removes extra blank lines from PDF or Word exports so chunks hold more real text | Can merge headings into body text; check that section titles still appear in retrieved chunks |
| Support articles | URL stripping on | Removes long tracking links that add noise to chunk text and embeddings | Users may need the help link; if answers cite a URL, test whether the stripped answer still gives the right next step |
| Support articles | Chunk summaries on | Summaries can capture the symptom and product name for short troubleshooting articles | Summaries may repeat content already short; compare recall with and without summaries on the same question set |
| Code-heavy docs | Whitespace collapsing off | Preserves indentation and line structure that give code meaning | Collapsing can break code blocks and configuration examples; test retrieval of snippets before enabling |
| Code-heavy docs | URL stripping off | Keeps endpoint URLs and package sources that developers search for | Stripping can remove the exact host or path a user asks about; leave it off unless tests show no loss |
| Contact-rich records | Email stripping on | Reduces personal data sent to the embedding model and stored in the index | Stripping may remove the contact a user needs to reach; confirm the answer still points to the right team or process |
| Contact-rich records | Chunk summaries off | Avoids summary generation over personal data and keeps indexing cost lower | Summaries sent to a model add a second processing path for sensitive text; confirm the summary model is approved for that data |
What a summary model adds to indexing cost
Enabling chunk summaries means every chunk triggers one call to the summary model you configure during indexing. The cost therefore scales with three things: the number of chunks in the corpus, the size of each chunk sent to the model, and how often you re-index. A knowledge base of a few thousand short pages produces far fewer summary calls than the same content split into fine-grained chunks, so the chunk mode and length you chose in the chunking settings directly set the bill. Summary generation also adds indexing time, because each chunk waits on a model response before the document finishes processing. Confirm with your team how long a representative document set takes with summaries on and off before you commit to a schedule.
The cost has several components, and they are paid to different parties. The model usage is billed by your model provider, typically per token, so the provider's pricing page and the contract you hold with it are the figures to check. The indexing workers that run the job consume compute in your infrastructure, and that compute is a recurring cost wherever the workers run. Re-index runs are the component teams underestimate. Changing a delimiter, length or cleaning rule changes the chunks, and every changed chunk needs a new summary, so each tuning cycle repeats the model spend. The initial build is one-time; edits, new documents and re-indexing are recurring, and they grow with how often the content changes.
When you request a quote from DeployOpen, ask for the scope to state the corpus size in documents and approximate chunk count per knowledge base, the expected change rate, the summary model and provider, the number of planned tuning cycles, and where indexing workers run. Ask the implementer to show how the re-index cost is estimated, not just the first load. Because this estimate depends on your own corpus and provider rates, DeployOpen does not publish a figure for it. Before enabling summaries on a large knowledge base, run a small pilot with a sample of documents, record the call count and duration from that run, and multiply those measurements against your real corpus and provider pricing.
How to configure and test preprocessing in Dify
This sequence tests each preprocessing option against real questions on a copy of your data, so you roll out a cleaning or summary setting only after you have measured its effect on retrieval. It also shows what the setting costs at indexing time before it reaches production.
Select a representative document set
Choose a set of documents that reflects what the knowledge base will hold in production, including the messy cases: pages with repeated headers, footers, boilerplate, long URL lists and contact blocks with email addresses. The owner of the content and the team that will use the app should agree on the set together. You know this step is done when the set is written down by file name and the team has confirmed it covers the document types they expect to ingest.
Record baseline retrieval results
Before changing anything, run retrieval testing in the knowledge base against the current chunk, index and retrieval settings. Use a list of real questions collected from users or support tickets, and save the returned chunks and scores for each question. You know this step is done when you have a baseline record you can compare against later, with the settings used noted alongside it.
Create a test knowledge base
Create a separate knowledge base, or duplicate the existing one, and load the representative set into it so production content is not affected by experiments. Confirm the chunk mode, index method and retrieval settings match the baseline, so the only variable is preprocessing. You know this step is done when the test knowledge base shows the same documents and settings as the baseline, with the names of the experimental copies clearly marked.
Enable one preprocessing option at a time
In the test knowledge base, turn on a single cleaning rule, such as collapsing whitespace, removing URLs or removing email addresses, or enable automatic summary generation. Keep the others off so each effect can be traced to one change. Confirm the exact menu labels in the Dify version you run, because they can differ between releases. You know this step is done when the change is recorded in your test log with its date and the setting name.
Re-index and inspect the chunks
Save the change and re-index the test knowledge base, since changes to chunk and preprocessing settings apply only to newly indexed text. Open several documents in the console and read the chunk text and any generated summaries. Look for lost content, such as URLs that were needed as citations or email addresses that identify a support queue. You know this step is done when sampled chunks show the intended cleaning and no required information has been removed.
Compare results against the baseline
Run the same real questions through retrieval testing on the test knowledge base and compare the returned chunks with the baseline record. Note where the right chunk moved up or down, where a summary changed the match, and where a cleaning rule removed text the answer needs. Also record the cost of the summary model used at indexing time, because it is charged on every document indexed. You know this step is done when each option has a written result, improved, neutral or worse, with the questions that support it.
Roll out the approved settings
Apply only the options that improved or held results to the production knowledge base, then re-index the affected documents there. Keep the baseline settings recorded so you can restore them if answer quality drops. Plan the re-index as a change to production data, since existing chunks will not change until they are re-indexed. You know this step is done when production retrieval testing matches the approved test results and the change record lists the settings, questions and date.
Data handling and security considerations
Chunk summaries and cleaning rules touch document content at indexing time, so the choices here decide where that content travels and what is kept. The cards below cover the summary model, its credentials, logs and the retrieval effects of cleaning, and they point to the deployment controls to confirm in your own environment.
Summary model receives chunk text
When summaries are enabled, each chunk is sent to the summary model you configure, so that model's endpoint decides where document content goes. A hosted provider means the text leaves your network under that provider's contract and data terms. An on-premises inference server keeps the text inside your boundary. Confirm the endpoint, its region and its retention terms before indexing sensitive documents.
Credentials for the summary model
The summary model uses its own provider credential, separate from the chat models in your apps. Give it a dedicated key with the narrowest scope the provider offers, and rotate it on the same schedule as other model keys. Dify stores credentials encrypted with workspace keys, and SECRET_KEY is part of that protection, so confirm how it is set and backed up before go-live.
Prompt and output logging
Summary requests carry raw document text, and the model's output is stored as summary text in the knowledge base. Check what your summary provider logs, how long it keeps prompts and completions, and whether it trains on them. Also check what your own logging stack captures from the Dify services, since application logs can echo chunk content during errors.
Cleaning rules remove contact details
Removing URLs and email addresses during preprocessing shrinks the text sent to the summary model and the index, which helps with data minimization. The same rule can remove the contact details a support or legal knowledge base needs to answer questions, such as a helpdesk address. Run retrieval tests on sample questions that depend on those details before enabling the rules.
Cleaning is not redaction
Whitespace collapse and URL or email stripping do not remove personal data in general, such as names, account numbers or patient identifiers embedded in prose. Treat these rules as formatting and minimization steps. If a document set requires redaction, handle it in a separate step before upload and document that step as part of the control set.
Deployment controls to confirm
Outbound summary calls pass through the same network path as other model traffic, so the SSRF proxy settings and egress rules in your deployment determine what the summary model can reach. Confirm these controls in your own environment with your platform team, and keep knowledge bases with different sensitivity levels separate so one summary configuration does not cover them all.
Questions teams ask before enabling chunk summaries
Do chunk summaries cost more than standard indexing?
Yes, enabling summaries adds a model call for each chunk at indexing time, so indexing costs more than a run with summaries turned off. The extra spend is the summary model's usage on your chunks, plus the embedding work on the summary text if it is indexed. Summaries do not add a retrieval-time model call in the same way, so the cost lands mostly when documents are ingested. Ask your team to estimate it from your own chunk count and the rate your model provider charges for the summary model, because the supplied facts do not give a per-chunk figure. Track the first full indexing run so you have a real number before you scale the corpus.
What makes summary indexing cost more or less?
Cost follows the number of chunks, the length of each chunk and the price of the summary model. Four factors matter most:
- Corpus size and how often documents change, since every new or edited document is summarized again.
- Chunk size and overlap, because smaller chunks create more summary calls.
- The summary model you choose, including whether it is a hosted API or a model you run yourself.
- Cleaning rules, which remove boilerplate such as repeated headers, URLs and emails before summarization so the model processes less noise.
Reducing the number of documents you index with summaries, or limiting summaries to the sources that benefit most, is usually the simplest lever.
How long does re-indexing a corpus with summaries take?
Re-indexing time depends on how many chunks you have, how fast your summary model responds, how many workers process the queue and how large the documents are. The supplied facts do not give a duration, so treat any figure you hear as a question to test in your own environment. Run a pilot on a representative subset, measure the time per document, then scale the estimate. Your Dify worker capacity and any rate limits on your model provider will often set the pace more than the document count does. Plan re-indexing as a maintenance window, because changing chunking or summary settings usually means reprocessing existing documents.
Can our own team enable and tune chunk summaries without a partner?
Yes, a team with Dify administration skills and access to a summary model can enable the setting and run a pilot. The work is mostly configuration and measurement: choosing the summary model, setting cleaning rules, indexing a sample and comparing retrieval results. Teams struggle when they skip the measurement step and cannot tell whether a change helped. Someone must own the retrieval tests, the model credentials and the cost tracking. If your team already runs self-hosted Dify in production with a documented upgrade and backup routine, the added work is modest, although the decision still needs review by the people accountable for data handling.
What does the work need from our team?
Three inputs make the work go smoothly:
- Document owners who can confirm which sources are current, which contain sensitive content and which should be excluded from summarization.
- A set of real test questions, with the expected source passages, so you can compare retrieval before and after the change.
- Access to a summary model, with credentials stored in the platform's secret handling and a decision on whether the model may receive your document text.
The test questions matter most, because they are the only reliable way to judge whether summaries improved recall for your users.
Does a summary model send our documents to a third party?
It depends on the model you choose. A hosted provider receives the text of each chunk it summarizes, so the provider's terms, data location and retention rules apply to that content. A model you run inside your own infrastructure keeps the text in your environment. Confirm the data handling terms with the provider in writing before enabling summaries on restricted documents. Route outbound calls through the controls your team already uses, and remember that the SSRF proxy settings in Dify block internal and private address ranges for outbound requests. Exclude documents your policy does not allow any external model to read.
When is outside help worth paying for?
Outside help is worth considering when the corpus is large, the documents are sensitive, or several knowledge bases must share one retrieval design. A partner can write the scope, set up a retrieval test suite, run the pilot and document the cost model, and the work is usually fixed-scope with deliverables agreed in writing before it starts. DeployOpen implements, configures and documents self-hosted Dify inside your own infrastructure and prepares evidence for compliance reviews, but it does not issue audit reports or certifications. Ask for a written scope and a quote that separates one-time setup from recurring model and maintenance costs.
How do I know summaries actually improved retrieval?
Measure retrieval with the same questions before and after the change, and compare the sources returned. Summaries help when users ask in different words from the source text, but they can also add noise if the summary model drifts from the document's meaning. Record which questions improved, which got worse and whether failures point to chunking rather than the summary model. Repeat the test after each change to cleaning rules or the summary model, because a new model version can shift results. Keep the test set under version control so the comparison stays valid.
How DeployOpen scopes preprocessing work for Dify
DeployOpen plans, implements, documents and supports self-hosted Dify inside your own infrastructure, including the knowledge base settings that shape chunk summaries and text cleaning. We start from written requirements for your documents and questions, then agree the preprocessing scope before any indexing changes are made.
How we scope
- Inventory the document sources, formats and cleaning problems that affect your knowledge bases before choosing rules.
- Define which knowledge bases get summary generation, which model performs it, and who approves that model for indexing.
- Agree the written scope, deliverables and timeline with you before work starts, since engagements are fixed-scope.
How we build
- Configure whitespace collapsing and URL or email stripping rules, then test them against sample documents from your own corpus.
- Measure the indexing cost of the summary model on a representative batch so you can budget before full ingestion.
- Run retrieval testing with real questions to confirm that summaries improve recall and do not hide relevant text.
How we hand over
- Document each cleaning rule, summary model setting and the reason it was chosen, so your team can audit later changes.
- Record the re-indexing steps needed when rules or the summary model change, including expected effects on stored data.
- Walk your platform and data teams through the settings, test results and the checks they should repeat after upgrades.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer for preprocessing changes after launch.
- Review new Dify releases for changes to knowledge base indexing, then plan a backup before any upgrade of a running instance.
- Re-run retrieval tests when document sets, embedding models or summary models change, and report the differences.
