Dify metadata filtering for knowledge retrieval starts with a field plan
A shared knowledge base that holds documents for several departments, products or time periods only gives correct answers when retrieval can be limited to the right slice. In Dify, document metadata is the mechanism for that. Each document can carry metadata fields of three types: *string* for labels such as department or product line, *number* for values you compare, and *time* for dates such as an effective or publication date. Retrieval conditions can then scope searches to documents whose fields match, so a finance question does not pull policy text from another unit. Confirm the exact condition operators and how untagged documents behave in the Dify docs for your version before you rely on them.
Teams need this before they need more tuning. If an answer comes from the wrong department's document, a different chunk size or rerank model will not repair it, because the failure happens at the scope step. Write the field plan first: the field name, its type, the allowed values for each string field, whether the field is required, who sets it, and what an untagged document should mean. Use a lowercase naming convention and a fixed value list for fields such as product, so one product does not appear under two spellings across uploads. Use a time field for date ranges, and reserve number fields for values you will actually compare, including numeric codes that you never compare.
The plan is hard to change later. Once documents are indexed, renaming a field, changing its type or correcting its values means revisiting every affected document, and naming and deletion rules apply across the whole knowledge base, so one scoping mistake can spread. Before any production upload, load a representative sample, run questions for each slice and confirm that answers stay inside it. Then review the release notes for each Dify minor release, which arrive roughly monthly, because retrieval behavior can change between versions. Treat the field plan as a versioned design document, signed off by the owner of each department, product or period it describes. Track changes in the Dify releases on GitHub page.
Metadata field types and when each fits
This table compares Dify's string, number and time metadata fields, along with the choices that affect all of them. Read each row as a question to confirm in your target Dify version before you design field names. The main takeaway is that the type you choose shapes which filters work later, so confirm operators and type-change rules first.
| Field type | Example use | What a filter can express | Main risk |
|---|---|---|---|
| String | Department, product line or document owner (illustrative) | Matches a value or set of values. Confirm the exact operators in your version. | Spelling variants such as Finance and finance split results. Use a controlled list of values. |
| Number | Policy revision, priority rank or product tier (illustrative) | Comparisons such as greater than or less than. Confirm which comparison operators are offered. | Numbers stored as text sort incorrectly, so 10 can appear before 9. Confirm the field is created as a number. |
| Time | Effective date, publication date or scheduled review date | Before, after or within a range. Confirm relative dates and time zone handling. | Dates entered as free text or without a time zone can return superseded documents as current. |
| Multiple values per document | A document that applies to several departments | Confirm whether a string field accepts more than one value per document. | A single-value field forces one primary owner and can hide the document from other teams. |
| Changing type after use | Converting a field once documents already carry values | Confirm whether the type can change after values exist. | If it cannot, every document value may need re-entry. Settle types before bulk loading. |
| Bulk editing | Applying one value to a large set of documents | Confirm which field types bulk edit supports and whether it overwrites existing values. | A mistyped value applied in bulk can change many documents at once. |
| Empty values | Documents that lack a field, such as older uploads | Confirm whether filters include or exclude documents with empty values. | Documents can drop out of scoped retrieval without any error, so test coverage per field. |
How to set up metadata filtering in a knowledge base
This sequence sets up metadata filtering so a knowledge base returns only the documents for the department, product or date range an app asks for. It moves from an agreed field catalog to a tested app, with a check at each step. The result is a tagged knowledge base and a repeatable test set that confirms scoped retrieval.
Agree the field catalog and naming convention
The knowledge owner, the team that maintains the documents and the app builders agree on the metadata fields before anything is created. Each field gets a name, a type (string, number or time), an allowed set of values and a named owner. Settle names early because renaming a field later affects every document that uses it, so confirm the rename behavior in your Dify version. Write the catalog as a short table, for example department, product and effective date, and have each app team sign off. Done when every planned filter maps to one field in the catalog.
Create the fields in the knowledge base
A knowledge base administrator adds each field from the catalog to the knowledge base's metadata settings and selects the type that matches its values. Use the exact spelling, casing and value list from the catalog, because a typo can create a separate value that filters will never match. Before loading content, confirm which field types your version supports and whether a field can be renamed or deleted once documents carry it. Done when the field list in the knowledge base matches the catalog one for one.
Tag a pilot set of documents
Pick a small, representative set of documents from each department or product and tag them by hand, one field at a time. Include documents that belong to several categories and a few boundary cases, such as a policy that covers two departments. Record each tagging decision in the catalog so the next person applies it the same way. Have a second reviewer check the pilot tags. Done when the reviewer finds no disagreement about which value applies to any pilot document.
Bulk edit metadata across remaining documents
After the pilot tags are approved, apply metadata to the rest of the knowledge base with bulk editing. Group documents by source folder, upload batch or naming pattern, and apply the approved values to each group. Compare the count of documents per value against the source inventory before and after each edit so unexpected documents surface early. Keep file names and folder structure consistent, since they are the most common basis for grouping. Done when every document carries the required fields and the counts match the inventory.
Test scoped retrieval in an app
Build or reuse a test app that connects to the knowledge base, then turn on metadata filtering for that app's retrieval. Ask questions that belong to one department or product and confirm the retrieved chunks come only from documents tagged with that value. Ask a question with a date condition and confirm that documents outside the range do not appear. Run the same questions without a filter to see how the results change. Done when each scoped question returns only the intended documents, and any failures are logged for correction.
Record the rules and gate new uploads
Publish the field catalog, the tagging rules and the scoped test questions as the reference for the knowledge base owner. Require every new document batch to carry the required fields before it is indexed, and rerun the test questions after any change to the catalog or the documents. Confirm who may create or delete fields and how deletion affects documents that already carry them. Done when a new batch can be checked against the catalog and the test set passes after the change.
Naming and deletion rules that affect every document
Metadata fields and values are shared across a knowledge base, so a single rename, deletion or bulk edit can change how many documents are retrieved. The cards below cover the rules to plan before tagging starts and the behaviors to confirm in your own Dify version.
Field names are shared
A metadata field belongs to the knowledge base, not to one document, so every document in that knowledge base sees the same field names. Choose names such as department or product_line once, with a short written definition, before the first upload. Renaming a field later is not a local edit: any retrieval filter, app configuration or API call that references the old name can stop matching. Confirm in your version whether a rename propagates to existing documents and whether dependent apps need updating.
One controlled vocabulary for values
Filtering works only when values match exactly, so Finance, finance and Fin produce three separate buckets. Maintain a short approved list of values per string field and assign it to a named owner. Put the list in a shared document that ingestion scripts read from, so manual uploads and API uploads use the same spelling. Test one known value against retrieval before you rely on the list.
Deletion removes values everywhere
Deleting a metadata field or a value removes it from every document that carries it, not only from the document you were viewing. Documents that relied on that value can then fall out of filtered retrieval without any warning in the app. Before deleting, export or record which documents carry the value and plan a replacement value. Confirm whether the deletion is immediate or requires re-indexing in your version.
Bulk edits apply broadly
Bulk metadata editing lets an operator change many documents in one action, which is efficient and also the fastest way to spread an error. A wrong department value applied to a selection can silently reroute documents to the wrong audience. Select documents by a verified query, run the change on a small test set first, and keep a record of the prior values. Confirm which selections the bulk tool supports in your version.
Access and sensitive data boundaries
Metadata often reveals more than the document text, such as customer segment, contract status or internal project codes. Field names and values are visible to anyone who can manage the knowledge base, so review them with the same care as the documents. Sensitive document sets belong in their own knowledge bases, because a shared field name cannot restrict who sees it. Confirm access behavior for metadata in your version and your identity setup.
Re-index and recovery checks
Some changes to metadata may take effect only after documents are re-indexed, and re-indexing costs time and embedding model usage. Record the knowledge base state, field list and value list before any rename or deletion so you can compare results afterward. Include a retrieval test with known questions in your change process. Confirm in your version whether metadata edits require re-indexing and how to roll back a mistaken change.
Bulk editing metadata across large document sets
Bulk editing is the right tool when a field was missing or inconsistent at upload and many documents need the same correction, such as a department value that was left blank for a batch or a product name that changed after a release. It is the wrong tool for values that differ document by document, because a bulk edit applies one value to every selected file. Before you start, export or note the current values for the affected set so you can see what changed and reverse it if needed. After the edit, sample a set of documents from different batches and check each metadata field against the source file, not against the list you just edited. Then filter the knowledge base by each field and scan the resulting value list for near-duplicates. A value like Finance next to finance or Fincance creates a new bucket that metadata filters will never match, so retrieval for that department quietly returns fewer results.
Record who made each change and when, and keep that record with the change request or ticket that justified it. Dify's own activity visibility for metadata edits may be limited, so confirm with your Dify version what is logged and where, and plan an external log (a ticket, a change record, or a script that writes the before and after values) for any edit that touches compliance-relevant fields such as data owner or classification. Re-run a few retrieval tests after each batch to confirm that scoped queries still return the expected documents.
Treat bulk editing as a repair step, not the long-term process. Ingestion pipelines should set metadata at upload, using values taken from the source system, a controlled list, or the upload form, so that new documents arrive with clean fields. Ask your implementation partner or internal team to define the allowed values for each field before the first load, and check whether the Knowledge API can set metadata on documents during ingestion in your version. If manual corrections keep recurring for the same field, the upload process is the thing to fix.
Security and data handling for metadata
Metadata fields look harmless, but they often carry the same sensitive labels as the documents they describe. The cards below cover what that means for editing rights, knowledge base separation, ingestion keys and where the metadata physically lives in a self-hosted Dify.
Metadata carries document sensitivity
Fields such as customer name, deal code or case identifier can reveal a client relationship even when the document body is redacted. Treat each metadata field as data with its own classification, and review the field list with the same people who approve the documents. If a field would be sensitive on its own, do not add it to a shared knowledge base just because it helps filtering. Ask the team which labels must never leave a restricted group.
Control who can edit fields
Editing metadata changes what retrieval returns, so an editor can widen or narrow which documents an app sees. Confirm in your Dify version who holds document edit rights in each knowledge base, and separate people who curate content from people who define field names. Record who changed a field value and when, using your own logging or database audit, because the console does not replace an audit trail. Illustrative example: a contributor with edit rights could relabel a restricted contract as a general document.
Separate knowledge bases by sensitivity
Metadata filters narrow results inside one knowledge base, but they are not an access boundary. Anyone who can query the knowledge base can often reach every document in it through a broad query, so documents of different sensitivity belong in separate knowledge bases with their own permissions. Use the access permissions topic to set those boundaries first, then apply metadata filters for relevance within each set.
Scope ingestion API keys tightly
Metadata often arrives through the Knowledge API, so the key that writes documents also writes their labels. Bind each key to the knowledge base it needs, store it server side, and rotate it when a pipeline owner changes. Confirm which key scopes your Dify version supports before you design the pipeline. A key that can create fields or update every document is a larger exposure than the pipeline needs.
Metadata lives in the metadata database
Field definitions and document metadata are stored in the metadata database you choose at deployment, which is PostgreSQL by default, with MySQL or OceanBase as alternatives. Your backup, encryption at rest, retention and restore procedures must cover that database as well as the vector store and file storage. Confirm with your platform team where it runs, who can reach it directly, and whether a restore would bring back deleted labels.
Keeping metadata stable across Dify upgrades
Filter behavior on metadata should be retested after every Dify upgrade, because minor releases arrive roughly monthly. According to the Dify releases page, 1.16.0 shipped on July 17, 2026, and 1.17.0 followed on August 25, 2026, with 1.17.1 on September 10, 2026. A retrieval setup that passed its checks in one release can behave differently a few weeks later, so treat each upgrade as a change to the retrieval path, not only to the containers. Confirm with your own test set that a filter on a department field, a product field and a date field still returns the same documents before you mark the upgrade complete.
Take a backup before every upgrade, as the Docker Compose upgrade guidance recommends. The backup should cover the metadata database, the vector store and file storage together, because a restore that brings back documents without their metadata values, or the reverse, will silently break scoped retrieval. Before upgrading, also check .env.example for new variables, since a changed default can alter how filters and retrieval run. Record the version you are leaving and the one you are moving to in the change ticket so a rollback has a clear target.
Keep a field catalog under version control. The catalog should list each metadata field name, its type (string, number or time), the allowed values for controlled fields, and which apps and API keys depend on it. Review the release notes for each target version for changes to retrieval, metadata or knowledge base handling, and compare them against the catalog. Any change that touches how fields are stored or matched should trigger a retest of the affected apps before rollout. Ask your DeployOpen or internal platform team to show where the catalog lives, who approves changes to it, and how it links to the test set.
Questions teams ask before scoping metadata filtering
How much effort does metadata filtering take, and what drives it?
Effort is driven mostly by the number of documents, the number of metadata fields, how clean the existing data already is, and how deeply the filters are tested. A small knowledge base with two or three well-defined fields is a contained job. A large corpus with inconsistent department names, mixed date formats and documents from many owners takes longer, because cleanup and bulk editing come before any retrieval testing. Ask any partner to break the quote down by these four drivers so you can see which one you can reduce internally. The effort is largely one-time, with recurring work only when new document types or fields appear.
Can I add metadata to documents that are already indexed?
Plan for it, but confirm the exact behavior for your Dify version before you commit. Dify supports document metadata fields of string, number and time types, and you can edit metadata on existing documents, including in bulk. Because filters depend on values stored with each document, the fields and naming should be settled before large batches are tagged. Changing a field name or type later usually means revisiting every affected document, so treat the field catalog as a design decision, not a formatting detail. Check the Dify docs and your own test environment before relying on any backfill approach.
Can my team do metadata filtering without an outside partner?
Yes, a team with Dify administration experience and a clear owner for the knowledge base can usually do it in-house. The work is mostly planning, data cleanup and testing rather than special software. The points where teams often need help are designing the field catalog so it holds up as the corpus grows, writing repeatable test questions that expose filter mistakes, and documenting the naming and deletion rules for other editors. If your staff are already maintaining the platform, the decision usually comes down to available time and whether an independent reviewer of the design is valuable to you.
What must our team provide before the work starts?
The engagement goes faster when your side supplies three things. First, named document owners for each knowledge base who can confirm what each file is and which department, product or date range it belongs to. Second, a field catalog: the metadata names, types, allowed values and who may edit them. Third, sample questions that real users ask, with the answers they expect, so retrieval can be checked against them. Also gather access requirements for each knowledge base, because filter design depends on who may see which documents. Your partner should list these inputs in the written scope.
How do metadata filters interact with access control?
Metadata filters narrow what retrieval returns, but they are not a substitute for access control. A user who can reach a knowledge base can usually query it, and a filter on department does not stop someone who can call the same knowledge base through another path from changing the filter. Sensitive document sets generally belong in separate knowledge bases with their own permissions and scoped API keys. Use metadata to organize and scope results within a set that a group is already allowed to see. Confirm the current permission model for your Dify edition and test it with accounts from each group.
What should we ask for in a fixed-scope quote?
Ask the partner to state the document count and sources in scope, the number of metadata fields and their types, the cleanup and bulk-editing work included, the test questions and pass criteria, and what is excluded. Request the written deliverables, including the field catalog, naming and deletion rules, and a test report. Ask who performs each task and what the client must supply, and what happens if the document count or source quality differs from what was assumed. Also ask whether the work is one-time or whether ongoing support, upgrades or retesting are proposed as a separate retainer. Pricing follows from this written scope, so a clear scope makes quotes comparable.
How DeployOpen scopes and delivers metadata filtering
DeployOpen treats metadata filtering as a design decision made before documents are uploaded, not a setting added later. We scope the metadata schema, build and test the filtering rules against your own retrieval questions, and hand over the schema, naming rules and runbooks your team needs to keep it working.
How we scope
- We inventory the document sets, the teams that query them and the fields (department, product, date) that should scope retrieval.
- We define the metadata schema, field types and naming conventions, and record which fields are string, number or time before any ingestion starts.
- Engagements are fixed-scope, so the written scope states the knowledge bases, deliverables and acceptance checks in advance.
How we build
- We configure the metadata fields in each knowledge base and apply them to a pilot document set before the wider load.
- We build bulk-editing and ingestion steps so metadata is applied consistently, and check that documents are not left with missing or conflicting values.
- We test filtered retrieval with real questions, confirm that scoped queries return only the intended documents, and record the results.
How we hand over
- We document the schema, naming and deletion rules, and the steps to change a field, so later edits do not silently break existing filters.
- We write runbooks for bulk metadata edits, reindexing and document removal, with a note on how each action affects every document in the set.
- We walk your engineers and knowledge owners through the handover so they can add sources and fields without repeating the design work.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, including checks after Dify minor releases that change knowledge base behavior.
- Where your team adds new departments, products or date ranges, we can review the schema and filter design before the change goes live.
- Confirm with us during scoping which Dify version and metadata database you run, since these shape the upgrade and backup steps we include.
