Why PDF, slide and email extraction decides knowledge base quality
The extraction choice decides how well a Dify knowledge base can be searched. In the standard pipeline, the ETL_TYPE environment variable selects the extractor. Its default, dify, handles txt, md, pdf, html, xlsx, docx and csv. Setting it to Unstructured adds doc, msg, eml, ppt, pptx, xml and epub, but it also requires UNSTRUCTURED_API_URL, so the extraction service must be reachable from your Dify deployment. In knowledge pipelines, the choice is made differently: you add an extraction tool plugin to the pipeline instead of setting a global variable. Either way, the reader decides which text, tables and slide content reach chunking and embedding, and errors at that stage are hard to see later.
The Unstructured and MinerU plugins are both listed on the Dify Marketplace. The Unstructured plugin converts documents into Markdown, JSON, HTML or text, and the MinerU plugin converts files into machine-readable formats. Both are useful for PDFs and slide decks where layout, tables and reading order matter. Because these extraction services can run inside your own network, sensitive email and contract files do not need to leave your environment for parsing. Before you commit, confirm with the plugin documentation which file types each plugin supports, how it reports failed or partial conversions, and which network endpoint and credentials it expects.
Treat extraction as a test-first decision. Pick a representative set of your own PDFs, slide decks and eml or msg files, run each candidate reader, and compare the output against the source documents before indexing a full corpus. Check table structure, headers and footers, speaker notes and attachments, since each can change retrieval results. Record the settings you chose, because changing the extractor later means re-processing documents that are already indexed. Confirm the supported versions of the extraction service with your vendor or the plugin listing before you schedule the rollout.
Which extraction path fits your document estate
If your document estate is limited to text, Markdown, PDF, HTML, spreadsheets, Word documents and CSV files, the default extraction path may be enough. With ETL_TYPE left at its default of dify, the platform reads these formats without a separate extraction service, so there is no additional container, endpoint or API key to operate in the deployment.
- No extra service: Nothing beyond the standard Dify deployment needs to run for these formats, which keeps the moving parts your team patches and monitors unchanged.
- Format coverage: The built-in path supports txt, md, pdf, html, xlsx, docx and csv. Confirm each file type you rely on against this list before you assume it will index.
- Upgrade exposure: Changes in extraction behavior arrive with Dify releases, so include a test ingestion of representative PDFs in your upgrade checks.
- Limits on layout: Complex PDFs with tables, columns or scanned pages may extract with less structure than you expect. Test those specific files rather than relying on the format name alone.
- Ownership: Your platform team keeps only the standard Dify upgrade and backup routine for this path.
Before committing, sample ten to twenty of your hardest real documents in a test workspace and review the extracted text against the originals.
If your knowledge base must ingest Word legacy files, Outlook messages, email files, PowerPoint decks, XML or EPUB content, you need to set ETL_TYPE to Unstructured and provide UNSTRUCTURED_API_URL. This changes the deployment itself: the extraction service becomes a dependency that must be reachable from the Dify services and kept in your network boundary.
- Added formats: This setting adds doc, msg, eml, ppt, pptx, xml and epub to the supported list, on top of the built-in formats.
- Required variables: Set
ETL_TYPEandUNSTRUCTURED_API_URLin the environment file, then confirm the variable names against the environment reference for your Dify version. - Network path: The API URL must resolve from the Dify api and worker containers. Check it from inside those containers, not only from the host.
- Operational ownership: Your team must version, patch, scale and back up the extraction service alongside Dify, and track its availability during ingestion jobs.
- Scope of change: The setting applies deployment-wide, so test the built-in formats again after switching, because your existing knowledge bases are not re-parsed automatically.
Confirm with your team whether emails and attachments can leave the Dify network for any reason before enabling this path.
The Unstructured and MinerU tool plugins from the Dify Marketplace let a knowledge pipeline call a parsing step you choose, instead of relying on the platform-wide setting. The Unstructured plugin converts documents into Markdown, JSON, HTML or text. The MinerU plugin converts files into machine-readable formats. Use this path when different knowledge bases need different parsing, or when one team needs a specific output shape.
- Per-pipeline control: The parsing step lives in the pipeline design, so one knowledge base can use one parser and another can use the default.
- Plugin installation: Install and configure the plugin from the Marketplace, then confirm its required credentials and endpoints against the plugin listing and its documentation.
- Output format decisions: Choose Markdown, JSON, HTML or text based on how your chunking and retrieval settings expect structure, and test both.
- Trust review: Treat each plugin as third-party code with access to document content, and record who approved it and which version runs.
- Dual maintenance: Pipeline logic now depends on the plugin version, so include it in your upgrade test plan.
Before building, check which plugin versions are compatible with your Dify release and where each plugin sends document content.
The three paths are not mutually exclusive. A common arrangement keeps the built-in parser for simple formats, enables Unstructured through ETL_TYPE for the legacy and office formats the platform cannot read, and uses a tool plugin in specific pipelines where layout or output shape matters. Each combination adds one more component to test and one more owner to name.
- Decide per file type: Map each file type in your estate to the path that reads it, and record the map so auditors and engineers can see it.
- Avoid duplicate parsing: Confirm which step runs for each knowledge base so the same file is not parsed twice with different results.
- Test the matrix: Run a fixed sample set through every path you enable and compare the extracted text.
- Document ownership: Name the team responsible for each extraction component and its upgrade schedule.
- Review on change: Repeat the mapping when you add a file type, change a Dify version or change a plugin.
Ask your implementation partner or internal team to produce this file-type map as a first deliverable before any environment changes.
What you need before changing the extraction path
Changing how Dify reads files touches the running deployment, the network, the configuration file and your test data. The cards below cover what should be in place before you edit ETL settings, so that a failed extraction can be traced to the right layer.
A running, supported Dify deployment
Start from a working self-hosted Dify on a version that is still current. Minor releases arrive roughly monthly, so check the GitHub releases page for the current line and its upgrade guide before you change anything. Confirm that document upload, indexing and retrieval already work with the built-in extractor, so you have a baseline to compare against. A broken deployment makes extraction results impossible to interpret.
Docker Compose host requirements
The Docker Compose deployment needs Docker Compose 2.24.0 or later, at least 2 CPU cores and 4 GB of RAM for the base stack. The stack already runs the api, worker, plugin daemon, PostgreSQL, Redis, Weaviate, Nginx, a code sandbox and an SSRF proxy. An extraction service adds its own memory and CPU needs on top, so size the host for both and confirm the figures against the vendor's guidance for the service you choose.
Where the Unstructured service will run
Decide whether the extraction service runs on the same host, another server in your data center, or a container platform you already operate. Consider who patches it, where logs go and whether it can be reached only from the Dify API and worker containers. Document documents may contain confidential or regulated content, so the location should match the same data handling rules that apply to the rest of your Dify data.
Network paths from Dify to the service
Map the connection from the Dify api and worker containers to the extraction endpoint, including DNS names, ports and any firewall or proxy in between. Dify's SSRF proxy blocks internal and private IP ranges by default, so an extraction service on a private address may need an explicit allowance. Test the path from inside the container network, not from your workstation, because that is where Dify will make the call.
Write access to the .env file
Changing extraction behavior depends on settings such as ETL_TYPE and UNSTRUCTURED_API_URL, which live in the Compose environment file. Confirm who can edit it, keep a copy of the current file under version control or in a secure backup, and check that secrets are not copied into tickets or shared channels. Recreate the affected containers after editing, and record which values changed so you can roll back quickly.
A representative test document set
Assemble a set of real but non-sensitive or approved PDFs, slide decks and email files that match your actual content. Include scanned pages, tables, multi-column layouts, embedded images and messages with attachments, since these are where extractors differ. Label expected outcomes for each file so you can compare text order, table structure and missing content before and after the change. Without this set, any improvement is anecdotal.
Vendor confirmation of the service image
Before deployment, confirm with the vendor the exact container image, its version and its license terms for the Unstructured service and for MinerU if you plan to use it. Ask whether the license permits your intended internal use and what it requires for redistribution or hosting. The Dify plugins and the extraction service are separate components, so check each one. Record the answers with the change request.
How to switch extraction in the order that keeps rollback simple
This sequence moves Dify from the built-in extractor to Unstructured or MinerU one change at a time, so each step can be verified and reversed if a file type fails. Every step ends with a check that tells the team whether to continue.
Back up the Dify environment and data
Before touching any setting, the operator who runs the deployment takes a backup of the database, the file storage and the
.envfile that holds the current configuration. Keep a dated copy of the existing.envso the previous ETL settings can be restored exactly. The step is done when the backup restores to a test host or when the team has confirmed the backup job completed and the files are readable. Dify's upgrade guidance also recommends a backup before every change, so this habit fits the same routine.Deploy the extraction service inside your network
The infrastructure team runs the Unstructured API service or the MinerU service on a host or cluster your Dify instance can reach, without sending documents to a public endpoint. Confirm which service versions your team has approved and record them with the configuration. The check is a direct request from the Dify api container to the service's health or base address, which should return a successful response. If the request fails, fix network routing and firewall rules before going further, because Dify cannot extract anything from a service it cannot reach.
Set ETL_TYPE and UNSTRUCTURED_API_URL
The Dify administrator edits the
.envfile to setETL_TYPEto Unstructured andUNSTRUCTURED_API_URLto the internal address confirmed in the previous step. Leave all other values unchanged so that any later problem points to this change alone. Check the environment variable reference for any other settings that depend on this one before saving. The step is done when the file diff shows only these intended changes and the values match the tested address.Restart the affected Dify services
The operator recreates the containers that read the environment file, which in a Docker Compose deployment is normally the api and worker services, and then confirms the rest of the stack came back healthy. Restarting only the containers the variable touches keeps the scope of any failure small. The check is that the api and worker containers show running status and that their logs contain no startup errors about the ETL configuration. If the logs report a missing or unreachable extraction URL, restore the previous
.envfrom the backup and stop here.Test one file of each newly supported type
The knowledge base owner uploads one sample each of doc, msg, eml, ppt, pptx, xml and epub into a test knowledge base, using non-sensitive files that reflect real layouts such as tables and slides. Review the resulting chunks in the dataset preview for readable text, correct order and no missing sections. Also check the extraction service's log for a line showing the call and its status, which confirms the request went to your own service and not to the built-in path. Files that fail stay out of production until the cause is understood.
Add and test the Unstructured or MinerU plugin
The Dify administrator installs the Unstructured or MinerU plugin from the Dify Marketplace and adds it as the extraction step in a knowledge pipeline. The pipeline is then run against the same test files, and the team compares chunk quality with the result from the environment-level setting. Confirm where each plugin sends file content, since the plugin may use a service address that differs from the one set in
.env. The step is done when the pipeline output matches the approved test set and the team has written down which configuration each knowledge base uses.Record the rollback point and approve production
The operator records the final configuration, the backup identifier, the service versions and the test results in the change record, then asks the data owner to approve the new extraction path for production knowledge bases. Keep the saved
.envcopy until the new path has run through normal use and an upgrade cycle. The step is done when the approver signs the record and the rollback steps have been written down, so any team member can restore the previous setting without guessing.
Which file types each extraction option reads
This table maps each file type to the built-in ETL, the Unstructured ETL setting, and the Marketplace plugin path, then lists what to check before relying on each route. Read across a row to see which path reads the format and what output to inspect. The main takeaway is that a supported extension does not guarantee faithful tables, slide notes or attachments, so test representative files on every route you plan to use.
| File type | Built-in ETL or Unstructured setting | Marketplace plugin path | What the team must verify |
|---|---|---|---|
| txt, md | Built-in ETL (default ETL_TYPE of dify) | Not needed for basic text | Encoding and heading structure survive chunking |
| Built-in ETL (default ETL_TYPE of dify) | Unstructured or MinerU for complex layouts | Multi-column text order, scanned pages and table fidelity | |
| html | Built-in ETL (default ETL_TYPE of dify) | Unstructured outputs Markdown, JSON, HTML or text | Navigation and boilerplate removed from indexed text |
| xlsx, csv | Built-in ETL (default ETL_TYPE of dify) | Not listed for the Marketplace plugins | Sheet names, merged cells and header rows map to readable records |
| docx | Built-in ETL (default ETL_TYPE of dify) | Unstructured or MinerU for embedded tables and images | Tables, footnotes and tracked changes appear as expected |
| doc, ppt, pptx | Requires ETL_TYPE set to Unstructured and UNSTRUCTURED_API_URL | Unstructured or MinerU as alternative converters | Slide notes, speaker text and table content are captured |
| msg, eml | Requires ETL_TYPE set to Unstructured and UNSTRUCTURED_API_URL | Unstructured for conversion to text or Markdown | Message body, headers and attachments are handled as intended |
| xml, epub | Requires ETL_TYPE set to Unstructured and UNSTRUCTURED_API_URL | Unstructured for conversion to text or Markdown | Element hierarchy and chapter order remain in the output |
Controls for the extraction service and the files it reads
Document extraction adds a network service and a parser that both handle untrusted files. The controls below cover where the service sits, what it can reach, how secrets and uploads are handled, and how email content is treated before it enters a knowledge base.
Keep the extraction service private
Run the Unstructured or MinerU service on a private network segment that only the Dify api and worker services can reach. Do not publish its port through the Nginx front end or a public load balancer. Confirm in your own environment which hosts can open connections to it, and record that scope with the rest of your Dify network design.
Confirm how the service authenticates requests
Dify 1.16.1 added token-based API authentication between services. Confirm whether that change applies to the extraction call path for your Unstructured or MinerU setup, and whether the extraction service itself checks a token or accepts any caller on its network. Ask the plugin or service vendor to document the exact header or key it expects before go-live.
Route outbound calls through the SSRF proxy
The SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges. Keep this path in place for any extraction request that fetches remote files or callbacks. Allow only the destinations the extraction service must reach, and test that a request to an internal address fails as expected.
Protect SECRET_KEY and plugin credentials
SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials. Changing it after deployment logs out all users, so set it once, store it in a secrets manager, and keep it out of shared configuration files. Store extraction API keys and plugin credentials as secrets in your deployment, not in app exports, since DSL exports do not include third-party API keys.
Limit who can upload to knowledge bases
Anyone who can add files to a knowledge base can push content through the extraction service. Limit dataset creation and upload rights to named roles, and review which workspace members can edit knowledge pipelines that call the extraction plugin. Record who uploaded which files so that a bad document can be traced back and removed from the index.
Treat email files as untrusted input
The msg and eml formats add attachments, embedded links and HTML bodies that the extraction step will parse. Treat these files as potentially containing sensitive or malicious content. Decide whether email uploads are allowed at all, strip or quarantine attachments before ingestion, and confirm with your security team how extracted text containing personal data is retained.
Failures that appear after the extraction change
This table pairs each failure operators commonly see after changing extraction with its likely cause, the check to run and the fix. Read it in order when a document upload or ingestion job fails, since the first rows are often configuration mistakes and the later rows usually point to the document itself or to plugin behavior.
| Symptom | Likely cause | Check and fix |
|---|---|---|
| File type rejected at upload | ETL_TYPE is still set to dify, which does not accept doc, msg, eml, ppt, pptx, xml or epub | Confirm ETL_TYPE is set to Unstructured in the environment file, then restart the api and worker services and retry the upload |
| Extraction job fails for every Unstructured file | UNSTRUCTURED_API_URL is missing, misspelled or points to a service the api container cannot reach | Check the variable value from inside the api container, confirm name resolution and the port, then correct the URL and restart |
| Unstructured calls blocked for an internal host | The SSRF proxy blocks requests to private IP ranges, which includes an extraction service on your internal network | Review the SSRF_PROXY settings and the proxy allowlist, add only the extraction host or network you control, and test again |
| Empty or near-empty chunks from some PDFs | Scanned or image-heavy PDFs contain little text layer, so the parser returns almost nothing | Sample the parsed output for a scanned file, confirm whether OCR is enabled in the extraction service, and route image-heavy files to a tool with OCR support |
| Slide text appears out of order | Presentation layout, text boxes and speaker notes are read in an order that differs from the visual reading order | Compare the parsed output with the slide, test a few decks, and adjust chunk boundaries or pre-process slides before indexing |
| Plugin output differs from the built-in parser | The Unstructured or MinerU plugin returns Markdown, JSON, HTML or text in a different structure than the dify parser's output | Re-index a sample set with each parser, compare retrieval results, and keep one parser per knowledge base so chunks stay consistent |
| Retrieval quality drops after re-indexing | Existing chunks came from the old parser, so new and old documents are split differently | Confirm which parser produced each document, plan a full re-index of the knowledge base, and budget for embedding calls you will repeat |
Questions buyers ask before changing document extraction
How much effort does switching Dify to Unstructured take?
The configuration change itself is small, but the effort sits in testing. Setting ETL_TYPE to Unstructured and pointing UNSTRUCTURED_API_URL at a service you run is a short edit to the environment file, followed by a restart. What drives the rest is your file mix, since the built-in extractor already handles txt, md, pdf, html, xlsx, docx and csv, while Unstructured adds doc, msg, eml, ppt, pptx, xml and epub. Volume, the size of the test set, and whether network owners must approve a new internal service also add time. Ask any partner to price the test set and the approval steps separately from the configuration work.
Can our own team make the extraction change without outside help?
Yes, if you have people who run Docker Compose deployments, understand the Dify environment file and can host an extraction service. The Dify docs and the environment variable reference describe the settings, and the change is reversible. Teams usually struggle with evaluation rather than configuration: deciding which documents must extract cleanly, comparing chunk output before and after, and checking how tables, slides and email attachments land in the knowledge base. A fixed-scope engagement from DeployOpen can cover that evaluation and the hosting work, with the scope and deliverables agreed in writing before work starts.
What do we need to provide for a document extraction project?
You need three things. First, read access to the environment file (or a copy with secrets removed) so the current ETL_TYPE and storage settings are known. Second, a representative document sample across the formats your teams upload, including scanned PDFs, slide decks and exported email, with any sensitive content handled under your own rules. Third, a named network owner who can approve the extraction service's placement, its outbound paths and any required proxy rules. Without the network owner, the project often stalls after the test phase, so confirm that role early.
How long does a document extraction change take?
The duration depends on scope, and no supplied source gives a figure for this work, so ask for a timeline tied to your inputs. A narrow change that only enables Unstructured for a few formats moves faster than a project that adds MinerU, tests a large corpus and needs new network approvals. Ask the provider to split the timeline into approval, hosting, configuration, test set runs and cutover, and to state which of those depend on your team's responses. Dify releases arrive roughly monthly, so also ask how the plan handles a version change during the work.
Can we use MinerU and Unstructured together in one Dify setup?
Dify's built-in ETL setting selects one extractor for the platform, so ETL_TYPE cannot be set to both at once. The Dify Marketplace lists Unstructured and MinerU as separate plugins, and the plugins sit within knowledge pipelines, so a pipeline can use the plugin that suits each source. Confirm the exact behavior for your Dify version in the plugin documentation before designing around it, and test how each route handles your file types, since the two tools convert documents into different output formats.
What does a fixed-scope extraction engagement from DeployOpen include?
A fixed-scope engagement covers the work listed in an agreed written scope, with deliverables and timelines set before work starts. For document extraction, that normally means reviewing the current ETL configuration, agreeing the format list and test set, hosting or configuring the chosen extraction service inside your network, running the test set, documenting the settings and rollback steps, and handing over operating notes. Anything outside the written scope, such as wider knowledge base redesign, requires a separate agreement. DeployOpen does not publish a price list, so a quote follows from the written scope.
What should we back up before changing the extraction settings?
Back up the database, file storage and the environment file before changing anything. The Dify Docker Compose docs recommend a backup before every upgrade, and the same caution applies to extraction changes because re-indexing documents changes what your knowledge bases retrieve. Keep a copy of the previous settings so the change can be reversed, and record which documents were re-processed. Confirm your backup and restore procedure in a test environment first, since a backup you have never restored is an untested control.
How DeployOpen scopes and delivers extraction changes
DeployOpen plans, implements, tests, documents and supports self-hosted open-source platforms inside your own infrastructure, so extraction changes for Dify are scoped against your file types, network and data handling before any build starts. Engagements are fixed-scope, with the deliverables and timelines agreed in writing first.
How we scope
- Inventory the file types your knowledge bases actually ingest, such as PDFs, slides, Word files and email, and which ones the built-in ETL already handles.
- Map where the extraction service would run, which outbound paths it needs, and which data classes may leave your network.
- Agree in writing which Dify version, plugins, ETL settings and test documents are in scope, with fixed deliverables and timelines.
How we build
- Set ETL_TYPE and UNSTRUCTURED_API_URL in a staging Dify instance, then route a representative document set through each extraction path.
- Install the Unstructured or MinerU tool plugins in knowledge pipelines only where the test results show a clear gain over the default path.
- Confirm the SSRF proxy and network rules allow the extraction service to be reached, while blocking any unplanned outbound calls.
How we hand over
- Provide a written record of the extraction configuration, plugin versions, network paths and the document set used for testing.
- Document the parsing results per file type, including known gaps, so your knowledge base owners know what to re-check after changes.
- Include upgrade and rollback notes for Dify releases, since minor versions arrive roughly monthly and may change plugin behavior.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer after handover, scoped separately from the original engagement.
- Re-run the document test set when Dify, the Unstructured service or the MinerU plugin is upgraded, and report any changed output.
- Confirm with your team and the vendors which extraction service versions and licenses apply to your environment before each change.
