Dify file upload limits for knowledge bases: why uploads fail before indexing starts
Dify file upload limits are set in several layers, so a rejected file may never reach the indexing pipeline. Before you load policy or product documents, confirm the default per-file size and batch file count for your target version in the Dify environment variable reference, and do not carry values over from older notes. A team usually needs to change these limits when large PDFs exceed the per-file size, when one batch holds more files than the default allows, or when scanned archives produce files far larger than their text content. Failures at these points often look like generic upload errors, so check each layer in order: the browser, the Nginx proxy in the Docker Compose stack, then the api service.
Changing a limit affects more than the form. The Nginx proxy has its own request body size setting, so a higher application limit does nothing if the proxy still rejects the request. Larger files also use more memory while the worker parses them and more object storage, whether you use the default storage or an S3-compatible, Azure Blob or Google Cloud Storage target. The largest effect often comes later: a large document produces more chunks, and every chunk sent to an embedding model adds indexing time and a per-token charge from your model provider. When you request a quote for ingestion work, ask who will tune these settings, who pays for the model usage during indexing, and how the team will re-index if the chunking changes.
The built-in extractor reads only some formats, so a supported extension does not guarantee a usable result. Confirm the list of formats your version reads in the Dify FAQs and in the release notes, then test one representative file of each type through retrieval testing before loading the full set. Scanned PDFs without a text layer usually need OCR or an extraction plugin before they index well. Very large files often need splitting by chapter or section, and exported office files may need cleanup to remove embedded images and repeated headers. Plan this preprocessing as part of the ingestion scope, because it changes the file count, the batch size and the timeline of the project.
Which upload limit settings to change for large document sets
This table compares the settings that decide how large a document set can reach a knowledge base and get indexed. Read each row as a control point along the upload path, from the browser through the proxy and storage to extraction and indexing. The main takeaway is that raising one limit usually requires matching changes further along the path. Confirm exact variable names and defaults for your version in the environment variable reference.
| Setting | What it controls | Change it when | What else it affects |
|---|---|---|---|
| Per-file size limit | Largest single document the knowledge base upload accepts | Source PDFs or exports regularly exceed the current cap | Proxy body size, parsing memory, indexing time per file |
| Batch file count limit | Number of files accepted in one upload action | Large folders need ingestion and the console becomes slow to work through | Size of each upload request, retry scope after a failed batch |
| Nginx client body size | Largest request body the reverse proxy forwards to Dify | Any increase to the per-file limit, or uploads fail before reaching Dify | Proxy buffering on disk, upload behavior after version upgrades |
| Proxy timeouts | How long the proxy waits while large uploads transfer | Large files stall or drop on slow or remote links | Open connection time, proxy load during bulk uploads |
| File storage provider | Where uploaded original files are kept | Volume, durability or sharing across nodes outgrows local disk | Storage growth, backup scope, retention, provider billing |
| API and worker capacity | CPU and memory for extraction and indexing jobs | Large PDFs fail to parse or indexing jobs queue for long periods | Indexing duration, embedding model call volume, queue depth |
| Vector store capacity | Size and write throughput of the vector index | Chunk count grows beyond what the current store handles comfortably | Query latency, index rebuild time, backup size |
| Host memory and disk | Container memory limits and temporary space during extraction | Containers restart during big file parsing or disk fills up | Other services on the same host, cleanup of temporary files |
Which file formats the built-in extractor reads and what to confirm
Dify accepts a file once it passes the size and count limits, but the extractor decides which text reaches the index. Extraction quality decides retrieval quality, and a successful upload says little about it. Confirm the exact supported list in the Dify documentation for your version before you set a document standard.
Plain text and Markdown
Plain text (.txt) and Markdown (.md) files carry the least extraction risk because the text is already readable without parsing. Chunking then works on the raw paragraphs and headings, so consistent heading levels you add at the source show up directly in chunk boundaries. Confirm in the Dify release notes and documentation for your version that both extensions appear in the supported list. Start new document sets in these formats when the source allows it.
Office documents
Word and PowerPoint files store text in packaged XML, so the result depends on how the built-in extractor handles tables, headers, footers, text boxes and embedded images. Content in those places can be dropped or flattened without any upload error. Export a sample of each document type to plain text, compare the resulting chunks with the source, and decide whether your document standard requires conversion first. A clean upload does not prove the content was indexed.
Spreadsheets and tabular data
Spreadsheets are read as tables, and the way rows become text decides whether a question about a specific value can be answered. A row separated from its column headers, a merged cell or a formula result can produce chunks that look complete but lose their meaning. Check whether your version reads every sheet, how it treats multiple sheets, and whether formula values and hidden rows survive. Test one representative sheet before loading the full set.
PDF files
PDFs range from text layers created by a word processor to scanned pages with no text at all. Simple single-column pages usually extract with their reading order intact, while multi-column layouts, tables and footnotes can be read out of order. Confirm whether your version reads PDFs directly, expects an extraction plugin, or needs an external conversion step. Sample pages from your most complex layouts before choosing a path.
HTML and saved web pages
HTML files and saved web pages contain navigation menus, cookie banners and script output alongside the article text. If the extractor keeps that boilerplate, it adds noise to chunks and can pull unrelated text into answers. Confirm how your version handles HTML markup, whether it removes page furniture, and what format crawled pages arrive in. Cleaning rules can remove repeated text, so decide these settings before ingestion to keep chunk counts stable on reruns.
Images and scanned documents
Scanned PDFs and standalone images have no text layer until optical character recognition runs. Whether that step is built in, supplied by a plugin, or must run before upload is a version-specific question to confirm. Recognition errors in names, numbers and tables pass straight into chunks, and a page that fails quietly will be missing from answers. Count the pages that produce text after extraction and compare that count with the source set.
Raising Dify file upload limits in a Docker Compose deployment
This sequence raises the upload limits for knowledge base documents in a self-hosted Docker Compose Dify, with a backup taken first and a retest that confirms indexing completes and the new documents reach the vector store and file storage.
Confirm the running Dify and Compose versions
The operator confirms the Dify image tags in use and the Docker Compose version on the host with
docker compose version. The Docker Compose deployment requires Docker Compose 2.24.0 or later, so anything older is upgraded before any limit changes. Compare the running Dify version against the upgrade guide on the GitHub Releases page so the upload settings are read from the same version's.env.example. Done when both versions are written in the change record.Back up the database and storage volumes
Before any edit, the operator takes a backup of the PostgreSQL database, the uploaded file storage, and the vector store data, along with a copy of the
.envfile and the Nginx configuration. The Dify Docker Compose guidance recommends a backup before every upgrade, and the same discipline applies to a configuration change. Confirm each backup file is present and readable, and record where it is stored. Done when a restore source exists for every data store the upload path touches.Record current upload behavior with a failing test file
Using a test account in a non-production knowledge base, the operator uploads one file just above the current per-file limit shown in the knowledge base settings, then uploads a batch with more files than the current batch count allows. Record the exact error text, the HTTP status, and whether the rejection came from the Dify interface or from Nginx by checking the Nginx access log and the api container log. This baseline shows which layer must change. Done when each rejection is matched to a specific layer in the record.
Change the application and Nginx limits together
The operator sets the new upload size and batch values in
.envusing the variable names listed in the.env.examplefile for the running version, then sets the Nginx request body size to a value at least as large as the new per-file limit. If the body size is smaller, Nginx rejects large files before Dify sees them, so the two settings are changed in the same change window. Confirm both files contain the matching values. Done when the application setting and the Nginx setting agree and the change is recorded.Recreate the affected containers
The operator recreates the api, worker, web and Nginx containers with
docker compose up -dso the new environment values and Nginx configuration reach the running services. Recreating the containers applies the configuration to processes that already loaded the old values. Then rundocker compose psto confirm every service reports healthy and check the api and worker logs for startup errors. Done when all core services are running and the logs show no configuration errors.Retest with a file and batch at the new limits
The operator uploads one file at the new per-file limit and a batch at the new file count into the same test knowledge base. Watch the document status until indexing completes, and check the worker log for failed embedding or extraction jobs. Then confirm the new documents appear in the vector store used by the deployment, which is Weaviate by default unless it was changed, and in the configured file storage location. Done when indexing completes with no failed documents and both stores hold the new files.
Check .env.example after any upgrade
After any Dify upgrade, and before the next limit change, the operator compares the current
.envfile with the.env.examplefor the new version to find added, renamed or removed variables. Upload limit settings can move between releases, so a value that worked before may be ignored after an upgrade. Record each difference, apply the required changes, and repeat the retest in step six. Done when the comparison is documented and the retest passes on the upgraded version.
Choosing a path by document set size and type
A handful of long manuals, specifications or policy documents usually fails for one of two reasons: the per-file limit in Dify or the request body size at the Nginx layer rejects the upload. Adjusting limits is often the first option because it keeps the work in the console and avoids a second ingestion system. The change is a configuration exercise, but the larger files also raise indexing load on the worker queue.
- Check both layers: confirm the per-file setting in Dify and the Nginx body size in your deployment, since either can reject the upload first.
- Set limits from your largest file: change values in
.envonly after you have chosen the biggest file you need to index, and record the reason in your change log. - Expect longer indexing: large files produce many chunks and longer embedding jobs, so the worker queue will be busy during the upload.
- Split at section boundaries: if a file is still too large, split it at chapter or heading breaks so the surrounding context stays intact.
After the change, confirm that each file uploads, finishes indexing and returns the expected passages in a retrieval test.
A large set of short files, such as support articles or product sheets, is limited less by file size than by how many files arrive per batch. Uploading through the console works for a few hundred files, but repeated batches are slow to run and hard to repeat. The Knowledge API is the usual choice when ingestion must be scripted, repeated or tied to a content system, and it moves the effort into code that your team owns.
- Batch in the console for one-off loads: split the set into batches that respect the batch limit, then upload and index each one.
- Script through the Knowledge API for repeat loads: write an ingestion job that creates or updates documents and reports failures per file.
- Keep API keys on the server: store credentials in your secrets system and never in the client, a pattern covered in the security hardening pages.
- Track document IDs: record which source file maps to which Dify document so updates replace content rather than duplicate it.
The operator effort is mostly the ingestion script and its error handling, which is a one-time build with ongoing maintenance when the source system changes. Afterward, compare document counts between the source and Dify and run retrieval tests on a sample from each batch.
A shared drive or export can contain Word files, spreadsheets, slides, plain text, HTML and older binary formats. The built-in extractor does not read every format equally well, so an archive that uploads without errors may still index unreadable or empty content. The safest starting point is to confirm the supported formats in the current Dify documentation for your version, then sort the archive by format before any upload.
- Sort by format first: group files into supported formats, convertible formats and formats that need a separate tool.
- Convert outside Dify: export unsupported files to text or Markdown with a script or a converter you control, and keep the originals in your records.
- Keep a manifest: list each source file, its format, the conversion applied and the resulting Dify document, so gaps are visible.
- Review the extractor output: open a sample of indexed documents and check for missing tables, lost headings or empty pages.
The effort is front-loaded in inventory and conversion, and it recurs whenever the archive changes. Verify the result with a test set that includes one file from every format group, and see the environment variable reference for the settings that affect uploads.
Scanned contracts, photographed forms and slide decks built from pictures often contain little or no extractable text. A file can upload and index without error while the knowledge base holds almost nothing searchable. Raising limits does not solve this, because the problem is the content itself. Decide early whether text must be recovered with optical character recognition outside Dify, or whether image handling inside Dify is part of the design, since the second route depends on models and plugins you have to select and govern.
- Test text presence first: open a sample of each scan type and confirm that text can be selected or searched in the original file.
- Apply OCR outside Dify when text is missing: run recognition with a tool your team can audit, then upload the text output with its page references.
- Preserve page and source identifiers: keep page numbers and file IDs in the converted text so answers can cite their origin.
- Check accuracy on sensitive content: recognition errors in figures, names or dates can change answers, so review a sample against the originals.
- Plan image indexing separately: if images must be retrieved, treat that as its own design decision, with model and plugin choices documented.
The effort is mainly the conversion pipeline and quality review, which recur as new scans arrive. Verify with retrieval questions whose answers exist only in scanned pages, and confirm that each answer traces back to the right source.
Security and operational risks of raising upload limits
Raising the upload limit moves load and exposure into parts of the stack that default settings keep small. Each card pairs one risk with the control that addresses it, from upload rights and proxy settings to storage sizing and review before indexing.
Memory and disk use in api and worker
Parsing and chunking a large file happens in the api and worker services, so memory and temporary disk use grow with file size and with the number of files in a batch. The Docker Compose deployment lists at least 2 CPU cores and 4 GB of RAM, which is a minimum for starting the stack and not a sizing for a large document set. Have your team run the largest expected file on the target hardware and record peak memory and disk use before setting worker counts for production.
Proxy timeouts on long uploads
Nginx sits in front of the api in the Docker Compose stack, so its body size limit and its timeouts both apply to every upload. Raising only the body size lets large files reach the application and then fail partway through when a timeout fires, which leaves a knowledge base with partial or missing documents. Change both settings together, then upload the largest expected file through the same path your users take and confirm it reaches an indexed state.
Broad upload access to knowledge bases
Anyone who can add files to a knowledge base can push large volumes into your storage and index, so upload rights set how much data enters the system. Limit who can create knowledge bases and who can add files to each one, and keep sensitive document sets in their own knowledge bases with narrower access. Name an owner for each knowledge base who approves what gets added, and review that list whenever team membership changes.
Sensitive text in extracted content
The extractor turns the text of each file into chunks that retrieval can return to any app that queries the knowledge base. A large batch can hide personal data, credentials or confidential passages inside hundreds of pages that nobody reads. Review before indexing: sample extracted text from each file type, remove material that should not be searchable, and move restricted content into a separate knowledge base. Confirm in a test knowledge base what file deletion removes in your version before you rely on it.
Outbound filtering does not cover uploads
The SSRF proxy routes outbound requests through a proxy that blocks internal and private IP ranges. A local file upload arrives inbound from a user's browser, so this filter does not guard that path and should not be counted as an upload control. Protect intake with network placement of the web and api services, authentication in front of the console and the access rules described above. Confirm the proxy's exact coverage for any source that fetches URLs.
Queue and disk monitoring during imports
Large batches push indexing work through the worker queue and fill the file storage and vector store faster than small uploads do. Watch queue depth, worker failures and free space on each volume for the metadata database, file storage and vector store, and set alerts before a volume fills. Take a backup before a large import, because a failed batch can leave partly indexed content behind, and the Docker Compose guidance recommends a backup before every upgrade for the same reason.
What raising upload limits costs and what it needs from your team
What drives the effort to change Dify upload limits?
The effort depends on how many layers must change together: the Dify application limit, the Nginx request body size in front of the api service, and the storage settings that receive uploaded files. The Dify version matters because settings can differ between releases, so confirm them against the documentation for your target version. Your storage backend (local storage, S3-compatible storage or another provider) and any load balancer or ingress you run in front of Dify also add work. Each layer needs a configuration change, a restart and a test with a file near the new ceiling before you call the change complete.
Can our team change Dify upload limits alone with the Docker Compose guide?
Often yes, if your team already runs Docker Compose and can read the environment reference. The Docker Compose guide requires at least 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later, and it starts the api, worker, web and Nginx services with their supporting components. Your team must apply each change consistently, record every key it edits and verify uploads through both the console and the API. Teams without a staging copy, or without a named owner for the reverse proxy and storage, usually carry the most risk.
How long does a test cycle take for large document sets?
No duration is published for this, so treat any figure you hear as something to confirm in writing. Cycle length depends on document volume, file sizes, the embedding model and its rate limits, and how many indexing jobs run at once. A useful test uses a representative set of files, for example (illustrative) a few very large PDFs alongside a batch of small ones. Measure indexing time, failed jobs and retrieval results, then repeat the test after each change. Before work starts, ask for the test set, the environment used, and the pass criteria written into the scope.
What recurring work follows after we raise upload limits?
The change adds work that continues after go-live:
- Monitoring indexing failures, worker queues and storage growth, since larger files consume more of each.
- Reviewing each Dify minor release, which arrives roughly monthly, and reapplying your settings against the upgrade guide for that version.
- Taking a backup before every upgrade, then checking
.env.examplefor new variables that your environment may need.
The recurring cost is mostly staff time and infrastructure. Plan an owner for these tasks before the first upgrade is due, not after.
What does raising upload limits cost, and who pays for it?
The Community edition of Dify is free for self-hosting and is limited to a single workspace, so the cost sits in the work and infrastructure around it. One-time costs include configuration, the test cycle and any scoping. Recurring costs include staff time for monitoring and upgrades, additional storage, compute for indexing, and embedding model usage that grows with document volume. Your own team pays in staff hours, while a partner is paid for the work in the written scope. DeployOpen works from fixed-scope agreements and does not publish prices, so a quote follows from that scope. Ask each bidder to itemize the layers, test runs, documentation and any retainer for ongoing support.
When should we bring in an implementation partner for upload limits?
Consider a partner when no one on your team owns the reverse proxy, storage and Dify configuration together, when the document sets contain regulated data that needs documented testing, or when you run several environments that must match. A partner is also useful when an upgrade is due in the same window. DeployOpen plans, implements, documents, backs up and upgrades self-hosted platforms inside your own cloud accounts, data centers or air-gapped networks. It prepares evidence for compliance assessments but does not issue audit reports or certifications. Ask for a written scope, deliverables and timelines before any work begins.
Does raising upload limits change our security exposure?
Yes, it can. Larger accepted files increase the load on workers and storage and widen the window for resource exhaustion, so decide who may upload before you raise any ceiling. Review which roles can add documents to each knowledge base, and confirm that outbound requests from document processing still pass through the proxy settings you intend to use. The Dify environment reference describes SSRF_PROXY behavior that routes outbound requests through a proxy blocking internal and private IP ranges, which you should verify in your own deployment. Record the new limits in your change log so auditors and operators can see when they changed.
Dify file upload limits to confirm for your version and environment
Upload limits in a self-hosted Dify come from several layers: Dify's own settings, the reverse proxy in front of it, the object store that holds the files, the embedding provider that processes them, and the extractor that reads each format. A limit at any one layer can stop a large document set, so confirm each layer for the exact release and environment you plan to run.
Dify's own upload settings
Confirm the default upload size and batch count for your exact release, and the environment variables that change them, in the Dify environment reference. Minor releases arrive roughly monthly, and defaults can move between them. Read the release notes for each version between your current build and your target, because a change to upload handling can ship as a security or maintenance item rather than a feature. Record the variable names and values in your .env file so that a later upgrade does not silently reset them. Confirm the variable names for your release before sizing any document set.
Nginx body size in your deployment
The Docker Compose stack places Nginx in front of the web and API services, so a request body larger than Nginx allows fails before Dify sees it. Check the body size directive in the Nginx configuration that ships with your release, then check any load balancer, ingress controller or cloud gateway you add in front of it, since each one can enforce its own limit. Your Dify limit and your proxy limit must agree, or the smaller value wins. Test an upload at the largest size you plan to accept, through the same path your users will take.
Storage provider object size rules
Dify stores uploaded files through the file storage provider you configure, which can be OpenDAL, S3-compatible storage, Azure Blob, Google Cloud Storage or another supported option. Each provider sets its own maximum single-object size, multipart upload rules and request timeouts, and those rules apply even when Dify accepts the file. Check the provider's documentation for your bucket or container type, and confirm that any lifecycle or retention policy does not delete files that a knowledge base still references. Verify the object store limit with a test upload to the real bucket.
Embedding model token and rate limits
A file can upload successfully and still fail during indexing. Indexing splits the document into chunks and sends them to the embedding model, so the provider's limits on tokens per request, requests per minute and daily quota all apply to the whole file set. Ask the provider for the current limits on the model your knowledge base uses, and confirm whether your contract applies a different tier. Watch the indexing status and worker logs during the test, since failures at this stage often look like generic indexing errors. Size batches against the embedding provider's limits, not only the upload limit.
Extraction behavior by file format
The built-in extractor reads some formats directly and handles others with less accuracy, and the supported list can change between releases. Confirm which formats your version extracts, and test representative files such as scanned PDFs, spreadsheets with merged cells, and documents with embedded tables or images. A file that uploads within the size limit may still produce empty or garbled text, which makes retrieval quality look like a model problem. Compare the extracted text in the knowledge base with the source document before you ingest the full set. Check extracted text on a sample of each format, not only the file count.
How DeployOpen scopes and delivers Dify upload and knowledge base configuration
DeployOpen treats Dify file upload limits as one part of a wider knowledge base design, covering document sizes, the proxy in front of the application, and extraction formats. Each engagement starts with a written scope, then configures, tests and documents the settings in your own infrastructure.
How we scope
- Inventory the document sets, file sizes, formats and batch patterns your teams plan to upload
- Record the Dify version, deployment method and reverse proxy in front of the application
- Agree the written scope, deliverables and timeline before any configuration work starts
How we build
- Set upload size and batch limits in the environment file and match the Nginx body size to them
- Test representative large files and mixed formats against extraction results and error messages
- Review storage, memory and worker settings that large document ingestion depends on
How we hand over
- Document each changed setting, its default, its new value and the reason for the change
- Provide a runbook for upgrades that checks new variables in the example environment file
- Walk your operators through a test upload and the troubleshooting steps for rejected files
How we support
- Re-check upload limits after each Dify minor release, which arrives roughly monthly
- Offer ongoing support, upgrades and continuous testing as a retainer after handover
- Confirm backups are taken before each upgrade so that configuration changes can be reversed
