Why move Dify storage and vectors off local volumes
If your Compose Dify has outgrown a single host, or you are preparing a Kubernetes deployment, the move covers three things: uploaded files and private keys go to object storage, and knowledge vectors move from the bundled Weaviate to an external store such as Qdrant or Milvus. The work is done with the documented flask commands and environment settings, and its effort depends on three factors: how large your corpus is, whether the index must be rebuilt, and how many environments (staging, production, any cluster) must change. Confirm those three numbers before you estimate anything.
The metadata database (PostgreSQL by default) stays where it is, and the SECRET_KEY value must not change. Changing it after deployment logs out all users, and it also governs the encrypted OAuth credentials, so keep it identical across every environment you migrate. Also plan around the storage model of the vector store. Qdrant needs block-level storage with a POSIX-compatible file system and cannot run on network file systems or object storage such as S3, so object storage is for files and keys, while the Qdrant data volume still needs local SSD or NVMe-class block storage. Milvus uses object storage for its own data, which is a different design choice you should evaluate separately.
Read the environment variable reference for the current storage and vector store keys before you change anything, and record the Dify version you run. Upgrades and migrations interact, so the Docker Compose guide remains the reference for backups before each change.
What maps cleanly from local storage to object storage and Qdrant
This table compares each piece of state in a Compose Dify install with where it lands after the move. Read the first two columns to see what changes location, and the notes column for what must stay identical. The main takeaway is that uploaded files and knowledge vectors move to new backends, while SECRET_KEY and the encryption keys must carry over unchanged, and Qdrant itself still needs block storage.
| State | Platform equivalent | Migration notes |
|---|---|---|
| Uploaded files | S3-compatible, Azure Blob or Google Cloud Storage, set through the file storage options in the environment reference | Copy existing objects into the new bucket or container before cutover. Confirm the storage backend name and credentials in the environment reference for your version. |
| Private key directory | Local path kept for private keys, or an external store as the environment reference allows | Keep the key material and its file permissions intact. Do not regenerate keys during the move, because encrypted data depends on them. |
| Knowledge base vectors on Weaviate | Qdrant, set as the vector store, with existing datasets moved using the documented flask migration commands | Qdrant needs block storage with a POSIX file system and cannot use NFS or S3 for its own data. Plan a local SSD or NVMe volume for Qdrant. |
| Metadata database | PostgreSQL (default), MySQL or OceanBase, as listed in the environment reference | Vectors and files do not replace the database. Back up the database and restore it alongside the new storage so references still resolve. |
| SECRET_KEY | Same value in the new environment file | Do not change it. Changing SECRET_KEY after deployment logs out all users, and it is used to encrypt OAuth credentials. |
| Workspace encryption keys | Local storage or Azure Key Vault, per the environment reference | Keep the same key source and values. Changing the key source without re-encrypting stored credentials can make them unreadable. |
| Plugin and app configuration | Plugin daemon storage and app DSL exports | DSL exports omit third-party API keys and knowledge base content, so re-enter credentials and confirm knowledge base connections after the move. |
Documented tools for moving storage and vectors
Moving Dify off local volumes involves two separate data sets: uploaded files and private keys in file storage, and knowledge vectors in the vector store. The cards below cover the settings that select each backend, the flask commands and Qdrant paths that move or protect the data, and the copy step for local files. Confirm exact flask command names against your target Dify version before you run anything.
File storage backend settings
The environment variables that select where Dify keeps uploaded files and related data decide the target for your migration. Dify's environment reference lists OpenDAL, S3-compatible storage, Azure Blob and Google Cloud Storage among the file storage options. Before cutover, confirm the bucket, credentials and region in your staging .env, and check that the object store is reachable from the api and worker containers. Keep the file and key data paths in the migration plan separate so each can be verified on its own.
Vector store selection variables
The vector store setting chooses which backend holds knowledge vectors, with Weaviate as the default and Qdrant, Milvus, pgvector, OpenSearch and Elasticsearch as documented options. Changing this value alone does not move existing vectors, so the old and new stores must be reconciled before you switch. Record the current value, the target value and any per-dataset index settings in your change record. Confirm that the target store's connection variables are set in every service that reads them.
Flask vector migration commands
Dify's documented flask commands move existing knowledge vectors from one vector store to another. Confirm the exact command name, its options and whether it runs inside the api container against your target Dify version before you plan the window. Run it against a copy of the database and a staging vector store first, then compare document counts and sample retrieval results. Because re-embedding and reindexing take time, ask your team to estimate runtime from a staging run instead of relying on a published figure.
Qdrant snapshots for backup and restore
Qdrant documents snapshot creation and restore through its API, which gives you a point-in-time copy of collections before and after a migration. Take a snapshot of every target collection, store it outside the cluster, and test a restore into a separate instance. Qdrant cannot run on network file systems or object storage, so snapshots are a backup of data held on block storage rather than a substitute for it. Confirm snapshot retention and disk headroom with your storage team.
Qdrant migration tool for data import
Qdrant publishes a migration tool for moving data from other vector databases into Qdrant. Use it when you need to bring existing vector records into a new collection without rebuilding them from source documents. Map collection names, vector dimensions and distance metrics to match what Dify expects, and confirm payload fields survive the move. Validate by comparing record counts and running the same retrieval queries against both stores before you retire the old one.
Compose volume copy for local files
Uploaded files and keys sit in Docker volumes or bind mounts under the Compose project. Stop the services that write to them, copy the volume contents with ownership and permissions preserved, and verify checksums on the destination. Keep SECRET_KEY and any workspace encryption settings unchanged, since altering them can break access to stored data. Run a test read from the new location before you remove the old volume.
How to sequence the storage move with a parallel run and cutover
This sequence moves uploaded files, keys and knowledge vectors out of the bundled stack in an order that keeps a rollback path open until the new storage has been checked against real questions and sign-off has been given.
Inventory files, keys, datasets and vector counts
The platform owner and a Dify administrator record where each workspace stands today: the uploaded files and their storage location, the knowledge bases and documents in each, the vector count per dataset, the model and embedding settings, and the current values of SECRET_KEY and any workspace encryption settings. Confirm the exact variable names and vector store options against the Dify environment variable reference for your version. The output is a signed inventory that becomes the acceptance baseline for every later check.
Back up PostgreSQL, volumes and SECRET_KEY
Before any change, the operator takes a consistent PostgreSQL dump, a copy of the file and Weaviate volumes, and a secure record of the current SECRET_KEY, kept in a vault and not in the same location as the backup. Dify's Docker Compose guidance recommends a backup before every upgrade, and the same discipline applies here. The step is done when a restore of the backup has been test-run on a separate host and the team can read the key back. Store the backups under the retention policy your organization already applies to production data.
Provision object storage and a hardened Qdrant instance
The infrastructure team creates the S3-compatible or cloud object bucket for uploaded files and a Qdrant instance on block storage, not on NFS or object storage, since Qdrant does not run on those. Qdrant needs TLS and API keys enabled before it receives data, because self-deployed instances listen on all interfaces without authentication by default. The step closes when a test write and read succeed over TLS with a restricted key and the bucket permissions are confirmed as least privilege.
Stage a copy in a non-production Dify
The team restores the backup into a separate, non-production Dify stack with its own environment and copies the object storage configuration to a staging bucket. Running the file and vector migration here first shows what breaks before production data is touched. Confirm that the staging SECRET_KEY matches the production value only where the restore requires it, and that outbound model calls are blocked or pointed at test endpoints. The step is done when the staging instance opens every workspace and existing apps load without errors.
Migrate files and rebuild vectors, then test retrieval
In staging, the team moves uploaded files to object storage and rebuilds knowledge vectors in Qdrant using the documented flask commands for your version, with the vector store setting pointed at Qdrant. Rebuilding embeds documents again, so the model usage and its cost should be estimated with the model provider before the run. Then the team runs a fixed set of sample questions per knowledge base and compares retrieved chunks and counts with the baseline inventory. The step is done when the counts match and reviewers accept the answers on the sample set.
Run old and new stacks in parallel for read checks
The production team keeps the existing Weaviate-backed stack live while the migrated stack runs on a separate hostname for read-only checks by named testers. Testers run the same sample questions against both and log any difference in sources, ranking or missing documents. Writes stay on the old stack during this period so the rollback path is clean. The step closes when the logged differences are resolved or accepted in writing by the knowledge owners.
Cut over, then watch queues and logs
During an agreed window, the team stops ingestion, takes a final backup, runs the last incremental file and vector sync, switches the production environment variables to the new storage and vector store, and restarts services. Afterward, operators watch the worker queues, the API and worker logs, Qdrant health and retrieval errors, and confirm that document indexing jobs complete. The step is done when queues drain normally and the first set of real user questions returns expected sources.
Decommission Weaviate only after backup and sign-off
The Weaviate volume is not deleted at cutover. It stays in place, with a retained and tested backup, until the platform owner, the security lead and the business knowledge owners have signed off on the new stack. Only then is the volume removed and the change recorded in the configuration history. The step is complete when the retained backup location, its retention period and the sign-off record are stored with the migration documentation.
What drives the cost and duration of this migration
This table lists the cost components of moving Dify uploads, keys and knowledge vectors off local volumes and the bundled Weaviate. Read each row for who does the work, whether the cost is one-time or recurring, and what to request in a written scope before work starts.
| Cost component | One-time or recurring | What drives it | What to ask for in a quote |
|---|---|---|---|
| Corpus size and number of knowledge bases | One-time | Total uploaded files and indexed chunks per knowledge base set the export, copy and re-index windows, and each knowledge base may need its own validation pass. | Inventory of knowledge bases, document counts, total storage in the volume, and a per-knowledge-base cutover plan. |
| Embedding model re-run needs | One-time, with recurring cost if models change | If vectors cannot be carried over directly, documents must be re-embedded, so model fees or local inference capacity become a planned cost. | Whether vectors are migrated as-is or re-embedded, which embedding model is used, and who pays the model provider. |
| Qdrant topology | One-time build, recurring infrastructure | A single node is simpler to run but has no quorum. Qdrant recommends at least three nodes for production clusters, and a replication factor of 2 doubles storage. | Node count, replication factor, shard count, storage type (SSD or NVMe, block storage), and the cost of the chosen cloud or on-premises option. |
| Object storage access and network design | One-time design, recurring hosting | Private bucket layout, credentials scoped to Dify services, private endpoints, and egress paths determine how much network and access work is needed. | Bucket and IAM design, encryption settings, network paths between Dify and storage, and the recurring storage and request charges for your provider. |
| Kubernetes conversion | One-time, recurring operations | Moving from Compose to Kubernetes adds manifests or Helm values, persistent volume claims, ingress, and secret handling, which adds review and testing effort. | Target cluster version and distribution, the deliverable format (Helm or manifests), and who operates the cluster afterward. |
| Backup and restore testing | One-time test, recurring backup storage | A restore test that rebuilds the database, object storage and vector store together takes longer than a file copy, and backup retention drives storage cost. | Documented restore runbook, a full restore test in a non-production environment, backup retention period, and the schedule for repeated tests. |
| Internal security review of keys and TLS | One-time review, recurring rotation | Confirming SECRET_KEY handling, storage and vector API keys, TLS on Qdrant REST and gRPC, and Milvus authentication takes review cycles with your security team. | Scope of key inventory, the TLS certificate approach, rotation procedure, and evidence the review will produce for your team. |
What usually goes wrong and how to prevent it
Can changing SECRET_KEY during the move log everyone out or break stored credentials?
Yes, so keep the value unchanged. The Dify environment reference says SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials, and changing it after deployment logs out all users. Copy the value from the existing .env into the new environment before the first start, store it in your secrets manager, and record a checksum so you can confirm it matches. Do not regenerate it during the cutover. Test one stored OAuth or tool credential after the move to confirm it still decrypts.
Why must Qdrant not run on NFS or S3-backed volumes?
Qdrant needs block-level storage with a POSIX-compatible file system, and the Qdrant installation docs say it cannot run on network file systems such as NFS or on object storage such as S3. Before provisioning, confirm the volume type with your storage team and choose SSD or NVMe-backed block storage, as the docs recommend. Object storage is the right home for Dify uploaded files and keys, not for Qdrant data. Keep the two destinations separate in your architecture diagram and in the Compose or Helm values.
How do you secure Qdrant when it listens on all interfaces by default?
Treat the default as unsafe and configure access before any Dify traffic reaches it. The Qdrant security docs state that self-deployed instances listen on all network interfaces and have no authentication configured by default. Prevent exposure by:
- Binding the service to the private network and blocking it at the firewall and in Kubernetes network policies, so only the Dify api and worker can reach it.
- Enabling an admin API key and a read-only key, and storing them in your secrets manager.
- Enabling TLS for REST and gRPC, and JWT-based per-collection permissions where your Qdrant version supports them.
- Turning on audit logging of authenticated operations and forwarding those logs to your central store.
Does a Qdrant replication factor double the storage needed?
Yes, at a replication factor of 2 the cluster stores two copies of each shard, so storage roughly doubles. The Qdrant distributed deployment docs describe the replication factor as the number of extra copies kept across nodes. Size disks for the replication factor plus growth, not for the raw vector size alone. Recommend at least three nodes for production so quorum survives a node failure, and choose a shard count that is a multiple of the node count. Ask your Qdrant admin to confirm these values before you create collections, because changing them later means more work.
How do you confirm vector counts match after migration?
Compare counts per knowledge base before and after the move, then spot-check retrieval results. Export a count for every dataset from the source Weaviate instance and from the target Qdrant collection, and treat any gap as a failed migration until explained. Then run a fixed set of test queries against known documents and compare the top results. Keep the dataset list, counts and query outputs as a signed record in your change ticket. If counts differ, check for documents that failed to re-index or were deleted during cutover before rerunning the migration.
What happens when variables in .env.example are missing after an upgrade?
The new release may fail to start or behave differently because a required setting is absent. The Dify Docker Compose docs advise checking .env.example for new variables after every upgrade. Diff your current .env against the example for the target version, add new keys with values chosen by your team, and keep the storage and vector settings you changed (for example the file storage type, S3 endpoint and vector store selection). Take a backup before the upgrade, as the docs recommend, so you can restore the previous state if the diff misses something.
Why do dataset index settings override the global vector store?
A dataset can keep its own index type, which can take precedence over the global VECTOR_STORE value, so some knowledge bases may still query the old store. Before cutover, list every dataset and its index configuration in the Dify console or through the documented API, then compare it with your target store. Use the documented flask vdb-migrate path for each affected dataset rather than assuming the global change moved everything. After migration, confirm each dataset queries the intended store by running a retrieval test on each one and checking the Weaviate container for unexpected traffic.
Should you ask a partner to run the storage and vector migration?
Ask a partner when your team lacks time to run the checks above, or when the migration must be documented for an audit. Scope it in writing: which storage and vector targets are in scope, how keys and volumes are moved, the validation method, the backup and rollback plan, and who signs off each stage. DeployOpen plans, implements, backs up, documents and supports self-hosted open-source platforms in your infrastructure, so the work can be fixed-scope with deliverables agreed before it starts. Cost depends on the number of knowledge bases, the data volume, the number of environments and how many sign-off checks you require. Ask for a quote that lists each of these items separately.
When Milvus is the better fit than Qdrant
Choose Milvus over Qdrant when the knowledge vector store must grow beyond one machine on infrastructure that already runs object storage and a managed Kubernetes platform. Milvus separates compute from storage: its stateless proxies, coordinator and worker nodes keep metadata in etcd and persist data in MinIO, S3 or Azure Blob, with Woodpecker, Kafka or Pulsar as the write-ahead log. That design fits teams with object storage skills and a Kubernetes-native operating model. It also offers Standalone and Distributed modes, so a small pilot can start on one machine and grow later. The supported index set (HNSW, IVF, FLAT, SCANN and DiskANN) gives you more choices when memory and disk cost trade against recall.
Qdrant fits better when your storage team can provide block-level, POSIX-compatible volumes on SSD or NVMe, because Qdrant cannot run on NFS or S3. Its distributed mode uses Raft for cluster topology and needs at least three nodes for production quorum. Its access model is simpler to reason about: admin and read-only API keys, TLS, and JWT-based per-collection read or write permissions. Note that self-deployed Qdrant listens on all interfaces with no authentication until you configure it, so it must be secured before it joins the Dify network. Milvus also documents authentication, TLS and role-based access control, so compare the two against your own role model.
Before you choose, confirm in your target Dify version which vector stores it supports for knowledge bases and the exact environment settings each one needs, since support can differ between releases. Then test both candidates with a copy of your largest knowledge base. Measure restore time, query latency under your filters, and the operational steps your team would own, such as etcd and object storage backups for Milvus or snapshots and node replacement for Qdrant. Record those results in the migration plan so the choice rests on your environment rather than on feature lists alone.
Official sources for this migration
The Dify configuration reference and Docker Compose docs define the variables and upgrade steps that govern storage and vector settings. The Qdrant and Milvus docs define what each target store needs from its disks, access controls and cluster layout. Read them together before you change a production stack.
Dify environment variable reference
This is the reference for the settings that decide where uploaded files and keys live and which vector store Dify uses. Check the file storage options, the vector store defaults and how SECRET_KEY affects existing data before you change anything. Confirm the exact variable names for your Dify version in the reference, because they are the ones you will put in .env.
Dify Docker Compose deployment
This guide covers the stack you are migrating from, including the bundled Weaviate container and the upgrade steps that call for a backup first. Use it to confirm which services you can remove or repoint after the move. Check .env.example for new variables after you upgrade, since storage settings can change between releases.
Qdrant installation requirements
Qdrant needs block-level storage with a POSIX-compatible file system and does not run on NFS or S3. This rules out object storage for vector data even though you are moving Dify files there. Confirm the disk type and mount for your target nodes before you plan the new vector tier.
Qdrant security defaults
By default, self-deployed Qdrant listens on all network interfaces and has no authentication. Plan API keys, TLS and per-collection access before Dify connects, and keep the service off public networks. Confirm which key types your Qdrant version supports and how Dify passes the key.
Qdrant snapshots for rollback
Snapshots give you a restore point for vector collections before you re-index or switch Dify to the new store. Test a restore into a scratch instance, not only the snapshot creation step. Record the collection names and vector counts so you can check them after the move.
Milvus architecture overview
If you choose Milvus, this overview shows that object storage, etcd metadata and a write-ahead log are separate components you must run and back up. Plan these as part of the target design, not as a single container. Confirm the deployment mode that fits your scale before you pick Milvus over Qdrant.
How DeployOpen scopes and delivers this migration
DeployOpen plans the move of Dify uploaded files, private keys and knowledge vectors as one written scope, with each storage change tested on a copy before production. The work runs inside your own cloud accounts, data centers or air-gapped networks, with backups and validation agreed up front.
How we scope
- We inventory the current volumes, the knowledge bases that hold vectors, and the SECRET_KEY and key storage in use today.
- We agree in writing which object storage provider, vector store and Dify version the migration targets.
- Engagements are fixed-scope, and the written scope names the flask commands, validation checks and handover items included.
How we build
- We stand up the target object storage and vector store in a staging copy of your Dify deployment first.
- We run the documented migration steps on that copy, then compare file counts, knowledge base retrieval and app behavior against the source.
- We prepare the production change plan, rollback steps and backup checkpoints, and run the cutover only after your review of the staging results.
How we hand over
- We deliver updated environment files, storage configuration and a runbook that records each step, parameter and verification result.
- We document the backup and restore procedure for object storage, the vector store snapshots and the Dify database.
- We walk your platform and security teams through the validation checklist and the operating notes for the new components.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers after handover.
- Retainers can cover upgrade planning for new Dify releases and checks of .env.example changes after each upgrade.
- We can review storage access policies, vector store API keys and backup restore tests on a schedule your team agrees.
