Dify database cleanup and storage reclamation

How Dify database cleanup keeps PostgreSQL from growing without limit

Cleanup in a self-hosted Dify instance requires three things: a retention policy for each data type, a backup taken before any deletion, and a planned VACUUM after large deletes. Dify is an LLM application development platform for building AI workflows, RAG pipelines and agents through a visual interface, and every app, conversation, workflow run and knowledge base record it creates lands in its metadata database. The default metadata database is PostgreSQL, so growth accumulates there, while uploaded documents and generated files accumulate in whichever file storage backend you configured, such as local storage, S3-compatible storage or another supported provider. The database and storage must be reviewed together, because deleting rows does not remove the files they point to.

This work becomes urgent in three situations. The first is when storage or database size alerts start firing and nobody can say which data type is responsible. The second is after a pilot moves into production, when message and run history that was tolerable during testing starts to dominate disk use. The third is before a compliance review asks how long conversation data and logs are kept, because you cannot answer that question without a written retention period that matches what the system actually deletes. Ask your team to confirm which Dify version you run before you start, since cleanup behavior and scheduled jobs differ between releases, and check the current Dify documentation for the cleanup settings your version exposes.

Set the policy first, then the mechanism. For each table or storage prefix, record the business reason for keeping it, the period, and who approves deletion. Take a consistent database backup and a copy of file storage, test that you can restore it, and only then run deletions in a maintenance window. After large deletes, PostgreSQL does not return disk space to the operating system automatically, so plan a VACUUM pass and check table size before and after. Confirm with your DBA whether a plain VACUUM or a stronger rewrite is appropriate for your table sizes, because the stronger option locks tables and needs its own window.

Cleanup options and when each one fits

This table compares five cleanup areas in a self-hosted Dify database and storage, showing what each removes, what cannot be recovered, the main risk and when to use it. Read each row as a decision: whether the data has a retention need that outweighs the cost of losing it. Confirm exact setting names for your version in the Dify docs and release notes before changing anything.

Cleanup area	What it removes and what it cannot recover	Main risk	Use it when
Message retention	Removes old chat messages and conversation records from PostgreSQL. Deleted conversation history cannot be recovered unless a prior backup exists.	Users or auditors lose history they still need, and reviewers cannot reconstruct how an answer was produced.	Your retention policy sets a defined window for end-user conversation data and you have confirmed the cleanup setting for your version.
Workflow run retention	Removes stored workflow run records, node execution details and related logs. Run-level debugging evidence is gone once cleaned.	Losing the trace needed to investigate a faulty or disputed workflow output.	You need bounded database growth from high-volume workflows and an export or log archive covers what investigators need.
Unused dataset and embedding cleanup	Removes knowledge base data and embeddings that no app references. Vectors must be regenerated by re-indexing if the dataset is needed again, which consumes model and worker capacity.	Deleting a dataset still referenced through an API integration or a shared app breaks retrieval silently.	Datasets are confirmed unused by owners, source documents remain available, and re-indexing cost is acceptable to the team.
Orphaned file removal	Removes uploaded files in object or local storage that no database record points to. Files removed this way cannot be reattached to a conversation or dataset.	A mismatch between the database and storage can delete files still in use, or leave storage growing if the match is never run.	You have verified that storage and metadata are consistent, and a backup exists from before the removal run.
PostgreSQL VACUUM after large deletes	Does not delete data. It reclaims dead row space inside PostgreSQL for reuse, and a full rewrite can return space to the operating system but locks tables.	Running a full rewrite during business hours can block the application, and expecting disk to shrink after a plain VACUUM is a common misreading.	Large cleanup has finished and the database files still hold space you need back. Schedule heavier operations in a maintenance window.

Retention settings by environment: development, staging and production

A development instance usually holds prototype apps, sample documents and test conversations that nobody needs to keep. Short retention is normally acceptable here because the data has little business value and the cost of losing it is low. The main risk is that developers assume their test data is safe, so the team should state the rules up front and make resets routine rather than exceptional.

Reset is the default: Agree that test data can be wiped on a schedule or on request, and say so in the team's onboarding notes.
Keep real data out: Do not load production documents, customer records or live API keys into development to make tests more realistic.
Use short cleanup intervals: Shorter retention keeps the database small and makes cleanup behavior easy to observe during early tuning.
Record the reset method: Note whether resets use a fresh install, a database restore or a scripted deletion, so the steps are repeatable.
Protect the environment anyway: Restrict access and keep the instance on the same network controls as other environments, since test systems often receive sensitive data by accident.

Before relying on development for cleanup experiments, confirm that no one has stored production content there and that resets leave no orphaned files behind.

Dify database cleanup setup order: backup, retention, then VACUUM

This sequence lets an operator shrink PostgreSQL and storage growth in a controlled order, so every deletion is preceded by a verified restore point and followed by proof that the application still works.

Record a baseline of every data store
Before any deletion, the operator records the PostgreSQL database size, the object storage usage for the file storage provider configured in the environment file, and the configured vector store (Weaviate by default, or another option such as Qdrant or pgvector if changed). Run a database size query in psql, for example checking the size of the Dify database with pg_database_size, and save the output with the date, the Dify version from the running images, and row counts for the message and workflow run tables. The step is done when these numbers sit in a change record that the later measurements can be compared against.
Take a consistent backup and test a restore
Stop writes or take a snapshot that captures the PostgreSQL database, the file storage and the vector data together, since the Dify Docker Compose docs advise a backup before every upgrade and the same discipline applies before deletions. Keep a copy of the .env file, because SECRET_KEY must match the restored data. Restore the backup into a separate environment and confirm that the application starts, that a knowledge base answers a query and that a known app loads. The step is done when the restore has been tested, not just when the backup job reports success.
Agree retention periods for each data type
Working with the application owners, the operator writes down how long each category of data must remain: conversation messages, workflow run records, uploaded files no longer linked to a knowledge base, and any embedding cache entries. Record who approved each period and which regulatory or internal rule drives it. Confirm the exact cleanup settings and scheduled jobs available in your Dify version against the environment variable reference and the Dify docs for maintenance tasks before changing anything. The step is done when a signed retention table exists for every data type being reduced.
Enable retention in staging and verify records
Apply the agreed retention settings in a staging instance restored from the backup, then run the cleanup once. Check that the application still responds, that published apps and knowledge bases still work, and that records inside the retention window are still present. Query the message and workflow run tables for rows older than the cutoff and confirm the count matches the expected deletions. The step is done when a written test result shows expected records remain and no user-facing feature breaks.
Enable retention in production during a planned window
Repeat the same settings in production only after staging passes, and schedule the first run in a window where the team can watch logs and the worker queue. Confirm the worker containers are healthy and that the cleanup job logs its progress. Keep the pre-change backup until the post-change checks in the next steps are complete. The step is done when the production logs show the job finished without errors and the apps respond normally.
Reclaim space with a planned VACUUM
Large deletes leave dead rows inside PostgreSQL tables, and the disk space they occupied is not returned to the operating system by the delete itself. Plan a VACUUM (or VACUUM FULL, which locks tables and needs a maintenance window and enough free disk for the rewritten table) on the affected tables, and run ANALYZE afterward so query planning stays accurate. Ask your database team to confirm whether autovacuum already handles this workload and what locking the chosen command causes. The step is done when the VACUUM completes and the table statistics are refreshed.
Re-measure against the baseline
Repeat the measurements from the first step: PostgreSQL database size, object storage usage and the vector store footprint, using the same queries and the same date ranges. Compare the results to the recorded baseline and note which numbers fell, which did not, and why. Keep the figures with the retention table and the backup reference so that the next review can repeat the comparison. The step is done when the before and after figures are documented side by side and any unexplained growth has an owner.

Why deleted rows do not shrink PostgreSQL files immediately

When PostgreSQL deletes a row, it marks that row version as dead but does not hand the space back to the operating system. The table's data files keep their size, and the freed space is reused for future inserts and updates inside the same table. A large cleanup of Dify message or workflow run records can therefore leave disk usage flat for a long time even though the tables now hold far fewer live rows. Before you judge a purge by disk usage, check row counts and dead tuple counts in the Dify metadata database, and compare the size of the database with the size of the individual tables that the cleanup touched.

A plain VACUUM removes dead row versions, updates the free space map and lets the table reuse that space in place. For ongoing retention, this is usually what you want, because it runs without blocking normal reads and writes. A rewrite that returns space to the operating system, such as VACUUM FULL, rebuilds the table and takes an exclusive lock that blocks reads and writes on that table for the duration. Plan it as downtime: stop or drain the Dify api and worker containers that use the table, take a backup first, confirm there is enough free disk for the new copy of the table, and schedule a window. Confirm the exact locking behavior and options for your PostgreSQL version in its documentation before you run either command.

Autovacuum normally runs in the background and handles routine cleanup, but a large purge can create more dead rows than its default settings expect to process promptly. Watch for tables where dead tuples keep growing and for autovacuum runs that fall behind during business hours. Tuning autovacuum for the affected tables may be the better long-term fix than manual runs, but changes to its settings should be tested in a non-production copy first. Confirm the current autovacuum parameters, their defaults and their per-table override options against the documentation for your PostgreSQL release.

Security and operations risks in deleting Dify data

Deleting data from a self-hosted Dify is simple to run and hard to undo cleanly. The risks below cover what remains in backups, who is allowed to delete, what cleanup leaves behind, and how changes are recorded.

Deleted data persists in backups

Once a conversation or workflow run row is removed from the live PostgreSQL database, copies still exist in every backup taken before the deletion. Those copies stay until your backup retention expires, so a cleanup job does not finish the erasure you may be asked to prove. Confirm your backup expiry period and whether restores can bring deleted records back into the live system.

Retention follows rules, not disk

Retention periods should come from the data owner's policy and the rules that apply to your sector, not from how much storage is free. A cleanup schedule set only to control growth can delete records a business or regulator still expects you to hold. Write the retention period for each data type, such as chat messages, workflow runs and uploaded files, and get it approved before enabling any job.

Cleanup credentials with delete rights

Cleanup needs database and object storage credentials that can delete rows and objects, which is a high-impact privilege. Scope these credentials to the tables and bucket prefixes the job touches, keep them separate from the application's runtime credentials, and log their use. Confirm who can read those credentials from the environment file or secret store.

Orphaned references after file removal

Deleting a file object from storage while its record remains in the metadata database leaves broken references that appear as failed uploads or retrieval errors. The reverse also happens when a record is removed but the object stays, which wastes storage without anyone noticing. Remove database references and storage objects in an order you have tested, and check for mismatches after each run.

Secrets are not cleared by cleanup

Data cleanup does not remove secrets held in the environment file, such as SECRET_KEY, or workspace encryption keys. Those values protect data that remains, and rotating them carelessly can break sessions or stored credentials. Treat secret rotation as a separate change with its own plan, and do not assume a cleanup run has reduced the sensitive material you hold.

Cleanup changes without a record

Cleanup jobs that run on a schedule with no change record make later investigations and audits difficult. Record each change with its date, the retention rule applied, the affected tables or buckets, the row counts and the approver, and link it to your change control system. Review these records regularly so that deletions match what was approved.

Dify database cleanup questions buyers ask before hiring help

What drives the cost of Dify database cleanup work?

Cost follows the amount of decision-making and verification the work needs, not the number of SQL statements. The main drivers are how many data classes need retention rules (messages, workflow runs, embedding cache, orphaned files), how many environments must be tested, how large the existing database and storage are, and whether backups are already restorable. Your team pays for retention decisions and sign-off. A partner is paid for analysis, a staged cleanup plan, the scripts or job configuration, and verification. Ask any quote to itemize each data class, each environment and the backup check separately.

Is database cleanup a one-time project or ongoing work?

It is both. The first pass reclaims space that has already accumulated and sets the retention rules. After that, cleanup runs on a recurring schedule, and the database still needs monitoring for growth, periodic VACUUM review after large deletes and checks after each Dify upgrade. The Dify docs describe monthly minor releases, so changes to schema or jobs can arrive regularly. Ask a provider whether the written scope covers only the initial pass or also ongoing monitoring under a separate retainer.

How long does a Dify cleanup engagement take, and what sets the duration?

No duration can be stated from the researched facts, so confirm it in the written scope. The main variables are database and object storage size, the number of retention classes, how long the team needs to approve each rule, whether a staging copy of production exists for trial runs, and how long large deletes take to finish on your hardware. Lock contention with a busy production system can also extend the window. Ask for a timeline tied to named milestones such as analysis, staging rehearsal and production run.

What must our internal team provide before cleanup starts?

Your team must provide a verified, restorable backup, read access to the PostgreSQL metadata database and the file storage bucket or volume, and a staging environment that resembles production. You also need a named owner who can approve retention periods for conversations, workflow runs and knowledge base files, since those rules depend on your compliance and business requirements. Provide maintenance windows, a change process, and the Dify version and configuration in use, including storage provider settings in .env. Request this list in the written scope so gaps are visible before work begins.

Can an operator clean up Dify storage alone using the Dify docs?

An experienced PostgreSQL and Docker operator can do much of it alone, but the Dify docs do not provide a complete reclamation procedure for every data class, so validation matters. The risk lies in deleting rows that other records reference, removing files still linked to knowledge bases, and assuming a vacuum returned space to the operating system. Start by reading the Dify maintenance documentation and the Dify FAQs, then rehearse every step in staging with a restored copy before touching production.

Does a DSL export keep usage logs so we can delete old data?

No. The Dify docs say DSL exports include app configuration, workflow nodes, model parameters, prompts and knowledge base connections, but not third-party API keys, knowledge base content or usage logs. An export is therefore useful for preserving app design, but it cannot replace a retention decision. If you need conversation or run history for audit, export or archive it through a separate process before any deletion, and record the approval. Confirm with your compliance owner which records must be kept and for how long.

Does the Community or Enterprise edition change the cleanup approach?

The core cleanup approach is the same because both editions run on the same PostgreSQL metadata database and file storage layout described in the Dify docs. The difference is licensing and support. The self-hosted Community edition is free and limited to a single workspace, while the Enterprise edition is custom priced and adds multiple workspaces and professional support, according to the Dify pricing page. Multi-workspace data needs per-tenant retention rules, so confirm with your Dify representative which retention and support options apply to your edition.

Limits to confirm in your Dify version and environment

Cleanup in a self-hosted Dify touches the metadata database, file storage and the vector store, and each one behaves differently. Dify ships minor releases roughly monthly, so the settings you tested last quarter may not match the release you run today. Treat each item below as a check to complete before you schedule deletes in production.

Cleanup settings and job names

Confirm which retention settings and scheduled cleanup jobs exist in the Dify release you run, along with their default values and whether they are enabled out of the box. Names and defaults can change between releases, so check the environment reference and the release notes for the exact version rather than copying settings from an older deployment. Record the outcome of a dry run on a staging copy before any job touches production data.

Metadata database engine and commands

Confirm whether your metadata database is PostgreSQL (the default), MySQL or OceanBase, because the maintenance commands and vacuum-style reclamation differ by engine. Ask which tables the cleanup jobs delete from and whether the engine reclaims disk space after deletes or only marks rows reusable. Write down the exact maintenance commands your engine needs and who runs them.

File storage deletion propagation

Confirm the file storage provider (OpenDAL, S3-compatible, Azure Blob, Google Cloud Storage or another supported option) and whether deleting a record in the database also deletes the stored object. If it does not, orphaned files will accumulate in the bucket or volume even after the database shrinks. Check bucket lifecycle rules, versioning settings and any storage-side retention that could keep deleted objects alive.

Vector store embedding removal

Confirm which vector store you use (Weaviate by default, or Qdrant, Milvus, pgvector, OpenSearch or Elasticsearch) and how embeddings are removed when a knowledge base document or dataset is deleted. Some stores keep index segments or tombstones until a compaction or optimize step runs, so disk use may not fall right after deletion. Test removal on one dataset and measure index size before and after.

PostgreSQL version support

Confirm the PostgreSQL version you run against the requirements for your edition. The Dify Enterprise resources checklist lists PostgreSQL 14 or later, and the Community edition documentation should be checked for its own minimum. Older versions may lack behavior your maintenance plan assumes, so verify the version before you plan large deletes and vacuum windows.

Managed PostgreSQL vacuum control

Confirm whether your PostgreSQL is managed by a cloud provider or platform team, and who controls autovacuum thresholds, maintenance windows and manual VACUUM permissions. Some managed services limit these settings or run maintenance on their own schedule, which changes when reclaimed space becomes visible to the operating system. Agree on who runs VACUUM after large deletes and how that work is scheduled.

How DeployOpen scopes and delivers Dify database cleanup

DeployOpen plans and implements database cleanup and storage reclamation for self-hosted Dify inside your own infrastructure, including PostgreSQL, file storage and vector data. The work starts from a written scope that names which data classes are retained, which are removed and who approves each deletion.

How we scope

Inventory the PostgreSQL tables, file storage paths and vector collections your Dify instance actually grows over time
Agree retention periods for message and workflow run records with the data owners before any deletion job is enabled
Record which cleanup jobs, embedding cache rules and orphaned file checks are in scope, and which are excluded
Confirm backup and restore coverage as a precondition, since cleanup is destructive and hard to reverse
Document the scope and deliverables in writing before work starts, as engagements are fixed-scope

How we build

Configure retention and cleanup settings against your Dify version, checking the release notes for changed behavior first
Test deletion logic on a restored copy of production data, measuring row counts and storage before and after
Plan VACUUM and reclamation windows for PostgreSQL after large deletes, with lock and load impact reviewed
Verify that orphaned file removal does not touch files still referenced by knowledge bases or apps
Run each cleanup step in a staging environment and record the results for sign-off

How we hand over

Deliver a runbook covering each cleanup job, its schedule, its owner and the rollback steps
Provide queries and checks that show table sizes, retained record ages and storage use over time
Document how to confirm cleanup outcomes after each Dify upgrade, since defaults and schema can change between releases
Walk operators through the restore path so they can recover from an over-aggressive deletion
Store configuration and scripts in your own version control so changes are reviewed

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Retainers can cover review of cleanup behavior when new Dify minor releases arrive, which come roughly monthly
Support can include periodic growth reviews of PostgreSQL and storage with your operations team
Confirm response expectations, service levels and included hours in the written retainer scope, not in this page