Dify scheduled maintenance tasks

Which Dify scheduled maintenance tasks to enable in production

The scheduled jobs in a self-hosted Dify are Celery Beat tasks that clean up or check the platform, and each one changes what data is kept. Some remove old records from PostgreSQL, some clear stored files that no longer have a referring record, and some check for newer plugin versions or poll for scheduled work. Before you enable any of them, decide your retention position for conversation history, workflow run records and unused knowledge base data, because a cleanup job applies that position on its own timetable. Deleted data cannot be recovered by restarting a container, so the decision belongs with the owners of records and audit obligations, not only the platform team.

The job list and default schedules belong to the Dify version you run, so confirm them against that release before you copy a list from a blog or an older guide. Minor releases arrive roughly monthly: 1.16.0 on July 17, 2026, 1.17.0 on August 25, 2026 and 1.17.1 on September 10, 2026, according to the Dify releases page. Check each release's notes for changed tasks, then use the Dify environment variable reference to see which settings control each job. Record the version, the enabled jobs and their schedules in your change log so the next upgrade starts from a known state.

Test each deletion job on a copy of your environment before production. Take a backup first, as the Dify Docker Compose guidance recommends before every upgrade, and measure PostgreSQL size, storage usage and the worker queue before and after a trial run. Ask application owners whether any app depends on data a job may remove, such as old runs they audit or files behind a published knowledge base. Enable one job at a time, and confirm with your team which owner signs off on each retention setting.

Beat jobs to review and when each one fits

This table compares the four Celery Beat jobs named for Dify scheduled maintenance: message and workflow run cleanup, unused dataset cleanup, plugin upgrade checks and the schedule poller. Read each row against your installed version, because the job names, default schedules and flags are not confirmed in the facts available here and must be checked in the Dify docs for that release. The main takeaway is that the cleanup jobs delete data, so they should only be enabled after a tested backup and a retention decision.

Job	What it touches and data risk	When production should enable it and what to confirm
Message and workflow run cleanup	Deletes old conversation messages and workflow run records from PostgreSQL. Deletion is irreversible without a backup, and audit or support history may disappear.	Enable after your retention period is written down and a restore from backup has been tested. Confirm the job name, schedule and retention setting in your release's environment reference.
Unused dataset cleanup	Removes knowledge base datasets and their indexed content that no app references. A mistaken match can remove documents other teams still query.	Enable only after reviewing which datasets are unreferenced and confirming owners agree. Confirm the job's selection rule and whether it also clears vector store entries in your release.
Plugin upgrade checks	Queries plugin sources for newer versions and may record availability. Low data risk, but an automatic upgrade path would change running code.	Enable notifications where useful, and keep installs manual under change control. Confirm whether the job only checks or also applies upgrades, and which sources it contacts.
Schedule poller	Checks scheduled workflow triggers and starts runs at their configured times. Affects workflow execution and may trigger outbound calls to model providers and tools.	Enable wherever scheduled workflows are in use, and verify it runs on the worker that owns the schedule. Confirm how it behaves across multiple worker replicas and during restarts.

Why retention decisions come before enabling cleanup

Write the retention policy before any scheduled cleanup job is switched on. Dify's scheduled maintenance tasks delete records that someone may still need later: conversation history tied to a published app, workflow run records used to investigate a bad answer, and dataset content that a knowledge base depends on. Once a job has run, the rows are gone from the live database. Before enabling anything, decide for each record type how long it must stay, who may read it, and who approves its deletion. Those answers come from your business owners, your compliance lead and any regulator or contract that governs the data, not from the default settings of the job. Confirm the exact retention windows with your compliance lead and with Dify's own configuration reference before you write them into the policy.

Retention and backup are one decision, not two. A cleanup job that runs once is recoverable only if a backup taken before the run exists, is complete, and has been restored successfully in a test environment. A backup that has never been restored tells you little about whether the deleted records can come back. Keep at least one backup generation older than your longest retention exception, and record which backup covers which period. Note also that Dify's documentation describes app exports as excluding usage logs and knowledge base content, so a DSL export does not preserve the evidence that a cleanup job removes. Ask your vendor or implementation partner which tables and storage locations hold each record type, and confirm that your backup scope includes all of them, including object storage and the vector store.

Ask three questions before the first run: which records must survive for audit, support or legal reasons; how long each must survive; and how a deletion is reversed if it was a mistake. Write the answers into the runbook next to the job names, and schedule a first run against a copy of production so the team can measure what would be removed. Treat the policy as a living document, reviewed whenever a new app type, model provider or data source adds records that the schedule did not anticipate.

What goes wrong when cleanup runs without a retention decision

Scheduled cleanup removes data on a timer, so a missing retention decision shows up later as a gap in an investigation, a broken external integration or a slow indexing queue. Each card below names the symptom an operator sees and the check that prevents it before the first scheduled run.

Run history gone during an investigation

A cleanup job that prunes workflow run records on a short window removes the evidence a team needs when a user reports a bad answer a week later. The symptom is an investigation that finds the conversation but no node-level trace for it. Before enabling the job, write down how many days of run history your incident and compliance processes need, then set the retention window to match that decision rather than the default. Confirm the window with your compliance owner and test it on a copy of the database.

Dataset removed while an API caller still used it

An unused dataset cleanup judges usage from inside Dify, so a knowledge base queried only through the service API can look idle and be removed. The symptom is an external application returning empty retrieval results or errors after a scheduled run, with no change on the Dify side that anyone recognizes. Before enabling the job, list every knowledge base that external systems query, confirm how each one is called, and exclude those datasets or move the retention decision to the owning team.

Large cleanup blocks indexing workers

A bulk deletion run during business hours can fill the Celery queues that also handle document indexing, so new uploads sit in a pending state while cleanup finishes. The symptom is a dataset that stays in indexing for far longer than normal and users who report that uploaded files never become searchable. Schedule large cleanup batches for a low-traffic window, watch the queue depth while they run, and confirm that the dataset queue monitor alerts are working before the first run.

Upgrade changes job behavior silently

Dify ships minor releases roughly monthly, and a release can change which Celery Beat jobs run, how often they run or what they consider eligible for deletion. The symptom is a cleanup pattern that differs after an upgrade, such as more data removed than the previous month, with no operator action behind it. Read the release notes for each target version, compare the scheduled job list in the new image with your enabled set, and check the .env.example file for new variables before you upgrade.

How to enable and verify the beat jobs in Compose

This sequence turns the scheduled maintenance jobs from a list of options into a tested production setting, with each job's effect measured in staging before it runs against production data. The result is a documented schedule, a record of the first runs, and a before and after view of database and storage size.

Confirm the installed version and read its release notes
Record the Dify version currently running, then check the GitHub releases page for that version and any minor releases after it. Dify minor releases have arrived roughly monthly, so the version you run may already differ from the one your runbook describes. Read the release notes for changes to scheduled tasks, cleanup behavior and environment variables, and confirm the target version's upgrade guide if an upgrade is part of the change. The step is done when you have a written version number and a list of notes that affect the jobs you plan to enable.
Write the retention policy and agree it with the data owner
Decide how long conversation messages, workflow run records and unused dataset content should be kept, and who in the business approves that period. The data owner, not the platform team alone, should sign off because cleanup jobs delete data that cannot be recovered except from backup. Write down which jobs are enabled, which are left off, and the reason for each. The step is done when the signed policy names every enabled job and its retention setting, so later changes have a reference point.
Take a backup and prove it restores
Take a consistent backup of the PostgreSQL database, the file storage and the vector data, along with the environment files that hold the configuration. Dify's Compose docs advise a backup before every upgrade, and the same discipline applies before deletion jobs run. Restore the backup into a separate environment and confirm that the application starts and the expected records are present. The step is done when a restore has been tested, not only when a backup file exists.
Enable the chosen jobs in a staging instance first
In a staging copy of the deployment, set the scheduled task configuration to match the signed policy and restart the services so the beat scheduler reads the change. Use the container names and the way beat is started from the Docker Compose docs for the target version, since these can differ between releases. Confirm in the Compose service list that the beat process is running alongside the worker. The step is done when the staging stack is up with only the approved jobs enabled.
Run each job once and watch its first scheduled run
Trigger or wait for each enabled job, then follow the worker and beat logs for the task name, start time, completion and any error. Run one job at a time so that a failure can be traced to a single cause. Check that the job logs completion and does not repeat on every cycle with the same errors. The step is done when each enabled job has a recorded successful run in the logs and no unexplained exceptions remain.
Compare row counts and storage use before and after
Before each job runs in staging, record row counts for the affected tables and the size of the database and file storage. Repeat the measurement after the run and compare the two sets of figures against the retention policy. Confirm that records outside the retention window were removed and that records inside it remain. Check that the application still answers test conversations and workflow runs. The step is done when the before and after figures are saved next to the policy, so the effect is measured rather than assumed.
Promote the verified schedule to production
Apply the same environment settings and beat configuration to production during an agreed change window, with the backup from the earlier step confirmed as current. Repeat the log check for the first scheduled run after the change, and compare storage figures at the next review. Keep the staging results with the change record so the production schedule can be traced to tested evidence. The step is done when production runs the approved jobs, the first runs are logged cleanly, and the change record links to the staging results.

Job settings by environment: development, staging and production

Development is where you can run cleanup often and with short retention windows so growth problems appear quickly. The dataset is small and disposable, which makes it easy to see what each job removes and to repair the environment if a job misbehaves. Keep this environment isolated so operators are comfortable running destructive settings.

Shorten retention deliberately: Lower the message and workflow run retention values to force deletions during test sessions, and record the values you chose so results can be compared later.
Seed known data: Create a fixed set of apps, datasets and conversations, then note row counts in PostgreSQL before and after each job cycle.
Log every run: Capture Celery Beat and worker logs for each scheduled job, including start time, end time, rows affected and any exception.
Practice restores: Restore the database and file storage after a cleanup run so you learn whether deleted records leave broken references in the console.
Keep credentials separate: Never point development at production storage, shared Redis or a shared vector index.

Confirm each retention and schedule setting in the environment variable reference before you change it, and note which values you tested so they can be carried forward.

Keeping cleanup jobs from destroying evidence and secrets

Treat every change to the beat schedule as a production configuration change. Whoever can edit the environment for the beat and worker containers, or the .env file they read, can change which jobs run and how often. Limit that write access to the people who own the deployment, keep the file under version control, and require a second reviewer for any change that enables a deletion job. The SECRET_KEY setting matters here too. The environment reference says it signs session cookies and JWTs and encrypts OAuth credentials, and that changing it after deployment logs out all users. Cleanup jobs do not rotate it, but you should confirm which stored values depend on it before a key change or a restore, and test that path in staging first.

A deleted message, workflow run or dataset record is gone unless a backup holds it. That makes retention settings a records decision as much as a storage decision. Before you enable a cleanup job, agree with your security and compliance owners how long conversation and run records must be kept, and confirm that a restorable backup covers the window you are about to delete. Check where workspace encryption keys live. The environment reference says they can be stored locally or in Azure Key Vault. If they are local, a backup of the database without the matching keys may be unreadable, and a backup of the keys without the database protects nothing. Restore a copy into a test environment and confirm that you can read the data back before you trust the schedule.

The plugin upgrade check job touches plugin state, so review it alongside your plugin signing and trust policy before it runs in production. Confirm which plugins it may check, whether it can change installed versions, and where it fetches information from. Any network call a job makes still passes through the outbound proxy settings. The environment reference describes SSRF_PROXY as routing outbound requests through a proxy that blocks internal and private IP ranges, so a job that needs an internal endpoint may fail in a way that looks like a bug. Verify the job's network path from the worker container, not from your workstation, and read the environment variable reference for the exact proxy keys your version uses.

Limits to confirm for your Dify version and deployment

Scheduled maintenance jobs in Dify are tied to the release you run and to how your database and storage are deployed. The cards below list the points to verify against your installed version before you rely on any cleanup or poller schedule in production.

Exact job list and schedules

The Celery Beat schedule is defined in the code of the release you install, so job names, intervals and retention windows can change between versions. Minor releases arrive roughly monthly, so the schedule you audited last quarter may not match today. Read the schedule definition in the source for your exact tag and record it in your change log. Check the Dify releases page for the version history before each upgrade.

Disable or only tune

Confirm whether each job can be switched off through configuration or only adjusted through its interval and retention settings. Some cleanup jobs may be required for storage hygiene, and disabling them can leave orphaned rows and files that grow over time. Ask the vendor or check the environment reference for documented switches. Do not assume a job is optional because it is not in your .env file.

External PostgreSQL and object storage

The default Compose stack runs PostgreSQL and file storage in containers, while production setups often use external PostgreSQL and S3-compatible storage. Confirm that cleanup jobs reach the same database and bucket your application uses, and that network paths, credentials and permissions allow deletes. Deletions of storage objects may behave differently on object stores with versioning or lifecycle rules, so test on a copy first.

Schedule poller and workflow triggers

Confirm how the schedule poller relates to scheduled workflow triggers. Determine whether the poller runs as part of the beat schedule, which worker queue it uses, and how often it checks for due triggers. A stopped or lagging poller can make scheduled workflows appear to fail without any error in the application. Verify the trigger timing in a non-production instance before you depend on it.

Enterprise on Kubernetes

Dify Enterprise on Kubernetes uses a different deployment layout from the Docker Compose stack. Confirm whether the same beat jobs run there, whether beat is a single replica or managed by the chart, and how leader election or duplicate scheduling is prevented if you scale. Ask the vendor for the Enterprise component list that matches your version, since the resources checklist describes infrastructure requirements rather than job behavior.

Questions buyers ask before enabling Dify maintenance jobs

How much effort does setting up Dify cleanup jobs take in-house?

The effort depends on how many maintenance jobs you enable and how much you must prove about what they delete. The one-time work includes reading the scheduled tasks in the release you run, agreeing retention windows with data owners, testing in staging and writing a runbook. The recurring work is reviewing the job list after each upgrade, since minor releases arrive roughly monthly. An in-house team pays for that time through its own staff. A partner works from a fixed scope agreed in writing, so you pay for the defined jobs, tests and handover rather than open-ended hours. Ask the team to estimate the one-time and recurring effort separately, and to name who makes each retention decision.

What does a Dify maintenance rollout need from our team?

A rollout needs three things from your side: data owners who can approve retention periods for conversations, workflow runs and unused datasets, a backup target that you have restored from before any deletion is enabled, and a staging instance on the same Dify version as production. You also need an operator who can read Celery logs, edit the environment file and restart containers during a change window. Legal or compliance staff should confirm retention requirements in writing before the schedule goes live. If any of these owners are missing, the rollout will stall at the first retention decision, so confirm them before scoping the work.

Can we test Dify cleanup jobs without deleting production data?

Yes, if you test on a restored copy rather than on production. Restore a recent backup into a staging instance running the same Dify version, then set shorter retention in staging so the jobs have something to remove. Record row counts and storage usage before and after each job, and check that conversations and datasets you expect to keep are still present. Staging only proves behavior if its data resembles production, so note any differences in volume, file types or indexing. Keep production retention unchanged until the staging results are reviewed and signed off in writing.

How long does a first safe rollout of Dify maintenance jobs take?

No duration is fixed for this work, and the length depends on scope. The main drivers are how many jobs you enable, how large the PostgreSQL database and file storage are, whether your backup and restore process is already tested, whether a staging instance exists, and how many approvals retention decisions need. A rollout that enables only one job on a small instance with ready staging is far shorter than one that covers every job across several environments. Ask for milestones in writing: scope confirmation, staging test, production enablement, and a monitoring window after go-live.

What should a quote for Dify maintenance job setup include?

A quote should follow from a written scope and list what is included and excluded. Ask for the Dify version covered, the specific jobs in and out of scope, who signs each retention decision, how backups are verified before deletions run, the staging test plan and pass criteria, the rollback steps, and the handover runbook. Include the upgrade checks you expect to run after each release, the monitoring period after go-live, and the support terms that follow. Separate one-time setup from recurring retainer work, so you can see what continues after the first rollout. DeployOpen does not publish prices, so the quote is the source of cost detail.

Should we run Dify maintenance tasks ourselves or hire a partner?

Run them yourself if your team already operates Celery, PostgreSQL and Redis, can staff a review after each release, and has data owners ready to make retention decisions. Consider a partner if those skills or people are thin, or if you need documented evidence of how deletions were tested and approved. DeployOpen plans and implements self-hosted platforms in your own infrastructure, and its engagements are usually fixed-scope with the scope agreed in writing. Ongoing upgrades and support are available as retainers. Compare both options on the same written scope so the trade-off is visible.

What happens to these jobs after a Dify upgrade?

Each upgrade needs a fresh check of the job list, because the schedule and the variables that control it can change between releases. Follow the upgrade guide for the target version on the GitHub Releases page, take a backup before every upgrade, and compare your environment file with the current .env.example for new variables. Then confirm which Celery Beat jobs are enabled, that their retention settings still match your decisions, and that the first run after the upgrade behaves as it did in staging. Treat this check as recurring work and assign an owner for it.

How DeployOpen scopes and runs Dify maintenance work

DeployOpen plans, implements, documents and supports self-hosted open-source platforms inside your own infrastructure. For Dify scheduled maintenance tasks, we start from a written scope that names which Celery Beat jobs run in your environment, how each one is verified, and who owns the schedule after handover.

How we scope

Written scope lists each Celery Beat job you intend to enable, with the retention or cleanup rule it applies.
We confirm your Dify version, deployment layout and data retention policy before any schedule change is proposed.
Engagements are fixed-scope, so deliverables and timelines for the maintenance work are agreed in writing before work starts.

How we build

We set job schedules and retention settings in a staging copy first, then record the exact configuration for production.
Each enabled job is tested against a restored backup so deletions are checked before they reach live data.
We verify worker logs, database growth and storage usage after each change, and record the results in the handover notes.

How we hand over

You receive a runbook listing every enabled scheduled task, its purpose, its schedule and the settings that control it.
The runbook includes a rollback procedure for each job and the checks your operators run after every Dify upgrade.
Your team is walked through the monitoring signals that show a maintenance job has stopped running or is falling behind.

How we support

Ongoing support, upgrade checks and continuous testing are available as a retainer, scoped separately from the initial work.
When a new Dify minor release arrives, we review its release notes for changes to scheduled jobs before you upgrade.
Quotes follow from the written scope, so ask for a breakdown of one-time setup effort and any recurring retainer work.