Why Dify Celery workers and queue backlog monitoring decide how fast your knowledge bases and workflows respond
When documents index slowly or workflow runs sit in a queue, the cause is usually background work that is not keeping up. Celery workers execute those background jobs, and queue monitoring plus worker sizing decide whether they keep pace with your load. In a self-hosted Dify, the Docker Compose layout starts a separate worker container alongside the api container, and it is the one that picks up indexing and other asynchronous tasks. Teams usually need to look at this layer when a bulk document upload takes far longer than expected, when several knowledge bases are being indexed at once, or when workflow runs that depend on background processing start to lag behind user requests.
The researched facts for this topic do not name the Celery queues Dify uses, state a backlog threshold, or describe the exact worker concurrency settings for each release. Confirm those details against the version you run, by checking the Dify Compose files and environment reference for that release and by reading the release notes for any change to queue names. Before you change worker counts, record what the backlog looks like at normal and peak load so you can tell whether a change helped. The Dify environment variable reference is the first place to check which worker-related settings your version exposes.
Scaling workers also has side effects on the rest of the stack. Each worker consumes CPU, memory and connections to PostgreSQL and Redis, and embedding and indexing jobs call external model endpoints, so more workers can move the bottleneck rather than remove it. Plan the worker count together with database connection limits, model provider rate limits and the sizing of the vector store. Shutdown behavior matters too: during a deploy, a worker that is stopped in the middle of a job can leave that job incomplete, so the way you restart workers is part of the same decision.
Scaling workers for indexing load versus workflow load
Indexing work starts when a document is uploaded or a knowledge base is synced. The worker parses the file, splits it into chunks, sends each chunk to the embedding model, and writes vectors to the vector store and metadata to PostgreSQL. One large file produces many embedding calls, so the load arrives in bursts and depends heavily on outbound calls and vector writes, with moderate CPU for parsing.
- Queue pressure: A single bulk upload can enqueue many indexing jobs at once, which delays any workflow job waiting in the same pool.
- CPU: Parsing and chunking are CPU work, most noticeable with large PDFs and office files.
- Memory: Large files are held in memory during parsing, so peak memory grows with file size and the number of parallel jobs. Confirm the peak with a test upload of your largest expected file.
- Outbound throughput: Embedding speed is often capped by the provider's rate limits or the capacity of a local model server, so extra workers can hit those limits sooner.
During a representative bulk import, check queue depth and embedding endpoint error rates before you change the worker count.
Workflow and agent runs are dispatched to workers as each step executes. They are latency sensitive, because a user waiting on a chat response notices queueing far sooner than they would notice a slow indexing job. Most of the time a run spends in a worker is waiting, not computing: the worker holds a slot while a model call or HTTP request returns.
- Long waits on I/O: Model and HTTP calls keep a worker slot occupied while idle, so concurrency limits usually matter more than CPU count.
- Sandbox and proxy dependence: Code and tool execution depend on the code sandbox and the SSRF proxy. A slow or blocked call holds its slot until it times out, so confirm the timeout behavior in your version.
- Memory across steps: Multi-step agents keep context and tool results in memory, so memory grows with run length and with the number of parallel runs.
- Fan-out: Iteration nodes and parallel branches multiply the work created by one trigger.
- Bursty triggers: Webhooks and scheduled runs can arrive together, which creates the same kind of spike as an upload.
Measure queue wait time for interactive runs separately from batch runs, since they usually need different limits.
Most single-team deployments run indexing and workflows on one worker pool, because the Compose worker service handles the queues unless you split it. The risk is head-of-line blocking, where a bulk import delays an interactive run that arrived later. Separating the workloads is the usual remedy, but the method depends on your Dify version, so confirm queue names and worker options in the release notes and environment reference before you design around them.
- Shared pool: One worker service with mixed traffic is the simplest setup, but latency becomes unpredictable during imports.
- Scheduled maintenance: Cleanup and maintenance jobs add background load at fixed times, so schedule them away from peak use.
- Shared dependencies: Redis and PostgreSQL are shared, so more worker processes also mean more broker and database load.
- Host contention: A host sized to the Compose floor of 2 CPU cores and 4 GB RAM runs the API, web, database and workers together, so mixed load will expose contention quickly.
- Separation options: Dedicated worker hosts for indexing limit the impact of a bulk import on interactive use. Confirm which separation options your version supports.
Run a mixed-load test that starts an import while users chat, and record which workload slows down.
The choice is between more worker containers, which add processes with their own memory and CPU share on whichever host runs them, and higher concurrency inside each worker, which runs more tasks in one process. Concurrency is cheap to change but shares one memory space, while containers isolate failures and spread load across hosts at a higher resource cost. The Docker Compose minimum of 2 CPU cores and 4 GB RAM is a floor for starting the stack, not a sizing guide. The Enterprise resources checklist gives one production example of six worker nodes with 8 CPUs and 32 GB RAM each, which describes that deployment only and is not a formula for your workload.
- Raise concurrency when CPU and memory have headroom and queued tasks are waiting on model or HTTP responses.
- Add containers when one host is saturated or you need failure isolation between indexing and interactive work.
- Watch memory first: A worker restarted for memory loses its in-flight tasks, so headroom matters more than peak throughput.
- Count connections: Each added process opens its own connections to PostgreSQL and Redis.
- Change one variable at a time: Measure queue depth and task duration after each change.
Before you change production values, confirm the concurrency settings and scaling options for your version in the Dify environment variables reference.
Worker and queue settings and when each fits
This table compares the main worker, queue and shutdown choices for a self-hosted Dify and states when each one fits and what it costs in return. Use it to decide what to change first, then confirm exact variable names and defaults in the environment variable reference for your version before editing anything.
| Setting | When it fits | Trade-off | Confirm before changing |
|---|---|---|---|
| Number of worker containers | Indexing batches or workflow runs wait for a free worker, and the backlog clears slowly after uploads or bulk imports | More containers raise CPU, memory and database connections on the same host, and can slow the api and web services if the host is undersized | Host CPU and RAM headroom, and the worker service name in your Compose file |
| Worker concurrency per container | One container has spare CPU but jobs still queue, so more parallel tasks per container may help | Higher concurrency multiplies memory use and parallel calls to model providers and the vector store, which can trigger provider rate limits | The concurrency setting for your version and its default value |
| Dedicated Redis for Dify | Dify shares a Redis instance with other applications, or queue traffic must be isolated from cache traffic for monitoring and recovery | A dedicated instance adds one more service to back up, patch and monitor, and needs its own memory sizing | Whether Redis is external or bundled, and the Redis connection variables in use |
| Shared Redis with key prefixes | A small deployment has spare capacity and one team operates both the Dify and other workloads | Shared instances mean a memory spike or restart in one application affects the Dify queue, and key naming must avoid collisions | Redis memory limits, eviction policy and the Celery broker database index |
| Dataset queue monitor enabled | You need to know when indexing jobs stall before users report missing knowledge base content | Monitoring adds a periodic check and an alert path that must be tested, and an unmonitored queue fails silently | The monitor variable name, its check interval and whether it is on by default |
| Alert recipients | A named on-call group or shared operations mailbox owns indexing health | Too many recipients create noise, and a single personal address leaves alerts unread during leave or role changes | Mail delivery is working for alert messages, see the email delivery configuration topic |
| Warm shutdown grace period | Deploys restart workers while long indexing or workflow tasks are running, and you want running tasks to finish | A longer grace period slows deploys and can hold up rollbacks, while a short one can interrupt tasks and leave items to retry | The stop grace period set in Compose or Kubernetes and the worker shutdown behavior in your version |
Setting up worker scaling and queue monitoring in order
This sequence confirms the platform can run workers, records what normal queue depth looks like, and then changes one thing at a time so you can see whether scaling and deploys behave as expected. Each step ends with a check you can run on the host.
Confirm the Docker Compose version
The platform engineer confirms that the host runs Docker Compose 2.24.0 or later, the minimum that the self-hosting docs give for the Compose deployment, before changing any worker settings. Run
docker compose versionon the host that runs the stack and compare the output with that minimum. Then rundocker compose psto confirm that the api, worker, web, plugin daemon, PostgreSQL, Redis and the other core services are running. The step is done when the version is 2.24.0 or higher and every core service reports running. If the version is older, upgrade Compose first so that every later change is tested on the version you will run in production.Verify the worker container connects to Redis
The operator checks that the worker container started and reached the Redis broker. Run
docker compose ps workerto confirm it is running, then rundocker compose logs --tail 100 workerand look for a clean startup and a successful broker connection with no repeated connection errors. Rundocker compose exec redis redis-cli pingand confirm the reply is PONG. Confirm the Redis credentials in your.envfile match the Redis service. The step is done when the worker is running, its logs show a stable connection, and Redis answers the ping. Do not scale anything until this passes.Record a baseline queue depth under normal load
During a normal working period, the operator records how many tasks wait in the Celery queues that handle dataset indexing and workflow execution. Confirm the queue names in the worker logs or in the source for your Dify version, then read their lengths with
docker compose exec redis redis-cli llenfollowed by the queue name. Take readings at intervals you choose and note what was happening at each one, such as a bulk document upload or a scheduled workflow. Also time one representative knowledge base indexing job. The step is done when you have a table of queue depths, job activity and completion times that you can compare against later.Enable the dataset queue monitor and email delivery
The platform engineer sets the variables for the dataset queue monitor and the mail provider in
.env, using the names listed in the environment variable reference for your version, and confirms each name against.env.example. Apply the change withdocker compose up -dso the api and worker read the new values. Send a test alert by temporarily lowering the monitor threshold, or use the test path the monitor documents if one exists, and confirm the email arrives. Test mail separately from the monitor too, since invitations and password resets use the same provider. The step is done when a test alert reaches the intended inbox and the mail provider logs show an accepted message.Scale worker containers and re-measure
The operator runs more worker containers with
docker compose up -d --scale worker=N, choosing N from the CPU and memory the host has free after the core services take their share. Before scaling, confirm the worker service has no fixed host port or fixed container name, since either blocks multiple replicas. Repeat the baseline load from the earlier step and compare how quickly the queue drains, the indexing completion time and any new errors. Watch PostgreSQL connection counts and Redis memory while you do this, because more workers add load to both. The step is done when the same load drains faster or with fewer waiting tasks and no new errors or connection exhaustion appear in the logs.Test a deploy with warm shutdown
The operator tests what happens to running work during a deploy. Start an indexing job or a long workflow, then run the same stop and start sequence you would use for an upgrade, such as
docker compose up -dafter pulling new images. Watchdocker compose logs -f workerto see whether the worker finishes its current tasks before it exits. Confirm the stop grace period for the worker service in your Compose file, since Docker sends a forced kill after that period. Afterward, check the document status in the knowledge base and the workflow run history. The step is done when the in-flight job either completed or shows as retried, and no index is left partly written.
What happens to in-flight tasks during a deploy
A worker container that is stopped abruptly can leave indexing or workflow tasks half finished, so the goal during a deploy is to let running Celery tasks complete before the container is replaced. Docker sends a stop signal and then a kill after a grace period. Celery treats a warm shutdown as a request to stop taking new work while the tasks already executing finish. The grace period your Compose setup allows (the stop_grace_period value on the worker service, or the docker compose stop timeout) has to be long enough for the largest indexing job you normally run. Confirm both values in your own file and compare them with how long your longest document batch takes in a test environment. The Dify docs do not give a number for this, so the value is yours to set and test.
The risk of a hard stop is concrete. A document can be partly chunked, with some segments written to the vector store and others missing, and the dataset may show a status that does not match its content. Workflow runs that were midway through a node can also end in an unclear state. Recovery is usually a re-index or a retry, which costs model and embedding calls again. To check behavior in the version you run, trigger a large indexing job, issue a docker compose stop worker during processing, and then review the dataset's document status and the worker logs after restart. Repeat the test after any change to the worker's command or stop timeout. Make sure the test is run against the target version, because worker startup and shutdown handling can change between releases.
Treat every deploy as an upgrade with a fixed order. Follow the upgrade guide for the target version on the GitHub Releases page, take a backup before you pull new images, and check .env.example for variables the new version expects. Drain the queue where you can by pausing new dataset uploads and waiting for the monitor to show no active work before you restart workers. Plan the window with the release cadence in mind, since minor releases have arrived roughly monthly. Ask your operations team to record the stop timeout, the backup identifier and the queue depth at the moment of the stop, so a failed deploy can be traced to a specific state.
Security and operational risks in the worker tier
The worker tier sits between the API, Redis, the sandbox and your mail server, so a weakness in any one of them reaches indexing and workflow jobs. The cards below cover the access, secret, network and data exposure points to review before you scale workers or turn on backlog alerts.
Redis exposure and service authentication
Celery uses Redis as the broker that carries job messages between the api and the worker containers. Anyone who can reach that Redis port can read, reorder or inject jobs, so keep it on the internal Compose network only and confirm it requires a password. Check that no port mapping publishes Redis to the host or a load balancer, and review the firewall rules on any host that runs workers.
Token-based authentication between services
Dify 1.16.1 added token-based API authentication between its services, which narrows what a compromised container can call. Confirm that your deployed version includes this change before you assume it applies, and check the release notes for any configuration it requires. Do not expose internal service ports to the network while relying on this control.
Sandbox and outbound request controls
Workers run indexing, tool calls and sandboxed code, and their outbound requests pass through the SSRF proxy and the agent sandbox network isolation. A blocked private address or a proxy timeout can fail a job that looked healthy in the queue. Review the proxy allowlist and timeouts together with worker concurrency so that retries do not pile up behind blocked calls.
Secrets in the environment file
The .env file holds SECRET_KEY, which signs session cookies and JWTs and encrypts OAuth credentials. Changing it after deployment logs out all users, so it must be generated once, stored in your secrets manager and backed up with the rest of the configuration. Restrict file permissions on .env, keep it out of version control and confirm that every worker container reads the same value.
Alert emails carrying operational data
Backlog alerts sent by the dataset queue monitor can include queue names, counts and timing, which tells an outsider how busy your indexing pipeline is. Route alerts to a named distribution list, avoid forwarding to personal mailboxes and confirm the mail provider's retention policy. Treat the alert body as operational data that needs the same handling as your monitoring output.
Worker logs holding document content
Worker logs can record failed indexing jobs, and error messages may include fragments of the documents being processed. Limit who can read container logs, set a retention period that matches your data policy and check whether log shipping sends this content to a third-party service. Ask your Dify team to confirm which log levels are enabled in production.
Questions operators ask before scaling Dify workers
What does it cost to run more Dify workers?
Most of the cost is the extra compute the worker containers consume, plus the database, Redis and vector store capacity that the extra indexing and workflow load draws on. The Dify Docker Compose deployment starts a worker service alongside the api, web and other core services, and the minimum for the whole stack is at least 2 CPU cores and 4 GB of RAM according to the Dify self-hosting docs. Recurring costs grow with how many workers you run, how large your documents are and how often knowledge bases re-index. Ask your team to size workers from measured queue depth and resource use, not from a guess, and to price the host capacity, storage and monitoring as recurring line items.
How long does a Dify worker scaling and queue monitoring build take?
Duration depends on how many hosts run workers, whether Redis and PostgreSQL are external or inside Compose, how mail alerts must route, and whether the team needs a documented warm shutdown procedure for deploys. A single Compose host with one alert recipient is a much smaller job than several hosts with shared Redis, SMTP relay approval and a tested deploy runbook. DeployOpen does not publish timelines; the written scope sets the schedule, so ask for the dependencies and milestones to be listed before work starts.
What does our team need to provide for worker scaling work?
Plan for Docker host access with permission to edit the Compose file and .env, the Redis connection details and the database host, and the SMTP or mail provider settings used for queue alerts. You will also need recent load data: indexing job counts, workflow run volumes, queue backlog history and the resource graphs for the current hosts. Confirm the Dify version you run before changes, because upgrade notes in the Dify releases page can change worker behavior, and keep a backup before any change as the Docker Compose docs recommend.
Can an internal platform team scale Dify workers and set up alerts alone?
Yes, a team with Docker Compose, Redis and mail administration experience can do the core work using the Dify docs and its environment variable reference. The harder parts are judging how many workers the load needs, confirming that a deploy does not cut off in-flight indexing jobs, and making alerts reach an on-call owner. Those tasks need test runs under realistic load, which takes staff time. Decide early whether the team has that capacity and who owns the result after handover.
What goes wrong when Dify workers are scaled without checking the queue?
The usual failures are indexing jobs that stall while workers compete for the same Redis queue, workflow runs that wait longer than users expect, and deploys that stop workers in the middle of a job. Adding containers also raises load on PostgreSQL connections and the vector store, so the bottleneck can move rather than disappear. Watch backlog size, job age, failed task counts and database connection use together before and after each change.
When should we bring in a partner for Dify worker operations?
Bring in a partner when the workload is business-critical, when several environments share Redis or the vector store, or when your team lacks time to run load tests and deploy drills. A partner is also useful when you need a written, fixed-scope change with acceptance checks. DeployOpen plans, implements, documents and supports self-hosted platforms inside your own infrastructure, and most engagements are fixed-scope with scope agreed in writing before work starts. Ongoing monitoring and upgrades are available as retainers.
Does the worker setup require secrets or access to our vector store?
Worker containers need the same SECRET_KEY and connection settings as the api service, so changing them incorrectly can break logins or background tasks. The Dify environment reference says changing SECRET_KEY after deployment logs out all users, so do not rotate it as part of a scaling change. Limit who can read .env, keep Redis reachable only from the Dify network, and confirm which credentials the mail alert path uses before you grant access.
How DeployOpen scopes and supports Dify worker operations
DeployOpen plans and operates Dify worker capacity inside your own infrastructure, starting from the indexing and workflow load your team actually expects. We set up the queue monitoring, alerting and shutdown behavior as part of a written scope, then document and hand them over so your operators can run them.
How we scope
- Written scope records which worker queues, workloads and deploy paths are in and out of the engagement.
- We inventory your current Compose or deployment layout, Redis and PostgreSQL placement, and where indexing and workflow runs happen.
- Scope names the alert recipients, mail delivery dependency and the target environments such as staging and production.
How we build
- We size worker containers against your indexing and workflow load, then test scaling changes in a non-production copy first.
- We configure the dataset queue monitor and its email alerts, and verify that a simulated backlog triggers the expected notification.
- We check warm shutdown behavior during a rolling deploy so in-flight indexing tasks finish or requeue as your team expects.
How we hand over
- Deliverables include a written runbook covering scaling steps, queue checks, alert thresholds and rollback for worker changes.
- We walk your operators through the monitor output, the alert emails and the commands used to inspect and drain queues.
- Documentation records the Dify version tested, configuration keys changed and the verification steps used for each item.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, so worker changes can follow each Dify release.
- Before a minor upgrade, we review release notes and environment variable changes that could affect Celery workers or queues.
- Ask us to confirm the retainer scope, response expectations and which environments are covered before you sign.
