Dify performance tuning: where slow workflows and API calls come from
When an app slows down under load, the first job is to find which component is the constraint, not to change settings across the board. A self-hosted Dify is several cooperating services. The Docker Compose deployment starts the api, worker, web and plugin daemon, along with PostgreSQL, Redis, a vector store (Weaviate by default), Nginx, a code sandbox and an SSRF proxy. A slow workflow run can be waiting on any of them: a database connection pool that is exhausted, a Redis call that times out, a worker that has too few slots for indexing or run queues, a sandbox or proxy pool that is at its limit, or an HTTP node waiting on an external system until its timeout ceiling. Each of these looks like the same symptom from the user's side, which is why tuning without measurement often moves the problem rather than removing it.
Teams usually need this work after a pilot moves to real traffic, when a workflow that ran in seconds during testing begins to queue, when API callers see timeouts during peak hours, or when a new knowledge base indexing job delays chat responses. Before changing any value, confirm the version you run and the Compose file you deployed, because the service names and defaults differ between releases. The Dify releases page shows that minor versions arrive roughly monthly, so a setting you tuned last quarter may have moved. Collect logs and resource metrics for each container during a slow period, and record which service is saturated before deciding anything.
This page walks through each candidate constraint in turn: database connection pools, Redis timeouts, worker counts, sandbox and SSRF proxy pool limits, and HTTP node timeout ceilings. The sibling pages on Celery workers, outbound request failures and database cleanup cover the operational detail behind several of these, so this page focuses on how to tell them apart. Ask your team to agree on a baseline for response times and queue depth before testing, and to change one variable at a time so that each result can be attributed.
Which tuning knob fits which symptom
This table pairs each main tuning lever with the symptom it addresses, the risk of raising it, and the first check to run. Read it from the symptom column inward, and treat it as a starting point because exact variable names and defaults change between releases.
| Lever | Symptom it addresses | Risk of raising it | Check first |
|---|---|---|---|
| Database connection pool | Requests queue or fail with connection errors while PostgreSQL CPU and memory look healthy | More connections multiply load on PostgreSQL and can push it into memory pressure or lock contention | Count active connections against the PostgreSQL connection limit and look for long-running queries or idle transactions |
| Redis timeouts | Workflow streaming, Celery task dispatch or cache reads stall intermittently with timeout errors in api or worker logs | Longer timeouts hide a slow or saturated Redis and delay failure detection for users | Redis latency, memory usage and eviction counters, plus network path between containers or hosts |
| Celery worker count and concurrency | Indexing jobs or workflow runs wait in the queue while api latency is normal | More concurrent tasks increase CPU, memory and database connections, and can starve the api container | Queue depth over time, worker CPU and memory, and whether the backlog is caused by a few large document jobs |
| Sandbox and SSRF proxy pool limits | Code nodes or tool calls fail under parallel load, or return errors only when several workflows run together | Larger pools raise memory use in the sandbox and proxy and can widen the outbound footprint | Sandbox and proxy container logs during the failing window, and the number of concurrent code or HTTP executions |
| HTTP node timeout ceiling | External API calls in workflows fail after a fixed period even though the upstream service eventually responds | A higher ceiling keeps user sessions and worker slots occupied longer and can hide a broken upstream | Upstream response time from inside the Dify network, and whether the failure is a timeout or a proxy block |
| Model server response time | Long model calls are misread as Dify slowness when the delay is in the provider or inference server | Raising Dify timeouts without fixing the model endpoint only postpones the same failure | Time to first token and total latency measured directly against the model endpoint, outside Dify |
How to isolate the bottleneck before changing settings
This sequence turns a vague complaint that a Dify app is slow into a measured location for the delay, so each later tuning change targets a confirmed constraint rather than a guess.
Reproduce the slow run with a timed workflow
Pick one workflow or API call that users report as slow, and run it several times from the same client, with the same inputs, at a quiet time and again under normal load. Record the end-to-end time, the time to first token where streaming applies, and the time spent in each node using the run history in the Dify console. Save these numbers as the baseline in a shared log with the Dify version, the .env values you have changed, and the host sizing. The step is done when you can repeat the run and get times within a range you have written down, and when you can say which node accounts for most of the time.
Read api and worker logs for timeouts
Using the container logs for the api and worker services in your Docker Compose deployment, search the timed run window for timeout errors, connection errors, and tracebacks. Note the exact message, the component that raised it, and whether it appears on every run or only under load. Match each message to a layer: database, Redis, a worker task, the sandbox, or an outbound HTTP call. The step is done when every error from the baseline window is either explained or listed as unknown, and you have a short list of candidate layers to check next.
Check PostgreSQL connection saturation
Ask the database team or the person who runs PostgreSQL to compare the number of active connections against the configured maximum during the slow window, and to look for sessions waiting on locks or long-running queries. Confirm the connection pool settings that the api and worker use in your environment before reading the counts, because the limit that matters is the one the application actually opens. The step stops here with a finding of either saturation, meaning connections are at or near the limit while requests queue, or no saturation, meaning the pool is not the constraint. If saturated, keep the database result and move on only after noting what the counts looked like.
Check Redis latency and timeouts
Measure Redis response time during the slow window using your Redis monitoring or the server's latency tools, and check for timeout errors in the api and worker logs that name Redis. Redis is used for caching and task brokering in Dify, so slow responses delay many paths at once. The step is done when you know whether Redis commands are returning quickly during the slow run, and whether timeouts coincide with the timed runs from the baseline. If Redis is healthy, record that and continue, since a clean result here rules out a common shared cause.
Check worker queue depth
Inspect the Celery worker logs and the queue backlog for the worker service during the timed run. A growing backlog while the workflow is waiting for a task points to too few worker processes or containers for the load, whereas an empty queue with a slow node points elsewhere. Confirm how many worker containers and concurrency settings your deployment runs before comparing numbers, and check whether indexing jobs or other background tasks are competing for the same workers. Stop when you can state whether tasks waited in the queue and for how long, using the timestamps in the logs.
Check sandbox and SSRF proxy waits
For workflows that run code or make outbound HTTP calls, check the logs of the code sandbox and the SSRF proxy for waits, rejected requests, and queue or pool exhaustion messages. Compare the timing of these entries with the node timings from the baseline. Also confirm whether the HTTP node timeout in the workflow is set to a value that matches the slow external service, without raising it to hide a fault. Stop when each slow node has either a sandbox or proxy wait attached to it, or has been cleared of these waits and remains slow for another reason.
Change one variable at a time against the baseline
Once a layer is named, change a single setting, such as one pool limit, one worker count, or one timeout, then repeat the timed run from the baseline under the same conditions. Record the new numbers beside the old ones with the exact setting and value. Revert the change if it does not improve the measured node or if it shifts the bottleneck to another layer without a clear gain. The sequence is complete when the slow run meets the target you agreed with the business owner, or when the evidence shows the constraint sits outside Dify settings, such as the model provider.
Why more workers do not always make Dify faster
Adding worker containers raises the number of tasks Dify can run in parallel, but it does not add capacity to the parts those tasks share. Every worker opens its own connections to the metadata database, reads and writes through Redis, and calls the model providers or self-hosted inference servers that the workflows depend on. If the database connection pool is already close to its limit, a new worker mostly adds waiting, connection errors and retries. The same is true of Redis, which carries task queues and coordination traffic, and of a model endpoint with a fixed rate limit or GPU capacity. Before scaling out, identify which shared resource saturates first, and confirm the connection and concurrency limits on PostgreSQL, Redis and each model endpoint with the teams that own them.
Indexing and workflow execution also compete for the same worker pool. Document ingestion, embedding and re-indexing are queued work that can be long and bursty, and they run on the same Celery workers that serve workflow and app execution. A large knowledge base import can fill the queues for a long stretch, so a user-facing workflow waits behind batch jobs even when the total number of workers looks sufficient. The useful question is not how many workers exist but how they are divided. Check the queue backlog during an import, note which task types occupy the workers, and decide whether indexing needs its own worker group or a scheduled window before you add containers to the general pool.
More workers can also move the bottleneck rather than remove it. Each extra worker sends more concurrent model calls, which can hit provider rate limits, produce timeouts in HTTP nodes and inflate latency that looks like slow Dify code. Sandbox and SSRF proxy limits add a further ceiling, since a higher volume of outbound or code execution requests can exhaust the same proxy and sandbox capacity. Treat each change as a test: raise one concurrency setting at a time, watch database connection use, queue depth, provider error rates and response times, and roll back if the constraint simply moved to another component.
Security and operational risks of tuning Dify
Most performance fixes in Dify touch a security control, a resource boundary or a setting that resets behavior for every user. The cards below cover the five changes that most often look like tuning but carry real exposure, and the habits that keep each change reversible. Check each one against your own network and upgrade plan before you apply it.
Loosening the SSRF proxy to fix timeouts
Outbound requests from HTTP nodes, tools and model calls pass through the SSRF proxy configured by SSRF_PROXY settings, which blocks internal and private IP ranges. A timeout or 403 can tempt operators to allow more hosts or disable the filter. Doing so lets a workflow reach internal services, metadata endpoints and admin consoles from inside your network. Before changing anything, confirm which destination fails, whether it is a private address you intend to reach, and whether an allowlist entry for that single host would solve it.
Raising sandbox limits widens exposure
The code sandbox runs untrusted code written in workflows and tools, and agent sandboxes gained network isolation through a Squid proxy in the 1.16.1 release. Raising memory, time or concurrency limits to clear a failing code node lets that code consume more of the host and makes a single runaway script more costly. Keep the sandbox network isolation intact when you change limits, and confirm the release you run includes the isolation changes before relying on them.
Changing SECRET_KEY is not a tuning step
SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials. The environment reference says changing it after deployment logs out all users, and it is not a fix for slow requests or failing sessions. Treat it as a fixed secret set at install time, stored with your other secrets. If a session problem appears during tuning, investigate proxy headers, cookie domains and time sync first.
Upgrades can change defaults
Dify ships minor releases roughly monthly, and the Docker Compose upgrade guidance says to check .env.example for new variables after upgrading. A tuned .env can silently miss a new default, or a renamed key can revert a worker count or pool size you set earlier. Compare your file with the target version, reapply your tuned values, and confirm the sandbox and proxy settings still match your intent after each upgrade.
Record every change with a rollback
Each tuning change alters behavior in a way that is hard to notice later, especially when several changes land together. Keep a change log that names the variable, old and new values, the time applied, the metric that motivated it and the command that reverts it. Take a backup before the change, as the Docker Compose guidance recommends before upgrades, and test the revert on a non-production copy where possible. A change you cannot undo quickly is not a tuning change.
Common questions about Dify performance and outside help
How long does a Dify performance tuning engagement take?
No duration is fixed in advance, because the length depends on the work the environment needs. The main drivers are the number of environments (production alone is simpler than production, staging and development on shared infrastructure), whether a load test can be run against a non-production copy, and how much usable telemetry exists. Without logs that show slow requests, queue depth and database wait times, the first phase is spent collecting evidence before any setting changes. Ask for the phases, the environments in each phase and the conditions each phase depends on before agreeing to a timeline.
What does DeployOpen need from our team before tuning starts?
Tuning goes faster when the inputs are ready on day one. The usual requests are:
- Staging access that mirrors production versions, Docker Compose configuration and the
.envfile, with secrets handled under your own rules - A set of representative workflows, API calls and knowledge base operations, with the expected peak concurrency
- Application, container and PostgreSQL logs covering a busy period, kept long enough to compare before and after changes
- A named person who can approve configuration changes in production and schedule restarts
If log retention is short, the first weeks may go to reproducing the problem rather than fixing it.
Can our own team tune Dify performance without an outside partner?
Yes, a team that is comfortable with Docker Compose, PostgreSQL and Redis can work through the steps in this guide without a partner. The work is methodical: measure first, change one setting at a time, and record the result. Teams usually struggle less with the commands than with reproducing a realistic load and reading the logs correctly, so the question is whether someone has time to run that loop carefully. An outside partner adds value when the same people are also responsible for production, when several environments need the same method, or when the team lacks database or container skills.
What should a Dify tuning quote include?
A useful quote states the scope in writing before work begins. Ask that it lists:
- The environments covered (for example, production only, or production plus staging)
- The symptoms and workflows in scope, and what is excluded, such as model provider latency
- The access the work needs, including load test tooling and log sources
- The deliverables, such as a findings report, changed settings with a rollback plan, and a monitoring checklist
- The handover format and any follow-up work that would need a separate agreement
Labor and any licensing costs should be shown separately, so you can see what recurs and what is one-time.
Do performance tuning changes need a fixed-scope agreement?
Most DeployOpen engagements are fixed-scope, which means scope, deliverables and timelines are agreed in writing before work starts. For tuning, a fixed scope works best when the environments, workflows and symptoms are defined clearly. If the investigation reveals a deeper issue, such as a platform defect or an infrastructure limit outside the agreed scope, the change should go through a written amendment rather than an informal extension. Ongoing tuning, load testing after each Dify release, or continuous monitoring can be arranged as a retainer instead.
Which logs should we keep so tuning findings hold up?
Keep application logs from the api and worker containers, PostgreSQL logs with slow query and connection information, Redis latency data where available, and reverse proxy access logs with response times. Retain them for at least one full business cycle, including a peak period, so that changes can be compared against a baseline. Record the Dify version, the .env values and the container resource limits alongside each log set, because a setting change without that context is hard to interpret later. Confirm your own retention requirements with your security and compliance owners before exporting logs to a support partner.
Do we need a load test environment before changing Dify settings?
A separate environment is strongly preferable, because pool sizes, worker counts and timeout ceilings all interact, and a change that helps one workflow can starve another. The environment does not have to match production in size, but it should run the same Dify version, the same database and Redis configuration, and similar data volumes. If no such environment exists, the work can still start with log analysis, but every production change should have a rollback step and a short observation window. Ask the partner how they plan to test changes before they touch production.
How DeployOpen scopes a Dify performance review
A performance review of a self-hosted Dify starts with a written scope agreed before any work begins. DeployOpen works on Dify inside your own infrastructure, whether that is your cloud accounts, data centers or an air-gapped network, so the review runs against your real deployment and your real workflows rather than a generic test setup. The scope usually names the services in scope (the api, worker, web and plugin daemon containers, plus PostgreSQL, Redis and the sandbox and SSRF proxy services), the applications and API endpoints that users actually hit, and the symptoms that prompted the review, such as slow workflow runs, timeouts in HTTP nodes or a growing queue.
Most engagements of this kind are fixed-scope. The written deliverables typically cover the findings with evidence, the settings changed or recommended, the order in which to change them, and how each change is verified. Ask for the exact list of metrics and logs that will be collected, which environment the tests run in, and who on your side must approve configuration changes. Load testing and ongoing tuning are different pieces of work: a one-time review can be followed by a retainer for periodic checks, load tests before a release, or support when a Dify upgrade changes behavior. Confirm with the vendor how each Dify release changes the relevant settings before you agree a retainer, since minor releases arrive about monthly.
Before you request a quote, gather the inputs that drive effort: the Dify version and deployment method, the number of workspaces and published apps, the model providers and knowledge base sizes, and whether load is steady or bursty. Cost and duration depend on how many components are involved and how much of the test environment must be built. DeployOpen does not publish a price list, so the quote follows from the written scope. Ask the proposal to state which work is one-time, which work recurs, and which items you must provide, such as a staging copy of your data and access to logs.
How DeployOpen works on this
DeployOpen approaches Dify performance tuning as a scoped engagement inside your own infrastructure, starting with measurements of where workflow and API latency actually comes from. We plan, implement and document changes to database pools, Redis, worker counts, sandbox and SSRF proxy limits and HTTP node timeouts, then hand the results to your team to run.
How we scope
- Written scope agreed before work starts, listing the apps, endpoints and symptoms covered by the tuning.
- Review of your current Dify version, deployment layout and the settings that govern connections, workers and timeouts.
- Agreed list of metrics, logs and load patterns needed to confirm each bottleneck before any change is made.
How we build
- Measured baseline for workflow runs and API calls, captured before any configuration is changed.
- Controlled changes to one setting group at a time, with the reason and expected effect recorded for each.
- Load tests against a non-production copy that reproduce the slow paths, followed by comparison with the baseline.
How we hand over
- Tuning runbook listing each changed value, its previous setting, its effect and the condition that justifies it.
- Monitoring checks and alert thresholds your operators can use to spot pool exhaustion, Redis timeouts and worker backlog.
- Walkthrough with your platform and operations staff covering how to repeat the measurements after future Dify upgrades.
How we support
- Ongoing support and re-tuning after each Dify release are available as a retainer, scoped separately from the original work.
- Review of new settings in the environment reference when you upgrade, since performance defaults can change between versions.
- Confirm with your team which costs are one-time (the initial tuning) and which recur (retainer, load testing, monitoring upkeep).
