Dify with Redis Sentinel or Redis Cluster: when the bundled Redis container stops being enough
Dify can be pointed at a highly available Redis topology, but the choice between Sentinel and Cluster, and the settings around it, must be confirmed against the Dify environment variable reference for the exact version you run. The Docker Compose stack bundles a single Redis container that serves both task queuing for the worker and caching for the API. If that container fails, background jobs such as knowledge indexing stop and cached data is lost, and a platform team with workflows that people depend on will feel it quickly. The Enterprise edition on Kubernetes lists Redis 6 or later as a requirement, which gives you a baseline for planning, though the topology itself is your decision. This section covers mode choice, TLS and ACL settings, separating the Celery broker database from other data, and key prefixes on shared instances.
Plan for a highly available Redis when uptime expectations are formal, when several teams or applications depend on the same Dify instance, or when Redis is already run as a shared service with its own failover process. Sentinel suits a primary with replicas and automatic failover, and it generally fits teams that want high availability without partitioning data. Cluster spreads keys across shards and is designed for capacity as well as availability. Confirm with your Dify version whether each mode is supported for the Celery broker, for caching and for the other keys Dify writes, because support can differ between those roles. For a small team with a single workspace and a tolerant maintenance window, a well-backed standalone Redis with monitoring is usually the more proportionate choice, and the effort is better spent on backups and host sizing first.
Before you change any setting, check which Redis database indexes and key prefixes your Dify version uses, then decide whether the Celery broker gets its own database or its own instance. Create a dedicated ACL user with only the commands and keys Dify needs, enable TLS on the connection if your network policy requires it, and test failover in a non-production environment by stopping the primary and confirming that queued tasks resume. Record the exact settings that worked for your version so the next upgrade can be checked against them. The configuration keys and their behavior should be verified in the Dify environment variable reference and the Dify Docker Compose guide before rollout.
Standalone, Sentinel or Cluster: which Redis topology fits which team
This table compares standalone Redis, Redis Sentinel and Redis Cluster across failover, client behavior, operating load and networking. Read each row against your team's operating maturity, then confirm which Redis modes the Dify version you run supports before committing to a topology.
| Dimension | Standalone Redis | Redis Sentinel | Redis Cluster |
|---|---|---|---|
| Failover behavior | No automatic failover. A container or host failure stops Dify's Celery queues and cache until the instance is restored from its own backup or restart. | Sentinels monitor a primary and promote a replica when it fails. Dify must connect through Sentinel-aware settings, and failover is not instant, so in-flight tasks may need retry. | Data is sharded across primaries with replicas per shard. Failover is automatic per shard, but multi-key operations must stay within a hash slot. |
| Dify support to confirm | Supported as the default Compose setup. Confirm the REDIS_* variables in the environment variable reference for your version. | Confirm in the environment variable reference and release notes for your Dify version whether Sentinel settings are exposed before choosing this mode. | Confirm in the environment variable reference and release notes for your Dify version whether cluster mode is exposed. Do not assume support from the Redis documentation alone. |
| Client and library needs | Any standard Redis client. Only host, port, password, TLS and database number need to be set. | Sentinel-aware client configuration with a sentinel list and a service name. Test failover in staging with your exact Dify build. | Cluster-aware client with node discovery. Confirm that Celery broker and cache usage in Dify tolerates cluster key distribution. |
| Operational load | Lowest. One instance to patch, back up, monitor and size. | Moderate. Three or more sentinel processes plus a primary and replicas to monitor, with failover drills to schedule. | Highest. Multiple shards and replicas, resharding procedures, and capacity planning per node. |
| Networking needs | One reachable endpoint on the internal Dify network. Simple firewall rules. | Clients reach sentinels and the promoted primary, so sentinel ports and announced addresses must be routable from every Dify host. | Every node must be reachable by clients and by other nodes on the cluster bus port. Plan for that port in firewall and network policy. |
| Separate broker database | Use distinct database numbers for the Celery broker and cache. Multiple logical databases are not available in cluster mode, so plan a separate instance instead. | Keep the broker and cache on separate logical databases or separate replicated sets, and confirm the Sentinel setup covers each. | Logical database separation is not available in cluster mode. Plan separate clusters or a different isolation method for the broker. |
| Key prefixes on shared instances | Use a distinct key prefix per Dify environment if the instance is shared, and document it in the runbook. | Apply the same prefix discipline. Keys still need separation when staging and production share a Sentinel group. | Prefixes help readability but do not replace isolation. Hash-tag behavior should be reviewed for multi-key operations. |
| When it fits | Single-host or small non-critical deployments where a short outage and manual recovery are acceptable. | Production teams with an existing Sentinel practice that need automatic primary promotion without sharding. | Large deployments where a single primary cannot hold the load, and the team already runs Redis Cluster operations. |
Why the Celery broker should get its own Redis database
Dify uses Redis for more than one job. It acts as the task broker that hands work to Celery background workers, and it holds cache-style data that the API and workers read and write during normal operation. When both roles share one logical database, a flush meant for cached data can also discard queued tasks, and memory pressure from cache growth can push out broker keys. Failover adds a second concern: after a switch to a new primary, you need to know which role lost data and whether in-flight tasks were replayed. Separating the roles makes those effects easier to reason about during an incident.
The decision comes down to two options. Placing the broker and cache in separate logical databases on the same Redis instance is simpler to deploy and monitor, and it works well when memory and failover behavior are acceptable for both roles. Running a separate Redis instance for the broker gives stronger isolation, because flushes, eviction policies and memory limits no longer interact. The trade-off is one more component to provision, secure and monitor. Whichever you choose, record the decision with the Redis topology so that the next person does not assume the roles are independent.
Confirm before you design this split. Check which Redis database index settings your Dify version exposes in its environment reference and in .env.example, and whether the broker and cache indexes can be set independently. Then check how those indexes behave under Sentinel or Redis Cluster. Cluster mode does not support multiple logical databases in the same way as standalone or Sentinel, so a layout that uses separate indexes on a standalone instance may need a separate instance or a different key strategy in Cluster. Test a flush and a failover in a staging environment before you rely on either layout in production.
Settings to plan before you change the Redis endpoint
Moving Dify off a single Redis container changes more than the host name. The endpoint, credentials, encryption, database index, key naming and timeouts all have to agree, or the API and the Celery worker will fail in different ways. Plan each setting once, then confirm its exact name and behavior in the environment variable reference for the Dify version you deploy.
Connection endpoint and discovery
Decide whether Dify connects to one stable address, such as a virtual IP or proxy in front of a Redis primary, or whether the client must discover the current primary through Sentinel or follow Cluster slots. Confirm in the environment variable reference which Redis host, port and Sentinel-related settings your version reads, and whether it supports Sentinel discovery at all. A standalone-style endpoint behind a failover proxy is often the simplest option to validate.
Password and ACL user
Decide whether Dify authenticates with a single shared password or a dedicated ACL user limited to the keys and commands it needs. Every Sentinel and replica node must accept the same credentials, or failover promotes a node that rejects Dify. Confirm which Redis username and password keys the reference defines and whether the Celery broker and cache use one credential or separate ones.
TLS on the client connection
Decide whether traffic between Dify containers and Redis crosses a network you do not fully control. If it does, plan certificates for every node the client may reach, including Sentinel nodes, and a trust store the containers can read. Check the reference for the TLS switch, certificate path and verification options in your version, and test a failover with TLS enabled before cutover.
Broker database index
Redis databases are numbered, and Sentinel and Cluster modes support only database 0 in standard deployments, so separating the Celery broker from cache data may require a different design. Decide whether the broker gets its own Redis instance or a database index that your topology allows. Confirm in the reference which variables set the broker and cache databases, and check whether your Redis mode permits the index you choose.
Key prefixes for shared instances
If several Dify environments or other applications share one Redis deployment, key prefixes stop their data from colliding. Decide a naming convention per environment before the first deployment, because renaming keys later means losing queued tasks or cached values. Confirm in the reference whether a prefix or namespace setting exists for your version and whether the Celery broker key layout can be separated from other keys.
Timeouts and retry during failover
During a primary switch, connections drop and commands fail for a short window. Decide how long the API and workers should wait before reporting an error, and how many retries are acceptable before a queued job is marked failed. Confirm the timeout and retry settings in the reference, then rehearse a controlled failover and watch workflow runs and document indexing for stuck or duplicated tasks.
How to move Dify from one Redis container to a Sentinel or Cluster topology
This sequence takes a Dify deployment from the bundled single Redis container to a Sentinel or Cluster topology, with a backup, a tested failover and a verified background workload at each stage.
Back up Dify data and configuration
The operator who owns the Dify host takes a backup of the metadata database, the storage volumes that hold uploads and knowledge base files, the current .env file and the docker-compose override files before any change. Store the copies outside the host and record the Dify version in use. Before you start, review the Docker Compose upgrade guidance for the target version on the Releases page, which advises a backup before every change. The step is complete when a restore of the backup has been tested on a separate host, or when the team has confirmed that the restore procedure works and who signs it off.
Stand up the Redis topology in a test environment
The platform or infrastructure team builds the chosen Redis mode in a non-production environment that matches the production network. Sentinel needs at least one primary, replicas and Sentinel processes spread across failure domains. Cluster needs multiple primary shards with replicas and a slot map that the Dify client can reach. Confirm the Redis server version against the Dify release notes and the Docker Compose documentation, then confirm that the chosen TLS and ACL settings are enabled. The step is complete when the team can list the topology, reach every node from the Dify API and worker hosts, and authenticate with the Dify account.
Confirm replication and controlled failover
The infrastructure operator checks replication lag and roles on each node, then restarts the current primary in a controlled way. For Sentinel, verify that a replica is promoted, that the Sentinel quorum agrees and that the old primary rejoins as a replica after it returns. For Cluster, verify that the slot ownership moves to a replica and that the cluster reports a healthy state. Record the time the failover took and any errors. The step is complete when the topology returns to a healthy state with no data loss on the keys the team has written during the test.
Update Dify environment settings for mode and TLS
The Dify operator edits the .env file in a copy first, using the environment variable reference for Redis to set the host, port, password or ACL user, TLS options and the mode-specific values for Sentinel or Cluster. Compare the result with .env.example to catch variables added since the installed version, as the Docker Compose documentation advises after an upgrade. Keep the SECRET_KEY value unchanged, because changing it logs out all users. The step is complete when the team has a reviewed .env diff that lists every Redis-related change and each change has an owner.
Separate the Celery broker database and set key prefixes
The Dify operator assigns the Celery broker to its own Redis logical database and gives Dify cache and session keys a distinct prefix, so that other applications sharing the same instance cannot collide with them. Confirm the database numbers and prefixes in the environment reference, and in a Cluster, confirm that multi-key operations are supported by the settings in use. Check the broker and cache keys in Redis after a restart to confirm that each one lands in the expected database and namespace. The step is complete when a key listing shows Dify keys only under the planned prefix and database.
Restart Dify and verify worker connections
The Dify operator restarts the api, worker and related services and reads the logs for the worker and api containers. Look for a successful Redis connection on startup and no repeated reconnect messages. Then trigger a background task that uses the worker, such as a knowledge indexing job on a small test document, and confirm that the job finishes and the document status changes to indexed. The step is complete when the worker logs show successful connections, the indexing job completes and the team has recorded the log lines used as evidence.
Repeat failover with Dify under load
The infrastructure team runs a light, repeatable workload that keeps workflows and an indexing job active, then restarts the Redis primary as in the earlier test. Watch the worker logs for reconnect attempts, failed tasks and retried jobs, and check that the indexing job and workflow runs complete after the topology recovers. Compare the results with the baseline failover test. The step is complete when the team has a written result showing whether tasks were lost, retried or left in a failed state, and the operator has decided whether the topology is ready for production.
Securing Redis traffic, credentials and shared instances
Moving Redis from a single container to Sentinel or Cluster adds nodes, ports and credentials that each need a control. The cards below cover transport encryption, least-privilege access, secret handling, network reachability and key hygiene. Dify's own secret and outbound request settings are separate from Redis, so they need their own review.
TLS for client connections
Decide whether Redis clients connect over TLS and apply the same setting to every Dify service that reads or writes Redis, including the workers that consume Celery tasks. Confirm in the Dify environment reference which Redis TLS variables are supported in your release, because not every Redis client option is exposed there. Plan certificate rotation before go-live, including how each node reloads its certificate and how Dify picks up the new trust chain. Test the failover path with TLS on, since a Sentinel promotion that works in plaintext can fail on certificate validation.
Least-privilege ACL users
Create separate Redis ACL users for Dify rather than running every component as the default user. Limit each user to the databases, key patterns and commands it needs, and keep administrative commands out of application accounts. Sentinel and replication traffic need their own credentials, which should differ from the application user. Confirm with Redis documentation which command categories your Redis version supports before writing the rules, and test each Dify feature after tightening them.
Redis password outside version control
Keep the Redis password and any ACL passwords in the host env file with restricted file permissions, or in your secrets manager, and inject them at deploy time. Do not commit the .env file or copy values into compose overrides stored in Git. Rotate Redis credentials through a planned change that updates every Dify service at once, because a partial update will stop workers while the web tier still connects. Record which secret store holds each value so that rotation has an owner.
Network reachability limited to Dify hosts
Allow inbound connections to Redis and Sentinel ports only from the hosts running the Dify api, worker and plugin daemon components, and from the Sentinel or cluster peers. Block the Redis bus and Sentinel ports from the wider network and from workloads that do not use Dify. Verify the rules from a host outside the allowed set, not only from inside Dify. Document the rules next to the topology so that adding a Dify host triggers a firewall change.
Dify secrets are separate from Redis
SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials, and the environment reference says changing it after deployment logs out all users. It is not a Redis password and should not be rotated as part of a Redis change. Likewise, SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges, which is a different control from Redis reachability. Review each setting in its own change record so that a Redis incident does not lead to a wrong fix.
Key prefix hygiene on shared Redis
If several Dify environments or other applications share one Redis deployment, assign each one a distinct key prefix or dedicated database number and confirm which option the Dify release supports. Keep the Celery broker database separate from cache and session data so that flushing one does not remove task queues. Ask your Dify support contact or check the release notes for any key names hardcoded by a given version. Restrict each ACL user to its own prefix so that a misconfigured client cannot read another tenant's keys.
Questions buyers ask before hiring help with Redis high availability for Dify
What drives the effort of moving Dify from the bundled Redis container to Sentinel or Cluster?
The effort depends mostly on how many Dify services need Redis reconfigured, how many Redis-backed features your deployment uses, and how much the environment already has in place. Dify uses Redis for caching and as the Celery broker, so the work covers connection settings for every service that talks to Redis, failover testing, and checks that Celery tasks survive a primary switch. Teams that already run Sentinel or a managed Redis for other systems usually have less to build. Ask any partner to list the Redis touchpoints in your specific Dify version before agreeing to a scope.
Can our own platform team do this internally?
Often yes, if the team has operated Redis HA before and can test failover without touching production. The work is familiar to Redis operators, but Dify-specific details matter: the broker database must be separated from other data, key prefixes must avoid collisions on shared instances, and TLS and ACL settings must match what each Dify container expects. Internal teams usually underestimate the test cycle, especially failover drills under load and upgrade rehearsal. An outside review can be a fixed-scope alternative to building the whole design yourself.
What does the work need from our team before it starts?
Plan for network access between the Dify hosts and every Redis node or Sentinel port, a certificate authority or certificate set if TLS is required, and credentials for ACL users that Dify will use. You also need a non-production Dify environment with a copy of your configuration, and an agreed maintenance window for the cutover. Confirm with your security team how secrets will be stored and rotated, since Redis passwords sit alongside other values in your environment files. Missing any of these items is the most common reason a fixed-scope timeline slips.
How long does this take, and what makes it take longer?
No duration is stated here because it depends on your environment. The main drivers are the number of Dify hosts and services, whether Sentinel or Cluster is chosen, how much TLS and ACL work the security team requires, the state of the existing test environment, and how many rounds of failover testing your change process needs. Approval cycles for certificates and firewall changes often take longer than the technical work. Ask for a schedule that lists each approval gate and test stage separately.
What recurs after go-live, including Redis upgrades and Dify releases?
Redis needs its own patch and major upgrade plan, and each upgrade should be rehearsed against a staging copy, because failover behavior and client compatibility can change between versions. Dify minor releases arrive roughly monthly according to the GitHub releases page, so your Redis settings and any custom configuration must be rechecked against each target version's upgrade guide and the .env.example file. Budget recurring internal time for these checks, failover drills and monitoring of replication lag. Ongoing support can be bought as a retainer if you prefer an outside team to carry that load.
Does Dify Enterprise on Kubernetes change the Redis picture?
Yes, the planning changes, although the Redis requirement is similar. The Dify Enterprise resources checklist lists Redis 6 or later as a requirement, alongside PostgreSQL 14 or later, Kubernetes 1.24 or later and Helm 3.14 or later. In Kubernetes you would normally use a managed or operator-run Redis service rather than hand-built Sentinel on virtual machines, so the questions shift toward operator choice, persistence and network policy. Confirm with the Enterprise documentation and your Dify contact which Redis topology is supported for your edition before choosing a design.
What should we ask for in a fixed-scope quote?
Ask for a written scope that names the Redis topology, the Dify services that will be reconfigured, the TLS and ACL controls, and the key prefix and database separation for the Celery broker. The quote should list deliverables such as the configuration files, a runbook, failover test results and a rollback procedure, plus the client-side checks you will receive. Ask which items are one-time work and which are recurring retainer work, and what your team must provide in each phase. Compare quotes on this written scope, not on headline totals, since the same words can hide very different test coverage.
Limits to confirm in your environment before you commit to a topology
The supplied documentation does not settle how Dify handles every Redis mode, so several points must be checked against the exact release you plan to run. Each card names what to confirm and who can answer it, so the topology decision rests on verified behavior rather than assumption.
Supported Redis modes and TLS options
Confirm which Redis topologies the Dify version you run accepts: standalone, Sentinel or Cluster, and whether TLS and ACL user authentication are configurable through the environment variables for each. Ask the Dify team or check the environment variable reference for the exact keys, because Redis settings are not uniform across every connection Dify makes. Record the answer per release so your upgrade plan can test it again.
Sentinel discovery in the client
Confirm whether the Dify API and worker clients perform Sentinel discovery, meaning they ask Sentinels for the current primary after a failover, or whether they need a fixed address that a proxy or DNS name must update. This determines whether a Sentinel setup removes the single point of failure or only moves it. Test a forced failover in staging and watch for reconnect errors in the api and worker logs.
Cluster slot redirects and key handling
Confirm whether the Dify client follows Cluster slot redirects (MOVED and ASK responses) and whether every multi-key operation Dify issues is compatible with hash slots. Celery and cache usage can behave differently from application keys. Ask the Dify team for a list of supported Cluster behavior and run a load test that exercises workflow runs and knowledge indexing together.
Redis version your team will support
The Dify Enterprise resources checklist requires Redis 6 or later, but that requirement is not a statement about Community Edition or about every Dify release. Confirm with your platform team which Redis major version they can patch, monitor and restore, and confirm with Dify which versions the release you run has been tested against. Include the Redis version in your upgrade test matrix.
Community and Enterprise parity for Redis
Community Edition and Enterprise share a codebase lineage, but their deployment guidance differs in scope. Ask Dify whether Redis topologies, TLS and failover behavior documented for Enterprise on Kubernetes also apply to the Docker Compose Community stack. Do not assume Enterprise guidance carries over until the vendor confirms it in writing for your edition.
Managed Redis in your cloud
Check whether your cloud provider's managed Redis offering exposes Sentinel endpoints, Cluster mode and TLS with the ACL users Dify needs, and whether it supports the Redis version you plan to standardize on. Managed services often hide the topology behind a single endpoint, which changes failover behavior and the connection settings. Confirm these details with your cloud account team and the Dify team before you choose.
How DeployOpen scopes and delivers a highly available Redis layout for Dify
DeployOpen plans the Redis layout for Dify as part of the wider platform design, then implements it inside infrastructure you control. Each step is written down in a fixed-scope agreement, tested against your topology, and handed over with the runbooks your operators need.
How we scope
- We confirm which Redis roles Dify uses in your deployment, including Celery broker and cache traffic, before choosing a mode.
- We compare standalone, Sentinel and Cluster against your failover expectations, network layout and client library support.
- The written scope lists Redis versions, TLS and ACL requirements, database numbering and key prefix conventions agreed up front.
How we build
- We configure Redis nodes, Sentinel or Cluster membership, and Dify environment variables in your own accounts or data centers.
- We separate the Celery broker database from other data and apply the key prefix scheme for any shared Redis instance.
- We test failover by stopping primaries and confirming Dify workers, API calls and background jobs reconnect without manual edits.
How we hand over
- You receive the topology diagram, configuration files and secret locations, with the reasons behind each setting documented.
- Runbooks cover failover checks, password and certificate rotation, and how to restore Redis data alongside Dify backups.
- Your operators walk through the failover tests so they can repeat them after upgrades or host changes.
How we support
- Ongoing support, upgrades and continuous testing are available as a retainer after handover.
- Retainer work can cover Redis version upgrades and rechecks after each Dify minor release, which arrive roughly monthly.
- We re-run failover tests after infrastructure changes so the layout keeps matching the documented design.
