Flowable high availability guide

How Flowable high availability works across multiple nodes

When process execution must survive a node failure or absorb more load than one engine instance can handle, you can run several Flowable engine instances side by side against one shared database. The async executor is what makes this work: it runs timers and asynchronous jobs from a thread pool, and it coordinates the nodes through lock owners and lock expiration times stored in that database. A node claims a job by writing its identity and a lock expiry, and when that lock expires, another node can take the work over. The pattern looks simple on a diagram and is harder in practice. The database becomes the single point of coordination, so its availability, failover behavior and connection limits now govern the whole cluster, not only persistence. Lock timing decides how quickly a crashed node's work is picked up. A lock duration that is too long leaves timers and jobs stranded after a failure, while one that is too short invites nodes to contend for work that is still in progress. Plan these values with your workload in mind and test them under failure, not only under normal load. Configuration drift is the next risk. If nodes differ in async executor settings, thread pool sizes, job acquisition behavior or the deployed process definitions, timers and jobs can run late, run more than once, or behave differently depending on which node picks them up. Treat the configuration as one artifact, version it, and verify each node against the same baseline before you admit it to the cluster. Schema changes add another coordination problem: database schema upgrades must be rolled across nodes in an order that keeps old and new engine versions compatible for the duration of the rollout, which needs an explicit plan. The open source Flowable documentation describes the lock-based mechanism, but it does not design your topology for you. Decide how many nodes you need, how the database is made highly available and how clients reach the nodes, then build and test those choices. Confirm with your own engineers which database type and version your topology will use, since Flowable documents a range of supported databases and version-specific caveats. The Apache 2.0 engines do not include a clustering product, so the orchestration, load balancing, database failover and monitoring around them are design and implementation work that your team or a partner must own. Before go-live, run failure drills that stop a node in the middle of active timers and async jobs, then confirm that other nodes resume that work within the window your business process can tolerate. Repeat the drill during a rolling schema change and during a database failover. Record the observed behavior, because it is the evidence that your configuration, not the documentation, meets your availability target.

Components that carry the cluster state

A Flowable cluster has no coordinator process. Each node is a stateless engine, and the shared relational database holds the process state, timers and job locks that let the nodes agree on who runs what. The cards below cover each component and the settings to verify before go-live.

Stateless engine nodes

Each engine node is either the Flowable engines embedded as a JAR in a Spring or servlet container application, or the Flowable REST application run as its own service. Nodes keep no process state in memory that other nodes depend on, so any node can pick up work from the database. Check that every node points at the same datasource and that no node holds a private copy of deployed process definitions.

Shared relational database

The database is the single source of cluster state, so its availability sets the availability of the whole platform. Flowable's configuration docs list H2, MySQL, Oracle, PostgreSQL, DB2 and Microsoft SQL Server as supported types. H2 suits testing only. Confirm the exact versions your database platform supports with the Flowable release you plan to run.

MySQL timer precision limit

Flowable's configuration docs note that MySQL versions below 5.6.4 lack millisecond precision. Timers with due times close together can then fire in the wrong order, which affects any timer-driven step in a multi-node setup. If you run MySQL, confirm the server version before design work starts and record the answer in the architecture decision log.

Async executor thread pool

The async executor runs timers and asynchronous jobs from a thread pool on each node. Nodes coordinate through lock owner and lock expiration columns in the database, so a job claimed by one node is visible to the others. Size the pool per node and test how many threads each node can run without starving the database connection pool.

Lock expiration and job recovery

When a node dies while holding a job, its lock expires and another node picks the job up. The lock expiration time therefore decides how long a stalled job waits before recovery. Test a forced node termination during a running job, then confirm the job completes once exactly and that no side effect in your service tasks runs twice without idempotency.

Load balancer in front of REST

When clients call the Flowable REST application, a load balancer distributes requests across nodes. Health checks should target a node-level endpoint so that a failed node is removed from rotation. Confirm session and authentication handling is stateless across nodes, and that sticky sessions are not required for correct behavior.

Actuator monitoring endpoint

The Spring Boot starters expose an actuator endpoint at /actuator/flowable for monitoring running processes. Use it to confirm that each node reports the process and job activity you expect. Restrict the endpoint to operations networks, since it describes live process state.

Flowable high availability decisions and their trade-offs

This table compares the main architecture choices for running Flowable on several nodes, from where the async executor runs to how the database is made highly available. Read each row as a decision: the first column names the choice, and the other columns say when it fits and what it adds to daily operations. The main takeaway is that node topology is the easier part, while database failover behavior determines how safely locks and timers survive a failure.

Decision	When it fits	Operational cost
All nodes run the async executor	Small or moderate job volume where one pool of identical nodes keeps deployment simple and every node can pick up timers and async jobs	Every node competes for job locks in the shared database, so lock contention, job queue monitoring and thread pool sizing must be tuned on each node
Dedicated job nodes	High timer or async job volume, or when you want REST and UI traffic kept away from background job execution	Two node roles to deploy, scale and monitor, plus configuration to disable the async executor on API-only nodes
Active-active engine nodes behind a load balancer	Normal production setups that need horizontal scale and no single engine instance as a point of failure	Health checks, session-independent configuration and shared schema upgrades must be coordinated across all nodes
Active-passive with failover	Teams with limited tolerance for multi-node coordination who accept a failover window before service resumes	Standby capacity still costs money, and failover runbooks, drills and automated promotion logic need regular testing
Engine embedded in your application	Java teams that want process calls inside their own service boundary and deployment pipeline	Each application instance carries the engine, so engine upgrades and application releases are tied together
REST application as a separate tier	Mixed-language callers or teams that want the engine operated as its own service with a stable HTTP interface	An extra tier to scale, secure and monitor, and network paths between application callers and the engine must be secured
Sticky sessions for REST callers	Callers that hold server-side state tied to one node, or legacy clients that cannot retry against another node	Uneven load across nodes, and a node failure drops the affinity so callers must be able to reconnect cleanly
Stateless routing for REST callers	Callers that can send each request to any node and retry on failure, which fits most engine APIs	Callers need retry logic and idempotency handling, and any state must live in the database rather than in node memory
Synchronous replica database	Teams that need the database to hold committed locks and timers through a node loss, with failover handled by the database platform	Replication latency adds to every write, and failover timing depends on the database product, so confirm failover behavior with the database team
Managed cluster database	Organizations that already run a managed database service and want the provider to handle node replacement and failover	Failover behavior, connection handling and minor version maintenance follow the provider's model, so confirm lock and transaction semantics with the database team
Multi-region replication	Requirements for regional survival where a whole site or region can be lost	Asynchronous replication may lose recent committed writes, so expired lock recovery and timer firing after failover must be tested against the replication lag you accept

Build sequence for Flowable high availability

This sequence takes a Flowable deployment from one engine to a pair of stateless nodes on a shared database that keep timers and async jobs running through a node failure. Each step ends with a concrete check, so the platform, database and process teams can see when the build is ready. The sequence closes with a scripted node-kill drill that is kept as evidence.

Confirm database type, version and precision
Confirm the database type, version and timestamp precision before anything else is configured. The Flowable configuration docs list H2, MySQL, Oracle, PostgreSQL, DB2 and Microsoft SQL Server as supported types, and they note that MySQL versions below 5.6.4 lack millisecond precision, which matters when nodes compare job due times and lock expiry. Your DBA checks the installed version and the timestamp column type against that page. Done when the version and precision are recorded in the build record and the platform team signs off on them.
Stand up two identical engine nodes
Deploy two Flowable nodes from the same build with identical configuration. Your platform team packages the application, typically a Spring Boot application using the official starters or the Flowable REST application, and runs two instances with the same engine version and the same datasource settings. Mixed engine versions sharing one schema are a risk to avoid, and the team should confirm the expected behavior with Flowable's documentation. Done when both nodes log the same engine version at startup and a diff of their configuration files is empty.
Confirm both nodes share one schema
Point both nodes at one shared schema and start them in sequence. Start the first node so it creates or upgrades the schema, then start the second node and confirm it validates the existing schema without attempting conflicting changes. Check the schema version record and the startup log on each node. Done when both nodes start, neither logs a schema mismatch, and both report the same schema version.
Enable async executor and test single execution
Enable the async executor on each node and prove that a timer or async job runs exactly once. Deploy a test process with a timer and an async service task that writes a marker row to a test table, then let it fire while both nodes are running. The Flowable advanced documentation describes how the async executor runs jobs from a thread pool and coordinates nodes through lock owners stored in the database. Done when the marker count is one per job across repeated test runs.
Review lock expiration before tuning
Review the lock expiration behavior in the advanced documentation before changing any tuning value. The async executor stores lock owners and lock expiration times in the database, so a job held by a node that fails becomes available to another node once its lock expires. Read the lock settings in the Flowable advanced docs, confirm each property name and its effect with the vendor documentation, and record the default values in your environment. Done when the team has a written note of the chosen values and the reason for each, and any change has been tested outside production.
Route REST traffic and verify health checks
Place a load balancer in front of both nodes and route REST traffic to each of them. Configure the health check to use the actuator endpoint, which the Flowable Spring Boot docs describe at /actuator/flowable for monitoring running processes. Confirm which health endpoint your Spring Boot version exposes and restrict access to the actuator endpoints so they are not public. Done when the balancer marks both nodes healthy, stopping one node removes it from rotation, and a REST call succeeds through the balancer on either node.
Wire database HA and prove failover
Set up database high availability and prove that the engine survives a failover. Your DBA configures replication or a clustered database according to the vendor guidance for the chosen engine, and the Flowable datasource connects through the endpoint the database team provides. In a test environment, start a process, trigger a database failover, and confirm the connection pool recovers. Done when a process instance started before the failover completes after it, and the test shows no lost or duplicated jobs.
Finish with a scripted node-kill drill
Finish with a scripted node-kill drill that repeats the earlier checks under failure. The script starts a batch of timer and async jobs, stops one node abruptly, waits for the lock expiration window to pass, and then checks that the surviving node completes every job exactly once. Run the drill on a schedule and keep its output as evidence. Done when the script passes on repeated runs, the logs show the surviving node picking up expired locks, and the results are stored with the build record.

What drives effort and cost for a multi-node Flowable setup

This table lists the factors that change the effort of a multi-node Flowable setup, why each one matters and how to keep it down. Read it as a list of failure modes to test, because effort grows with the number of scenarios that must be verified, not only with the number of nodes. The main takeaway is that the test matrix, the database platform and the async workload usually matter more than the node count itself.

Driver	Why it matters	How to keep it down
Number of nodes and expected job volume	Each node runs its own async executor thread pool and competes for locked jobs in the shared database, so more nodes add lock contention and more recovery cases to verify.	Size the cluster from measured async job throughput in a load test, and start with the smallest node count that meets your failover requirement.
Timer count and due-date spread	Large numbers of timers due at the same moment, such as month-end batches, produce bursts of lock acquisition and can delay jobs that other nodes must later recover.	Spread timer due dates where the business rules allow, and load test the peak due-date window rather than average volume.
Database platform and existing HA database	Flowable supports H2, MySQL, Oracle, PostgreSQL, DB2 and Microsoft SQL Server, and the cluster depends on the database for lock owners and expiry times, so failover behavior follows the database platform.	Reuse a database platform your team already operates with proven high availability, and confirm the MySQL version supports millisecond precision if you use MySQL (versions below 5.6.4 do not).
Number of integrations that create async work	Every service task with asynchronous continuation, call activity or external event adds jobs that must survive node loss, and each integration adds its own retry and timeout behavior to test.	Limit async boundaries to steps that need them, and document retry and idempotency rules for each external call before testing starts.
Number of environments for failover testing	Killing nodes, pausing the database and simulating network partitions must run on a production-like topology, so each extra environment repeats the test matrix.	Use one parameterized test suite that runs in a shared pre-production environment, and reserve a full copy of production topology for final sign-off.
Customization of job handling or custom executors	Replacing the default async executor or adding custom job handling changes lock and retry semantics, which means the expired lock recovery behavior must be re-verified from scratch.	Keep the default async executor where possible, and confirm with Flowable documentation and your own tests before adopting any custom job handling.
Compliance evidence requirements	Assessors often ask for proof of failover, recovery time and job integrity, so the team must capture logs, test results and configuration records in addition to running the tests.	Agree the evidence list before the test plan is written, and automate the capture of test output and configuration snapshots. Confirm the exact evidence expected with your auditor or assessor.

How the setup changes by deployment model

A single node is the baseline for the rest of this topic. Flowable runs in one process, so no other node can take over jobs that a stopped node was holding. Teams use this model to validate the engine, the database schema and job behavior before adding coordination between nodes. It gives no failover, so any recovery plan depends on restarting the node quickly.

Node lifecycle: one process is started and stopped, so timers and asynchronous jobs pause until the node returns.
Database access: one connection pool points at the database type you selected from the supported list in the configuration documentation.
Load balancing: none is required, since clients reach a single endpoint directly.
Health checks: basic liveness checks are enough, and a Spring Boot deployment can use the actuator endpoint at /actuator/flowable to monitor running processes.
Failover drill: no automatic failover exists, so the drill is a controlled restart followed by confirmation that due timers and queued jobs resume.

Confirm in your environment that a restart recovers pending timers and jobs without manual intervention, and record how long the restart takes. That measurement becomes the baseline against which the multi-node models are judged.

Securing the database, nodes and monitoring endpoints

A multi-node Flowable cluster has more places where access can go wrong than a single engine: the shared database, every node's connection to it, and the management endpoints that expose running processes. The cards below cover the controls that matter most for this topic. The full hardening scope, including script and expression restrictions and dependency updates, is in the Flowable security hardening topic.

Least-privilege database user

Each Flowable node connects to the shared database with one account. Create that account with only the rights the engine needs to read, write and lock rows in its own schema, and keep schema creation and upgrade rights with a separate migration account. Confirm during testing that the engine still starts, runs the async executor and recovers expired locks under this restricted account, since failures often appear only at those points.

Encrypted node-to-database connections

Every engine instance sends job lock updates and process state over the network to the database, so an unencrypted connection exposes that traffic on any segment it crosses. Enable TLS on the JDBC connection, verify the server certificate on each node rather than trusting any certificate, and confirm the setting against your database vendor's driver documentation. Check that the setting applies to every node, including nodes added later.

Credential rotation without a cluster restart

Database passwords and client secrets used by service tasks will eventually need to change. Decide in advance whether your nodes read credentials from a secrets manager at startup or at connection time, because that choice determines whether a rotation needs a rolling restart. Test a rotation on a staging cluster by changing the password, confirming that new connections succeed and that the async executor keeps running on existing nodes.

Restricting actuator and REST API access

The Spring Boot starters expose an actuator endpoint at /actuator/flowable for monitoring running processes, and the REST application exposes the engine's APIs over HTTP. Both reveal process instances, variables and task data. Limit which networks and identities can reach these endpoints, keep the management port off public interfaces where your deployment allows it, and verify with a test request from outside the approved network that access is refused.

Auditable admin actions across nodes

When an administrator suspends a process, retries a job or changes a definition, the action should be traceable to a named identity and a time, on whichever node handled it. Confirm that your identity integration passes the user identity into engine calls, that history records capture the actions you need to review, and that node logs are forwarded to a central store. Test the trail by performing a sample admin action and tracing it end to end.

Verifying that Flowable high availability fails safely

A Flowable cluster is only as trustworthy as the tests that have tried to break it. The core check is a node kill during active work: start a batch of asynchronous jobs and timers across several engine instances, terminate one instance abruptly while its jobs are locked, and confirm that the remaining nodes pick those jobs up once the lock expiration time in the database has passed. Record how long the recovery took from the kill to the last completed job. Flowable's advanced documentation describes lock owners and lock expiration times as the coordination mechanism, so the test should measure that mechanism directly rather than assume it. Set the acceptable recovery window from your own service levels and the lock duration your team has configured, and confirm the exact lock settings with your Flowable version before you run the test. The second group of tests targets duplicate execution, which is the failure that causes the most damage in process systems. After the node kill and recovery, query the job and timer history for each process instance and confirm that every asynchronous job and every timer produced one completion record. A job that runs twice may charge a customer twice, send two notices or call an external system twice, so the pass criterion is zero duplicates for side-effecting service tasks. If your delegates call systems that are not idempotent, the test will expose that gap, and the fix belongs in the delegate design or in the integration layer, not in the cluster configuration. Run the same check with the node killed at several different points in a job's life, including just after a lock is taken, because the outcome in each case is what your operations team will be asked about. Timers need a separate check because a timer that fires late or early can look healthy in the logs. Compare the actual fire time of each timer against its expected due date, using the process instance's recorded timer creation and duration values. Measure the offset distribution across the whole run, not only the average, and pay attention to timers due during the node kill and during the database failover. Agree on the tolerable offset with the business owner of each process before the test starts. Stating a numeric threshold in advance keeps the review honest, and you should set that number from your own requirements rather than from a generic figure. Database failover and schema checks cover the layer under the engine. Run a planned database failover while the engine is under load, then confirm three things: the engine reconnects without a restart, no process instance, task or job state is lost, and the work in flight at the moment of failover either completes or is retried correctly. Compare row counts and sampled instance states before and after. Separately, verify that every node reports the same database schema version before and after the test. A node running a different schema version than its peers is a common cause of intermittent faults during rolling upgrades, and it is simple to check through the database and the engine's startup logs. Finally, confirm that the monitoring alerts fire. Trigger each alert on purpose: a stopped node, a growing count of overdue jobs, a timer backlog, and a database connection failure. Confirm that the alert reaches the on-call channel, that it names the affected node or component, and that it clears when the condition ends. The Spring Boot starter exposes an actuator endpoint for running processes, which can feed these checks if you use that integration. The pass criterion is that every alert you rely on fires within the window you set and is acknowledged by the team that owns the response. A cluster that passes the failure tests but has silent alerts still fails safely only on paper.

What to monitor, back up and document after go-live

A clustered Flowable deployment keeps working only if someone watches the async executor, protects the shared database and rehearses failures before they happen. The cards below cover the signals to track, the backup and restore checks, the upgrade gate and the runbook that ties them together.

Job backlog and failed job monitoring

Track the size of the async job queue, the count of jobs in a failed state and how long timers wait past their due time. Rising backlog with idle nodes usually means lock contention or a stalled thread pool, while a growing failed count points to a delegate or external service error. Set alerts on trend, not only on a single threshold, and review failed jobs with the owning team on a fixed schedule.

Expired lock recovery events

Each node holds jobs under a lock owner and a lock expiration time stored in the database. Watch for recovery events, where a surviving node picks up jobs whose lock expired after a node stopped responding. Frequent recoveries suggest long-running service tasks, garbage collection pauses or a lock duration that is too short for real workloads, and the cause should be found before the lock settings change.

Actuator output and engine metrics

The Flowable Spring Boot starters expose an actuator endpoint at /actuator/flowable for running process information. Confirm which endpoints your Flowable version exposes, restrict them to internal networks and feed the output into the same dashboards as your JVM and database metrics. Compare figures across nodes to catch an instance that has stopped taking work.

Database backup and restore testing

The shared database is the state store for process instances, tasks, jobs and history, so every node depends on it. Take backups on the schedule your recovery objectives require, and restore them into a separate environment on a regular basis. Confirm that a restored copy starts the engines, resumes waiting timers and passes a smoke test of in-flight process instances.

Rolling upgrades with schema compatibility checks

Replace nodes one at a time only after confirming the database schema change is compatible with the nodes still running the previous version. Run the schema check against a copy of production data before the first node is replaced, and keep the old version available for rollback until the last node is upgraded. Upgrades that cross major versions need their own regression plan, which is covered in the Flowable upgrades and support topic.

Runbook for node loss, failover and stuck jobs

Write a runbook that names the steps for losing an engine node, failing over the database and clearing jobs that will not complete. Each procedure should state the signal that triggers it, the owner, the commands or console actions, and the check that confirms recovery. Rehearse the runbook at least once against a non-production environment so the steps are tested before an incident.

Flowable high availability questions buyers ask before hiring help

Does Flowable open source include a clustering product for high availability?

No. The open source Flowable engines do not include a clustering product. Multi-node operation relies on stateless engine instances sharing one database, with the async executor coordinating through lock owners and expiration times stored in that database. Flowable's commercial products are built on the same engines and are licensed separately, so confirm with Flowable which products, if any, provide the clustering, monitoring or failover features you need and in what form. Get that answer in writing before you plan the architecture, because it determines whether your team builds the coordination layer itself. Load balancing, session handling and database failover remain your responsibility either way.

Which databases does Flowable support for a shared high availability setup?

The Flowable configuration documentation lists H2, MySQL, Oracle, PostgreSQL, DB2 and Microsoft SQL Server as supported database types. In a shared high availability setup, the database is the component that matters most, so your choice should follow what your organization already runs and can make highly available. MySQL versions below 5.6.4 lack millisecond precision, which affects timing and job ordering, so check your version against your plan. Confirm with your database administrators that the replication, failover and backup approach you intend to use is supported on the platform you choose. H2 is typically used for development and testing.

What drives the cost of a Flowable high availability project?

The main driver is scope: the number of nodes, the number of environments, the databases involved, and how much failover behavior you must prove before go-live. One-time work includes architecture review, async executor and load balancer configuration, database setup, load testing, failover drills and runbooks. Recurring work includes patching Java and Spring dependencies, upgrading the database and repeating failover tests after changes. Your own platform and database staff time is a cost even when no outside firm is involved. Ask any quote to list each activity, who performs it, which environments are covered, what testing is included and what is excluded. DeployOpen does not publish a price list, and a quote follows from the written scope.

How long does it take to build and test a multi-node Flowable setup?

Duration depends on the number of nodes, the database platform, how much existing configuration must change, and how much failover testing your standards require. No researched figure covers this, so ask for a timeline tied to a written scope and a list of test cases. Build work usually starts with a single node on a non-production database, then moves to several nodes under load. Plan separate time for the load test and the failover drill, because those steps often expose lock timing and connection pool issues that configuration reviews miss. If your change board requires a fixed window, raise that in scoping so the test plan fits it.

What does a Flowable cluster need from our database and operations teams?

Your database team owns the cluster database, its high availability design, backups, connection limits and schema change windows. Platform engineering owns the node runtime, load balancer, health checks, secrets and deployment pipeline. Operations owns monitoring and alerting, including the actuator endpoint at /actuator/flowable in Spring Boot deployments, and the failover runbook. Application teams must confirm that process models and Java delegates are safe to run on several nodes, which means tolerating retries and possible duplicate execution. Agree these responsibilities in writing before work starts, because gaps between teams are where failovers tend to fail.

Can our internal engineers build Flowable high availability without an outside partner?

Yes, if your team has experience with the Java stack, your chosen database, and distributed failure modes, and if it has time to test them properly. The open source engines are licensed under Apache 2.0, so there is no license fee for the engines themselves. The difficult part is proving behavior under node loss, database failover and concurrent job execution, not editing configuration. An outside partner adds most value through an independent design review and a tested runbook. Ask your team to name the lock timing settings it changed and to show a failover test result. If it cannot, a fixed-scope engagement covering design review and testing may be a better fit.

How do expired async executor locks get recovered when a node fails?

Flowable's advanced documentation says the async executor runs timers and asynchronous jobs from a thread pool and coordinates nodes through lock owners and lock expiration times stored in the database. When a node stops, its locks expire, and other nodes can pick up the jobs. Verify this in your own environment before go-live. Check three things: the lock duration setting, the clock behavior of your database and application nodes, and whether long-running jobs can exceed the lock window. Test by stopping a node during job execution and confirming the job completes once. Also confirm that your delegates are safe to repeat, because a job may run more than once after recovery.

How DeployOpen scopes, builds and hands over Flowable high availability

DeployOpen plans, builds, tests, documents and hands over multi-node Flowable deployments inside your own infrastructure. Each engagement starts from a written, fixed scope that covers the stateless engine nodes, the shared database and the async executor behavior your processes depend on.

How we scope

Inventory which BPMN, CMMN and DMN engines run, and whether they use the REST application or embedded Spring Boot.
Identify timers, asynchronous jobs and business-critical processes that must keep running when a node fails.
Agree in writing which database platform, failover model and recovery targets the deployment must meet.

How we build

Configure identical stateless engine nodes that share one database, with async executor settings reviewed per node.
Verify lock owner and lock expiration behavior by stopping a node mid-job and confirming another node resumes it.
Test database failover against your supported engine database type, and document any version-specific precision limits.

How we hand over

Deliver runbooks for node addition, node removal, lock recovery checks and database failover procedures.
Provide monitoring setup, including the Spring Boot actuator endpoint for running processes where it applies.
Hand over test results, configuration records and open items so your team can operate the cluster without us.

How we support

Offer ongoing support and upgrade work as a retainer, scoped separately from the initial fixed-scope build.
Re-run failover and lock recovery tests when Flowable, Spring or database versions change.
Confirm with your team which support and escalation terms apply, since these are set in the written agreement.