Why Flowable timers and async jobs stop running
A timer that never fires, or a process that sits after an async step and never moves on, usually traces back to one mechanism: the Flowable async executor. The executor runs timers and asynchronous jobs from a thread pool. When several nodes share one database, they coordinate through lock owners and lock expiration times stored in the job tables. A job is only worked when some node holds a valid lock on it, and when that lock expires, another node is meant to recover the job. Most stalls come from one of three places: the executor is not active on any node that should be running it, the lock state in the database no longer matches what the nodes believe, or the definition or job the executor would pick up is blocked from running. Start by asking which of those three it is, because the fix for each is different.
Teams need this diagnosis in three situations: day-two operations, when a timer that should have fired last night is still sitting unexecuted; after a crash or forced restart, when jobs may still carry locks owned by a node that no longer exists; and after a deployment, when a new process version, a changed configuration or a node that started with the executor switched off can leave work waiting. In each case the first step is read-only. Confirm which nodes are running and whether the executor is active on them, then query the job tables for locks that have not yet expired. Confirm with your platform team how node restarts and clock synchronization are handled in your environment before you change any setting, because a wrong guess about lock timing can cause the same job to run twice.
Clock and time zone problems look like executor problems, which is why they are easy to miss. Timer due dates are stored as absolute times, so a node whose clock runs ahead or behind the database can release or hold locks at the wrong moment, and a misconfigured time zone can shift the time a schedule fires. Ask your database and infrastructure owners to confirm that all nodes and the database use synchronized time, and check the Flowable configuration and advanced documentation for the version you run before you assume the engine behaves the same way. The Flowable advanced documentation describes the executor's locking model and is the next read for the engineers who will own this diagnosis.
Diagnose a stalled timer or async job in order
This sequence moves from the engine as a whole down to the individual job, so you can rule out a node-level problem before inspecting one record. Working in this order usually shows whether the fault is in the executor, the lock, the definition or the clock.
Confirm the async executor is active
Check that the job executor (async executor) is started on at least one node that shares the stalled database. Look at the engine startup logs for the executor starting, and if you expose the Spring Boot actuator endpoint at /actuator/flowable or use JMX, check the executor state there. A healthy result is a running executor on at least one node. If no node reports it active, the timer or job will not run anywhere, and you should confirm the configuration that enables the executor for each deployment.
Check the job lock and its owner
Query the job table in the engine schema for the stalled job and read its lock owner and lock expiration time. A job that is locked by a live node and has a future expiration is being worked on and should clear once the work finishes. Compare the lock owner against the nodes you know are running. Record the job ID, lock owner and expiration so you can judge the next steps against them.
Verify whether the lock has expired
Compare the lock expiration time with the current database time. An expired lock on a job should be picked up by another node once the executor on that node polls, which is how Flowable recovers work after a node crashes. If the lock is expired and no node takes the job over, return to step one and confirm that at least one executor is running and can reach the database. If the lock is still in the future, the owning node may still be processing it or may be stalled.
Check for suspended definitions or instances
Look up the process definition and the process instance for the job, and check whether either is suspended. Suspension can apply to a whole definition, which stops new instances from starting, or to a single instance, which stops its timers and async steps from running. A normal result is an active definition and an active instance. If either is suspended, record who suspended it and whether that was intended before you resume it through the engine API or your admin tooling.
Compare clocks and time zones across nodes and the database
Confirm that every engine node and the database server report the same current time within the tolerance your timers need, and that the timer due date stored for the job is what you expected. Check the time zone the application server and database use, and whether the database column type stores millisecond precision. Mismatched clocks can make a timer look overdue on one node and not yet due on another. A consistent result is that all nodes and the database agree on the time and the due date matches your model.
Confirm the fix and document the cause
After correcting the cause, watch the job move from the scheduled or locked state to completed, or confirm the timer fires at its due date. Check the engine logs for the executor picking up the job and for any new exception. Record the root cause, such as an inactive executor, a dead lock owner, a suspended definition or clock drift, so the same check can be repeated quickly next time. The job is resolved when it completes and no new failure appears in the logs.
Which async executor settings to check first
This table compares the four configuration areas that most often explain jobs that never run: executor activation, thread pool and node setup, lock timing and database time precision. Read it by matching the symptom you see to a row, then confirm the exact property names against your Flowable version before changing anything.
| Setting area | When it fits | Symptom that points to it |
|---|---|---|
| Async executor activation | Check first when no timer or async job ever leaves the queue on any node, including a fresh test deployment. | Jobs appear in the job tables with due dates in the past, and no thread logs show the executor starting. |
| Thread pool and node setup | Check when the executor is active but work is slow to start, or when only one of several nodes ever picks up jobs. | Jobs run on one pod or server only, or queued jobs wait while the pool appears idle in thread dumps. |
| Lock timing and lock owner | Check after a node crash, forced restart or long pause, when jobs stay locked to a node that no longer exists. | Jobs show a lock owner and a lock expiration time in the future, and they resume only after that time passes. |
| Database time precision | Check when timers fire late, out of order or not at all on MySQL, particularly older MySQL versions. | Timers due within the same second fire together or in the wrong order, and due dates look truncated in the database. |
| Clock and time zone alignment | Check when nodes run in different time zones or their clocks drift, since due dates and lock expirations depend on time values. | Timers fire hours early or late, or lock expirations appear to have passed or not passed on different nodes. |
| Suspended process definitions | Check when one definition's timers stop while other definitions on the same engine still run. | Timers for one process version never fire, and the definition shows as suspended in the repository or API. |
| Database connection and schema | Check when the engine logs database errors during job acquisition or the schema version does not match the engine. | Repeated acquisition exceptions in logs, or the engine refuses to start against an older schema. |
Five causes that account for most stalled jobs
When a timer never fires or an async step never continues, the cause is usually one of a small set of conditions in the engine configuration, the database, or the process definitions. Work through these five in order, because each one leaves a different trace in the job tables and logs.
Async executor not activated
The async executor runs timers and asynchronous jobs from a thread pool. A profile switch, a missing property, or a Spring Boot configuration change can leave it disabled, so jobs are written to the database but nothing picks them up. Check that the executor is active in the startup logs and confirm the setting in your deployed configuration, not only in the source repository. Compare the active profiles on the node that is stalled with a node that works.
Expired locks from a crashed node
Each job is locked by an owner until a lock expiration time. If a node crashes mid-execution, its locks remain until they expire, and then other nodes should recover the jobs. If jobs stay locked with an owner that no longer runs, check the lock expiration settings and whether the surviving nodes still have their executors running. A lock that never expires points to a clock or database problem rather than the crash itself.
Suspended definitions or instances
A suspended process definition or instance stops its timers and async jobs from executing, and this often happens after an operator suspends a version for maintenance and forgets to reactivate it. Query the definition and instance state directly in the engine tables or through the management API before changing any job. Confirm whether suspension was intentional, and check whether suspension applies to the definition, the instance, or both.
Clock skew between nodes and the database
Due dates are compared against the time on the node that evaluates them and the time stored in the database, so drift between application servers and the database server can make timers fire early, late, or not at all. Time zone differences in the JVM or database session add a second source of error. Compare the system time on every node with the database server clock, and confirm that the JVM and database use the time zone you expect.
Database timestamp precision limits
Timer due dates depend on the precision of the timestamp column in the database. The configuration docs note that MySQL versions below 5.6.4 lack millisecond precision, which can cause timers that are close together to collide or fire out of the order you modeled. Check the database version and the column type for the job due date before debugging the engine. If the version is older, plan an upgrade rather than tuning the process model.
How expired locks recover jobs after a crash
When a node dies while holding an async job or timer, the job is not lost. Flowable's async executor records a lock owner and a lock expiration time for each job in the database, and once that lock expires another node that is still alive can acquire the job and run it. Recovery therefore has two dependencies: the lock window has to pass, and at least one other node running an active async executor has to be up to pick the job up. If you run a single engine node, or every node shares the same crash, nothing recovers the job until the node restarts. Check the lock duration settings for your version in the Flowable advanced documentation, and confirm that your deployment keeps more than one executor node available.
The window is a trade-off. A short lock expiration means a stalled job is retried sooner after a crash, which shortens the period where a timer is overdue or an async step sits idle. A short window also raises the risk of duplicate execution: a node that is slow, paused by garbage collection or briefly partitioned from the database can lose its lock while the delegate is still running, and a second node then runs the same job. A long window protects against that overlap but leaves processes stalled for longer after a real crash. Set the window against the longest normal runtime of your slowest async delegate, not against an average.
The duplicate-run risk is in your code, not in the engine. Before you tune the lock window, review every service task delegate, script task and async listener that runs under the executor and ask one question: if this runs twice with the same process variables, does the outcome stay correct? Payments, emails, ticket creation and calls to external APIs need an idempotency key or a check against the target system before the side effect. Confirm this in your own delegate code and in the downstream systems it calls, and test it by forcing a job to expire mid-run in a staging environment before you rely on crash recovery in production.
Set up monitoring so stalls surface before users report them
This sequence confirms that the async executor is active on every node, then adds checks on job counts and overdue timers, routes engine logs to the tooling your team already uses, and proves the recovery path with a crash test. The result is a stall that alerts your team before a user notices a process has stopped.
Confirm the async executor is active on each node
After any change to configuration, deployment or infrastructure, check each engine node separately, since a single healthy node can hide a stopped one. Use the job executor state exposed through your Flowable runtime, such as the actuator endpoint or JMX if your deployment enables them, and confirm the executor reports as active on every node that should run jobs. Record the node names and states in your change ticket. This step is done when every node that should run jobs shows an active executor and a sample timer scheduled on that node has fired.
Add a check on job counts and overdue timers
Create a query or dashboard panel that counts executable jobs, timers past their due date and jobs with an expired lock owner. Run it against the engine database or the monitoring data your deployment already exposes, and confirm the field names against your Flowable version's documentation before you build alerts on them. Set thresholds with the process owners, because a healthy backlog differs between a nightly batch and a customer-facing flow. This step is done when the query returns the expected counts on a quiet day and an alert fires when you insert a deliberately overdue test timer in a non-production environment.
Route engine logs to existing tooling
Configure the logging framework in your application so that engine packages log at a level that shows executor start, lock acquisition and job failure messages, and send those logs to the platform your operations team already uses. Add the process definition, instance and execution identifiers to the log context so an alert can be traced to a specific case. Confirm the log level for engine packages in your configuration file, not only in a developer profile. This step is done when a forced job failure in staging appears in your log search with its identifiers within the time your on-call process expects.
Alert on executor and lock anomalies
Turn the checks from the previous steps into alert rules with named owners and a documented response for each one. Cover three conditions: no executor activity on a node that should be running, overdue timers above your agreed threshold, and locks that remain held past their expiry. Route each alert to the team that can act on it, and state in the runbook whether the response is a restart, a data fix or an escalation to the vendor. This step is done when each alert has an owner, a runbook link and a test trigger that a team member has successfully reproduced.
Rehearse a node crash in a non-production environment
Run a node with a long-running timer or async job in a staging cluster, then stop that node abruptly while the job is held. Watch whether another node picks up the work after the lock expires, using the lock expiration behavior described in the Flowable advanced documentation as the reference for what to expect. Measure the observed recovery time against your own tolerance, since the engine does not promise a fixed figure. Keep the test in a repeatable script so it can be run after each upgrade. This step is done when the job completes on a surviving node and the alert from the previous step fired during the outage.
Confirm clock and time zone consistency across nodes
Check that every engine node and the database server report the same time and the same time zone configuration, because timers compare due dates against the clock of the node that evaluates them. Compare the system time on each host with a reference source your infrastructure team trusts, and confirm that application servers do not set a different default time zone from the database. Include this check in the monitoring baseline so a drifting host surfaces as an alert, not as timers that fire late. This step is done when a comparison across all nodes shows no drift beyond your agreed tolerance and the result is recorded with the monitoring configuration.
Review the monitoring setup after each change
Add the executor check, job count query, log routing and crash test to your change procedure for every upgrade, configuration change and new process deployment. Ask the team that owns each process to confirm the expected timer volumes in writing so that alert thresholds stay accurate. Store the scripts and dashboard definitions in version control next to the deployment pipeline. This step is done when a change to the engine or its nodes cannot close without the checks passing and the record of the run is attached to the change.
Operational risks when changing job execution
Fixing a stalled timer usually means touching the async executor, a diagnostic endpoint or the clock, and each of those can create a new problem on a live cluster. The cards below cover the four changes most likely to make job execution worse before it gets better, with the control or check that keeps each one contained.
Tuning executor settings under load
Raising the number of job executor threads or shortening the wait between acquisition cycles can speed up a backlog, but it also increases database contention on the job tables and the lock updates that every node performs. Change one setting at a time in a non-production copy, then watch lock wait times and the count of waiting jobs in the production database before and after. Record the previous values so a rollback is a single step.
Exposing diagnostic endpoints while troubleshooting
Opening the /actuator/flowable endpoint or enabling JMX to inspect a stuck job can reveal process counts, instance data and, through JMX operations, the ability to enable the job executor or deploy definitions. Keep these endpoints bound to an internal management port, require authentication, and remove the exposure once the investigation closes. Confirm which operations your JMX service URL permits before anyone connects.
Duplicate execution after lock recovery
When a node crashes, its expired locks are recovered and the jobs are picked up by another node. If the crashed node was mid-way through a delegate that called an external system, the external action may run twice. Make delegates idempotent with a business key, and check that downstream services reject repeated requests. Review which async steps call payment, notification or provisioning services before enabling aggressive lock recovery.
Changing time settings on live nodes
Timers are scheduled against the time the engine records, so moving a node's system clock, time zone or JVM time zone while jobs are pending can make due timers fire early, late or not at all. Synchronize all nodes with NTP before any change, and schedule clock or time zone edits in a maintenance window with a count of due and overdue timers recorded beforehand. Confirm the database session time zone matches the application servers.
Deploying definitions during a job backlog
Deploying a new process definition version or suspending a definition while jobs are queued can leave existing jobs pointing at a version that behaves differently or is no longer active. Check how the engine handles running instances on the old version, and inventory pending timers and async jobs before the deployment. Keep a rollback definition available and verify a sample of pending jobs after the change.
How the fix differs for Spring Boot and plain Java embedding
Spring Boot applications pull in Flowable starters, which cover all engines, the REST application, and the individual process, CMMN and DMN engines. In this model, most timer and async problems trace back to application configuration, so begin there before looking at the database.
- Starter selection: Confirm which starters are on the classpath. A process-only starter and an all-engines starter can behave differently when a timer depends on an engine that is missing.
- Spring Boot version: Match the Flowable version to a supported Spring Boot line. The 8.0.0 release notes state there is no backward compatibility for Spring Boot 3, so a mismatched build may leave the executor in an unexpected state.
- Actuator endpoint: Expose
/actuator/flowableand watch whether running process counts change as timers come due. A count that never moves points to the executor or to definitions that were never deployed. - Executor settings: Find the async executor activation setting in the Spring Boot docs and confirm its name in the version you run.
- Lock owners per replica: Confirm every replica runs its own executor and that each one appears as a lock owner in the job tables.
Check that the same configuration is present in every replica, that the actuator shows the expected definitions, and that the executor reports as active after startup.
When Flowable is embedded as a JAR in a servlet container such as Tomcat or Jetty, the engine lives inside a web application. Its lifecycle follows the deployment, so a redeploy or a stopped context can leave executor threads idle while the database still holds locks from the previous run.
- Configuration location: The engine configuration sits in the application's own code or configuration files, not in the container. Review the code that builds the process engine configuration and confirm the executor is activated there.
- Context lifecycle: Confirm the executor starts when the web application starts and stops cleanly on undeploy, so that a reload does not leave threads orphaned.
- Shared databases: When two web applications share one schema, confirm which one owns the executor for each set of jobs. Ask your team to document this ownership before the next release.
- Container thread limits: Check that the container's own thread pools leave room for the engine's executor threads under load.
- Monitoring path: Without the Spring Boot actuator, monitoring usually runs through the engine's Java API or direct queries on the job tables. Confirm which method your team uses to count due and locked jobs.
Verify that a redeploy restores job execution without manual lock cleanup, and that the executor is running in the deployed context rather than only in a development setup.
In a plain Java SE process, the application owns the engine lifecycle from start to finish. Nothing starts the async executor for you, and nothing restarts it after a crash unless your launcher or service manager does. The first check is always the startup code path.
- Startup code: Locate where the process engine is built and confirm the async executor is activated in that path. The Flowable getting started guide describes the embedding options this model relies on.
- Shutdown handling: Confirm the process closes the engine on shutdown so that locks held by the node are released or expire on schedule.
- Supervision: A service manager or container orchestrator should restart a crashed node. Confirm the restart policy and check how long a node can be down before jobs are picked up elsewhere.
- Engine logs: Confirm engine logs reach your log store. Executor failures often appear there first, before a process stops visibly.
- Node identity: Each JVM that competes for jobs needs its own lock owner. Confirm that nodes do not share identifiers, since shared identifiers can make one node's locks look like another's.
Before sign-off, stop one node on purpose in a test environment and confirm that another node picks up its due timers after the lock expires.
The Flowable REST application runs the engines behind an HTTP interface. Timers and async jobs still execute inside that server process, so troubleshooting starts with the server and the database behind it, not with the HTTP calls your clients make.
- Deployment target: Confirm the REST application is running and that it is connected to the job tables your team is checking. A second schema can make the job list look empty.
- Clustered REST nodes: When several REST instances share one database, confirm the executor is active on each instance that should run jobs and that lock expiry is in effect across them.
- Configuration: Set executor behavior through the REST application's server configuration. The Flowable configuration docs list the supported databases and related settings to confirm against your version.
- Health checks: A healthy HTTP response does not prove the executor is running. Add a check on due job counts, using the actuator endpoint where your deployment exposes it.
- Version alignment: Confirm the REST application version matches the engine and the schema version. A mismatch can make job state hard to interpret.
After any change, trigger a test timer, confirm it fires, and confirm the due job count drops back to its normal level.
Questions teams ask before bringing in outside help
How much effort does diagnosing timers that never fire take?
Effort depends on how many places the fault could sit, so a diagnosis is quick when the symptom points to one cause and longer when several layers interact. The main drivers are the number of process nodes and deployed definitions, the database platform and its clock behavior, whether custom delegates or listeners touch the job tables, how many application nodes share one schema, and how far apart the servers and users sit in time zones. Ask any partner for a written estimate that lists which of these they will check and which they assume. A fixed-scope diagnosis states its deliverables and boundaries before work starts, so you can see what is included.
Can our internal team find the cause of stuck timers without help?
Often yes, if the team has someone who can read the Flowable job tables, the application logs and the deployment history together. The checks are well defined: confirm the async executor is active on at least one node, look for jobs with lock owners and lock expiration times that are in the past, check whether the process definition is suspended, and compare server clocks and the database time zone. Outside help adds value when the cause is intermittent, spans several nodes, or involves a custom delegate that holds a transaction open. Be honest about who on your team will own the fix after the diagnosis.
What do we need to give a partner to start a diagnosis?
Plan to provide read access to the relevant logs and the job tables, a copy of the affected process definitions, and the deployment history for the period when the problem started. You will also need the Flowable version and the Spring Boot or servlet container setup, the database type and version, and the number of application nodes. Confirm with your security team how database access will be granted, whether through a read-only account, a masked copy or a session on your network. Agree in writing which systems the partner may touch and who approves any change.
Does the work include changing our production database or process models?
That depends on the written scope. A diagnosis normally ends with findings, a recommended fix and an estimate for applying it, while the fix itself can be a separate line item. Changes to job rows or lock fields in production need a documented rollback plan and a named approver on your side. Ask the partner to state whether they will change code, configuration, data or only advise, and whether the change will run through your normal review and deployment pipeline.
How is a fixed-scope engagement agreed and documented?
A fixed-scope engagement is agreed in writing before work starts. The written scope names the systems in scope, the deliverables, the access your team will provide, the assumptions and the exclusions, and it links to a timeline agreed with you. Changes after signing go through a written change request that states the effect on scope and schedule. Ask for the deliverable format in advance, such as a findings report, a runbook or a verified fix, so acceptance criteria are clear.
What recurring costs follow a timer or async job fix?
The one-time cost is the diagnosis and fix. Recurring costs come from monitoring the async executor, reviewing failed jobs after releases, keeping the clocks and time zone settings consistent across nodes, and retesting timers when Flowable or the database is upgraded. Ongoing support, upgrades and continuous testing can be covered by a retainer, which you should price against the monitoring and response expectations you set. Ask for the retainer terms in writing, including what is covered between releases.
Will you guarantee that our timers fire after the engagement?
We cannot promise a guarantee of outcomes, because timers depend on your infrastructure, database behavior and application code that we may not control. What we can do is define the diagnosis steps, the evidence you receive and the verification checks, such as a test timer that must fire across two nodes after a simulated node crash. Ask for those verification steps in the written scope so you can judge whether the result meets your need.
How a Flowable job diagnosis is scoped and delivered
A diagnosis of stuck timers or async jobs is scoped in writing before any work starts, and that scope decides what you get back. DeployOpen works on self-hosted Flowable engines inside your own infrastructure, so the first step is agreeing which environments, database and engine deployments are in scope, and who on your side grants read access to the job tables, logs and configuration. Most engagements are fixed-scope, with the deliverables and timelines agreed in writing. For a job diagnosis, a typical deliverable set is a findings report covering the async executor state, lock ownership, suspended definitions and clock settings, a prioritized list of fixes, and the changes that need your change board approval. Confirm with us whether the scope includes a fix-and-verify cycle or stops at a written diagnosis, because those two outcomes are priced and timed differently.
Ask for the scope to name the evidence you will receive: the affected job and timer records, the ACT_RU_JOB and ACT_RU_TIMER_JOB rows that matter, the executor configuration as deployed, and the node logs for the window in which jobs stopped. Also agree how production data is handled, including whether a copy of the database is needed and where it may be stored. DeployOpen does not publish a price list. A quote follows from the written scope, so the inputs that move cost are the number of engines and nodes involved, the number of database systems, how much history must be read, and whether the fix must be tested in a non-production environment before release.
Ongoing support, upgrades and continuous testing are available as a retainer. A retainer suits teams that see timers or async steps stall repeatedly, or that plan Flowable version changes and need someone familiar with their deployment on call. Ask for the retainer terms in writing, including response expectations and what is excluded. Treat a one-time diagnosis and a retainer as separate purchases, and confirm which costs recur and which are one-time before you approve either.
How DeployOpen scopes and supports Flowable operations
DeployOpen approaches Flowable timers and async jobs that stop firing as a bounded operations engagement inside your own infrastructure. We start from the written scope, reproduce the stalled behavior in your environment, and document the fix so your team can run the engine after handover.
How we scope
- Inventory the Flowable engine version, database type, async executor settings and how many nodes share one schema.
- Define which timers, async steps and job types are in scope, and which nodes and environments the work covers.
- Agree in writing the diagnostics, configuration changes and deliverables before any work starts, since engagements are fixed-scope.
How we build
- Reproduce stalled timers and async jobs in a test environment that matches your database, clock setup and node count.
- Verify async executor activation, lock owner and lock expiry behavior, and suspended definitions against the engine's own job tables.
- Test clock and time zone handling across application servers and the database, then apply and validate the fixes in your pipeline.
How we hand over
- Document the root causes found, the configuration keys changed and the reasoning behind each change for your engineers.
- Provide runbook steps for checking job state, releasing expired locks and restarting the executor after a node crash.
- Walk your platform and engineering teams through the monitoring signals to watch so stalled jobs surface before users notice.
How we support
- Scope ongoing support, upgrades and continuous testing as a retainer if your team wants DeployOpen to stay involved.
- Plan Flowable version upgrades and engine changes as separate written scopes so job behavior is retested each time.
- Ask any partner, including DeployOpen, to confirm response expectations and handover responsibilities in the written scope.
