Configuring Flowable failed job retries for downstream outages
When a process step calls an external system or runs a delegate that fails intermittently, Flowable retries the failing async job rather than failing the instance at once. The decision you need to make is whether the retry schedule the engine applies by default will outlast the outages your downstream systems actually have. Teams usually need to tune retries in three situations: an outage that lasts longer than the default retry window, a downstream rate limit that rejects calls until a quota resets, and a call that is not idempotent, so a retry could create a duplicate payment, record or notification. Before you quote any figure, confirm the default attempt count and the syntax of the retry configuration against the Flowable configuration documentation and the advanced documentation for the exact version you run, because retry behavior has changed between major releases.
Start by mapping each service task that calls outside your engine to the outage it must survive and the cost of a repeated call. Read the retry settings that apply to your engine version, including any per-task or per-process-definition overrides, and check how a retry interacts with the async executor and its lock expiration. Confirm with your application team whether each delegate is idempotent, using a business key or an external reference to detect repeats. Where a call is not idempotent, the safer design is usually to move the side effect into its own step with a check that runs before it, rather than to lengthen retries globally. Record the retry window you chose for each dependency, so that operations can tell an intended pause from a fault.
Test the retry plan against a realistic failure in a non-production environment that mirrors your production configuration. Stop the downstream service, watch the job retry count and due dates in the job tables, restore the service, and confirm that the job completes without duplicate side effects. Then exhaust the retries deliberately and confirm that the job moves to the deadletter table and that your alerting fires. Keep the retry values under change control with the process definitions they belong to, since a change to a delegate's dependency often requires a change to its retry window. Ask any integrator quoting this work to state which retry settings they will test, which failure modes they will simulate, and who owns the values after handover.
Failed job retry settings and when each fits
This table compares the retry-related settings Flowable exposes for async jobs, timers and HTTP calls, along with the executor settings that control how often retries run. Read each row for the situation it suits and the risk it carries. Confirm exact property names and defaults against the Flowable configuration and advanced docs before you set any of them in production.
| Setting or pattern | When it fits | Risk to plan for |
|---|---|---|
| Default retry behavior for async jobs and timers | Simple processes where a short, automatic retry covers brief transient errors such as a momentary database lock | Default attempts may run out during a longer downstream outage, pushing the job to the deadletter table |
| failedJobRetryTimeCycle expression | Downstream services with a known recovery pattern, where you want retries spaced out over a defined window | A poorly chosen cycle can retry too fast during an outage or wait too long for a fast recovery; confirm syntax in the docs |
| Retry handling inside a service task or delegate | Calls to external systems where you can catch errors, classify them and decide whether to throw for engine retry | Swallowed exceptions hide failures from the engine; rethrowing without classification can retry non-recoverable errors |
| HTTP task retry pattern | Outbound REST calls where timeouts and 5xx responses are expected and a bounded number of retries is acceptable | Retries that ignore idempotency can duplicate side effects such as payments or record creation |
| Async executor job acquisition settings | Clusters where you need predictable pickup of retried jobs across nodes and time-based locks recover crashed work | Aggressive acquisition adds database load; slow settings delay retries after an outage ends |
| Timer-based manual retry from operations | Teams that want to review failures before reprocessing and control when a downstream system is ready | Relies on people noticing failures; pair with alerting so stuck processes do not wait unseen |
| Deadletter handling after retries are exhausted | Failures that need data or code fixes before the job can safely run again | Jobs left in the deadletter table do not run on their own, so ownership and monitoring must be assigned |
How Flowable failed job retries move through the async executor
The async executor is the component that decides when a failed job runs again, so any retry setting only takes effect through it. According to the Flowable advanced documentation, the async executor runs timers and asynchronous jobs from a thread pool. When several engine nodes share one database, it coordinates them through lock owners and lock expiration times stored in the database. A node that starts a job marks it as locked by that node until a lock expiry time. If the node crashes or stalls, the lock expires, and another node can pick the job up. Retries and locks therefore work together: a failed job is released for another attempt, and the lock is what prevents two nodes from running the same job at once.
When a job throws an exception, the engine records the failure on the job and reschedules it according to the configured retry cycle. Each failed attempt consumes one of the job's remaining retries, and the job is retried until those retries run out. Confirm in the advanced docs and your engine version how the retry count and the next due time are stored on the job, and whether the lock is cleared on failure or only when it expires. Test this on a non-production engine with a delegate that throws on purpose, and watch the job row change between attempts. Do not assume that a job which just failed is being held by a lock in the same way as a job that is running.
Plan for the case where retries run out. The engine stops rescheduling the job, so the process instance stays waiting at that step until someone intervenes. Confirm with the Flowable documentation for your release where exhausted jobs are kept, and how your operations team finds and re-triggers them, because this determines the alerting you need. Also confirm how a job that is locked by a crashed node is recovered, since a long lock expiry can delay recovery and a short one can allow a slow but healthy job to run twice. Match your retry windows and lock timing to how long your downstream services stay down, and document both settings together so a change to one does not silently break the other.
Choosing retry windows that match downstream outages
Transient errors are short failures such as a dropped connection, a brief timeout or a single slow response from a service that is otherwise healthy. They usually clear within seconds, so the retry window should be short and the spacing should grow only modestly. Early retries cover most of these cases, and the total attempt count matters less than making sure the last attempt still lands while the failure is recent.
- Keep early intervals short: a first retry within seconds or a few tens of seconds often succeeds without any load on a struggling service.
- Use a small attempt count: if every attempt fails within a short span, more attempts add delay without adding recovery chances.
- Avoid synchronized retries: many jobs failing together and retrying on the same schedule can hit the dependency at the same moment again.
- Check timeouts first: if the HTTP task timeout is longer than the retry gap, a slow call may still be running when the next attempt starts.
Before setting values, confirm from your logs how long these blips last in your environment, because a dependency that is usually fast may behave differently under your peak load.
Planned maintenance is the easiest pattern to plan for because the start and end are often known in advance. The goal is to span the window with enough attempts that jobs keep trying until the downstream service returns, without retrying so often that the process backlog grows out of control once the service is back. The timing of the window relative to your batch or peak processing schedule matters as much as the retry values.
- Size the total span to the window: the sum of the retry gaps should cover the expected maintenance duration with some margin, and you should confirm that margin with the provider.
- Schedule the maintenance window with the process calendar: avoid planned downtime that overlaps with month-end runs or timer-driven notifications.
- Use moderate intervals: spacing of a few minutes keeps the number of attempts reasonable across a window measured in hours.
- Plan the deadletter path: if the window overruns, jobs will move to the deadletter table, so define who reviews and moves them back.
- Ask the provider for the schedule: request the planned start, end and any rollback window in writing.
Before relying on any span, compare it against the maintenance notices your provider has published for the last several changes.
Extended outages last hours, and no reasonable retry sequence will cover them. Retries that continue for a long time cost little per attempt but can fill the deadletter table and create a large backlog of stalled processes. The better design treats the retry window as one defense and combines it with monitoring, deadletter handling and, where the process allows it, a designed fallback path.
- Do not stretch retries to cover the outage: very long intervals or many attempts mean jobs sit idle for long periods and reappear late in ways users may notice.
- Accept that the window will end: decide what happens to a job after its final attempt, including the alert, the owner and the recovery steps.
- Design a compensating path: where a business step can wait, route work to a queue or a manual task instead of exhausting retries.
- Alert on deadletter growth: a sudden rise in failed jobs often signals an outage before users report stuck processes.
- Plan for reprocessing: after recovery, confirm that moving jobs back does not repeat side effects that already happened.
Before choosing values, review your incident history for outages longer than the window you are considering and count how many jobs each one would have affected.
Rate-limited APIs return errors when calls exceed a quota, and retrying immediately makes the situation worse. The retry window must back off enough for the quota to reset and should avoid bunching retries from many jobs into the same reset period. Many providers signal limits through specific status codes and headers, so the pattern depends on what your provider documents and on how your HTTP task treats those responses.
- Back off beyond the reset interval: spacing retries shorter than the provider's quota window produces repeated failures and burns attempts quickly.
- Spread the load: add jitter or stagger job start times so many retries do not land on the same second.
- Respect provider headers: if the API returns a retry-after value, align the interval with it where your implementation allows.
- Limit concurrency upstream: lowering the number of concurrent jobs against the same API often prevents the errors that trigger retries.
- Count attempts per quota window: make sure the total attempts do not exhaust the quota for other workloads sharing the same credentials.
Before setting values, read the rate limit documentation for your provider and pull a week of error logs to see when limit errors cluster.
Configuring Flowable failed job retries step by step
This sequence takes a team from inventorying external calls to proving that failed Flowable jobs retry on the schedule they intend, so retries match real downstream outages instead of burning attempts during them.
Inventory external calls and their repeat safety
List every service task, script task, delegate and HTTP call in the process and CMMN models that reaches an external system, and for each one record whether repeating the call is safe. Mark calls that create records, send messages or move money as non-idempotent, and note whether the downstream system accepts an idempotency key or a client-supplied reference. Ask the owners of each downstream system what their outage behavior looks like. The output is a table with one row per call, a safe-to-repeat decision and an owner, which will drive every later retry choice.
Confirm the Flowable version and starter
Record the Flowable engine version, the Spring Boot starter or plain engine dependency, and the Java and Spring baseline the application runs on. Retry behavior and configuration keys are tied to the engine line in use, so check the release notes for the version you run before you copy any example. Run a dependency report in the build and confirm the resolved version in the running application, for example through the deployed artifact manifest. Done means the team can name the exact engine version and the configuration mechanism (Spring properties or Java engine configuration) it uses.
Set the retry expression on definitions or the engine
Decide where the retry policy lives: a default on the engine configuration for all async jobs, or a per-definition setting for tasks that need different windows. Express the policy as a failedJobRetryTimeCycle expression, which sets the wait between attempts, and confirm the default attempt count in the configuration documentation for your version rather than assuming it. Keep windows that match the downstream outage you recorded in step one, and avoid a short cycle for calls to systems with long recovery times. Commit the change in the same pipeline as the process definition so that the setting is versioned with the model. Done means the expression is in source control and visible in the deployed configuration.
Add per-task retry handling for HTTP calls
For HTTP service tasks, decide how retries inside the call differ from engine-level job retries. Short retries for transient network errors can sit in the delegate or HTTP client, while the engine retry covers longer outages. Apply the rule that only calls marked safe to repeat get automatic retries, and send the idempotency key from the process instance or business reference on every attempt. Non-idempotent calls should fail into a deadletter path with a clear exception message instead of retrying blindly. Done means each HTTP task has a documented retry owner and a tested exception mapping.
Test failure paths in a non-production environment
Deploy the changed definitions to a non-production environment that mirrors the production database type and async executor settings. Trigger each failure mode you identified: the downstream returns an error, times out, or refuses connections. Confirm in the database that the job's retries count decreases with each failed attempt and that the due date moves according to the cycle. Check that a job which exhausts its retries lands in the deadletter table with its exception stored. Done means every failure path in the inventory has an observed result recorded against it.
Verify retry behavior by forcing a failure
Before release, force one production-like failure on a staging instance by pointing a service task at an endpoint that returns errors. Query the job table and confirm the retries value, lock owner and lock expiration time after each attempt, and confirm that the job is picked up again only when its due date passes. Restore the downstream dependency and confirm the process continues without duplicate side effects. Record the result, including the exact retry count and timestamps, in the change record. Done means the team can show the retry sequence matches the design for each safe-to-repeat call.
Document the policy for operations
Write a short runbook entry for each retry policy that names the downstream system, the expected outage window, the configured cycle and the deadletter owner. Include the database query operators will use to check a failed job's remaining retries and the exception field to read first. Link the policy to the monitoring and alerting work for deadletter jobs so that exhausted retries are seen before users report stuck processes. Done means an on-call engineer who did not write the policy can find the retry setting, the expected behavior and the next action from the runbook alone.
Matching HTTP task retries to idempotent downstream calls
Retrying a call is only safe when repeating it does not repeat a side effect. A failed async job in Flowable is re-executed from its last transaction boundary, so if the job includes a call that already charged a card, sent an email or created a record before the failure was recorded, a retry will do that work again. Before you set failedJobRetryTimeCycle on a process that calls an external system, confirm for each call whether the downstream API is idempotent, meaning a repeated request with the same key produces the same result instead of a second effect. Payment, notification and record-creation endpoints differ widely on this point, and the answer determines whether automatic retry is a safeguard or a source of duplicates.
You have two places to put retry logic. The engine can retry the whole job, which covers every step in the job's transaction, or the HTTP call can retry internally with its own backoff and stop before the engine sees a failure. Engine-level retry is simpler to observe, since failures appear in the job tables and can be counted and alerted on, but it re-runs everything in the job. Retry inside the call keeps the engine out of transient network errors, but the engine then no longer records those attempts, so you need your own logging to see them. Many teams keep the call's internal retries short and let the engine handle longer outages, and this split should be a documented decision for each service task.
Where the downstream API supports it, agree an idempotency key with the owner of that system before enabling any retry. The key should be derived from something stable in the process, such as the process instance ID combined with an activity or business identifier, and it must be the same on every attempt. Ask the owner how long keys are remembered, what response is returned for a repeated key with a different payload, and whether a repeated key after a timeout returns the original result. Also confirm which error codes they consider retryable, how they signal rate limiting, and what their own maintenance windows look like, so your retry windows match real outages. Confirm these answers in writing and test them in a non-production environment before you rely on them.
Security and operational considerations for retry settings
Retries change what sits in logs, what credentials get used, how load lands on shared resources, and who can see stuck work. The cards below cover the checks to make before tuning retry settings in a Flowable engine, and most of them depend on details you should confirm for your Flowable version and deployment.
Exception messages can carry payloads
Each failed attempt records an exception on the job, and the message often includes request details, variable values or downstream responses. A retry that repeats the same failure repeats the same data into logs and the job tables. Before raising retry counts, check which logger packages capture job exceptions, who can read them, and whether your log retention policy fits the data in them. Confirm the exact fields your version writes to the job and deadletter records.
HTTP task credentials during retries
An HTTP task that calls a secured service reuses whatever credentials its configuration or delegate supplies on every attempt. Retries therefore multiply authentication calls against the downstream service, which can trigger lockouts or rate limits on the account. Keep credentials out of process models and resolve them at call time from a secret store. Confirm how your HTTP task implementation handles token refresh and expiry between attempts, since a long retry window can outlast a short-lived token.
Alerting when retries run out
A job that exhausts its retries stops moving without any further attempt, and the process it belongs to can stall silently. Decide who receives an alert when a job reaches the deadletter state, and how quickly. Tie the alert to the process definition and business key so an operator can find the affected case. Test the alert path during a planned outage, because retries that succeed after a short window will never trigger it.
Thread pool and database load
Every retry is a job the async executor must lock, run and update in the database. Many jobs retrying against a slow dependency can occupy executor threads and add write load to the job tables at the same time. Size the executor and the retry windows together, and set the window so retries pause during a known outage rather than firing continuously. Confirm the executor settings and lock behavior for your Flowable version before load testing against your own database.
Change control for retry expressions
Retry settings live in the process definition as expressions, so changing them means deploying a new definition version. Running instances may keep the version they started with, which means a change does not always apply to work already in flight. Put retry expressions under the same review and deployment pipeline as the process model, record which version each change affects, and test the expression against the configuration you intend to run in production.
Access to the deadletter store
Jobs that have exhausted their retries sit in the deadletter store with their exceptions and data. Anyone who can read that store, or move jobs back for execution, can see sensitive content and can replay work against live systems. Restrict read and write access to the operators who handle incidents, record who moved each job back, and review those permissions when staff change roles. Confirm the table and API access controls that your engine version and database provide.
Questions teams ask before configuring Flowable retries
How much effort does it take to tune retries for one process?
The effort depends on how many async steps and timers the process has, how many of them call external systems, and how well those systems' failure modes are understood. A process with one or two internal database steps needs little more than confirming the default retry behavior and reviewing the deadletter path. A process that calls several external services needs each call classified by failure type, a retry window chosen per downstream system, and tests that prove the retries do not repeat side effects. Ask any partner to break the estimate into those per-call items, since that is where the hours actually go.
What drives the cost of outside help with Flowable retries?
Cost is driven by three components: the number of external calls that can fail, the idempotency analysis needed for each one, and the test coverage that proves the chosen retry settings behave as intended. A call to a payment API that must never be charged twice takes longer to analyze than a read-only lookup. Costs also rise if the team needs a test environment built or if downstream outage history has to be reconstructed from logs. Ask for a quote that lists each external call, the analysis and test work attached to it, and whether any work is one-time versus recurring during later reviews.
Can our team configure Flowable retries without outside help?
Yes, if your engineers know the Flowable engine configuration and can reason about idempotency in each external call. The configuration keys themselves are small, but choosing values that match real outages requires knowing how long each downstream system is typically unavailable and what happens when a retried step runs twice. Teams without a clear owner for each external system often find that the hard part is the decision, not the setting. A partner can help with the analysis and review, while your team keeps ownership of the final values.
What does the work need from our side?
Plan to provide three things: a record of downstream outages, including how long they lasted and how often they occurred; named owners for every external system a process calls, who can confirm expected recovery times and rate limits; and a test environment where failing calls can be simulated without touching production data. Without outage history, retry windows are guesses. Without owners, idempotency questions stall. Without a test environment, the team can only verify retry behavior after release, which is the riskiest point to learn that a step runs twice.
How long does a typical retry configuration change take?
Duration depends on scope: the number of processes affected, the number of external calls per process, and whether tests and deployment changes come with it. A change limited to one process with a single internal call is a small task that can be reviewed and tested in a short cycle. A change across several processes that call multiple external systems needs analysis, testing against simulated outages, and a staged rollout. No duration is given here because none was supplied for this topic. Ask any provider for a timeline tied to a written scope with listed deliverables.
Do retry changes require a database or deployment change?
Changing retry settings in process definitions or engine configuration usually requires a deployment, because the values are read from the deployed model or the application configuration. The Flowable engine stores job state in its database, so already queued jobs may keep their existing retry counts until they are processed or moved. Confirm with your Flowable version how the retry configuration is read and whether existing jobs pick up new values. Schema changes are a separate matter and are not normally needed for a retry window change, but you should verify this against the Flowable configuration documentation for your release.
Limits to confirm in your Flowable environment
Flowable's retry behavior depends on the engine version, the async executor's clustering model, the job type and the clock your nodes share. The cards below name the settings to confirm in your own environment rather than values to assume, so a retry window you configure matches what the engine actually does.
Retry defaults and expression syntax
Confirm the default attempt count and the failedJobRetryTimeCycle expression syntax against the release notes for the exact version you run. Flowable 8.0.0 changed the Spring and Java baseline, so a configuration copied from an older example may not match the version in your build. Check the version in your dependency tree, then test one failing job to see how many attempts occur and at what intervals.
Behavior across clustered nodes
The async executor coordinates nodes through lock owners and lock expiration times stored in the database. Confirm how long a lock is held before another node can recover a job after a crash, because that window adds to any retry delay you set. Run a test where one node fails mid-job and check which node picks the job up and whether the attempt counter increments once or twice.
Timers versus async jobs
Timers and asynchronous jobs both run through the async executor, but their scheduling and retry behavior may differ in your version. Confirm whether a failing timer retries on the same schedule as a failing async continuation, and whether a retry cycle changes the timer's due date. Test each job type separately before assuming one policy covers both.
Clock and time zone effects
Retry timing is computed from the clock on the node that schedules it and compared against due dates stored in the database. Nodes with skewed clocks or different time zone settings can fire retries early, late or in an unexpected order. Confirm that all nodes and the database server run time-synchronized clocks, and verify the time zone configuration your engine and driver use.
Limits for non-Java delegates
Retry settings apply to jobs the engine executes, so an HTTP call or external worker outside the Java delegate model may not follow the same retry path. Confirm whether your HTTP task pattern uses engine retries, a service-level retry library or an external worker queue. Decide which layer owns retry counts so a single failure does not multiply across layers.
Database precision and MySQL timing
The Flowable configuration docs note that MySQL versions below 5.6.4 lack millisecond precision. Timestamps with coarser precision can shift due dates for short retry intervals and affect ordering of retried jobs. Confirm your MySQL version if you use it, and check the column types the schema creates for due and lock timestamps on your other supported databases.
How DeployOpen scopes and delivers Flowable retry configuration
DeployOpen approaches Configuring Flowable failed job retries as a written, fixed-scope piece of work inside your own infrastructure. We start from your downstream services, their outage patterns and your Flowable deployment, then agree the retry settings, tests and handover before any change is made.
How we scope
- Inventory every async task, timer and HTTP call that can fail, and record which downstream service each one depends on.
- Agree the retry policy per service in writing, including attempt counts and failedJobRetryTimeCycle values for each case.
- Confirm which Flowable engine version, database and deployment model the retry settings must support before work starts.
How we build
- Configure retry cycles and HTTP task retry patterns in your process definitions and engine configuration under version control.
- Test retry behavior against simulated downstream outages, covering recovery windows and the handoff to the deadletter table.
- Check that jobs retried during an outage are picked up correctly by the async executor across all nodes in your cluster.
How we hand over
- Document each retry window, the reasoning behind it and the downstream outage it was matched against.
- Deliver runbooks that explain how operators read retry counts, exceptions and deadletter entries after a failure.
- Walk your platform, operations and development teams through the configuration and the tests used to verify it.
How we support
- Review retry settings again when your downstream services change, when you upgrade Flowable or when outage patterns shift.
- Offer ongoing support, upgrades and continuous testing as a retainer, scoped separately from the original engagement.
- Ask for a quote built from the written scope, so cost follows the number of services, processes and environments covered.
