Handling Flowable deadletter jobs when retries run out
A process instance has stopped advancing, nobody has been paged, and the first sign of trouble is a user asking where their request went. In Flowable, the most likely cause is an async job that exhausted its retries and was moved to the deadletter. The engine's async executor runs timers and asynchronous jobs from a thread pool, and when a job keeps failing past its retry limit, it stops being picked up for execution and waits in a deadletter state for a person to act. The job does not disappear, and the process instance does not complete on its own. Confirm the exact table and column names for the deadletter job store against the schema for the Flowable version you run, since they change between major releases.
Treat deadletter handling as a recurring operations task rather than a one-time repair. Every failure needs a decision: is the cause a downstream outage that will clear, a bad variable or missing data in one instance, or a defect in a delegate or script that will fail every time it runs? Each answer leads to a different action, such as waiting, correcting data and retrying, or deploying a fixed class and then moving the job back for execution. Retry settings, covered in the sibling topic on failed job retries, determine how many jobs reach this state. The deadletter process itself determines how long they stay stuck. Ask who owns triage, how quickly a deadletter is reviewed, and what evidence is recorded before a job is retried.
How to find deadletter jobs and read the exception
This sequence takes an operator from a suspected stuck process to a confirmed cause for each deadletter job, so the fix targets the right layer. It produces a short triage record per job that says what failed, where, and whether the job can be safely retried.
Confirm the symptom and scope the search
Start from the signal that raised the question, such as a user report, an alert, or a process instance that has stopped advancing. Identify the process definition key, the instance IDs if you have them, and the time window in which the work stopped. Narrow the search to that scope so you do not read every failed job in the engine. The step is complete when you have a written list of candidate process definitions and a time window, agreed with the application owner.
Query deadletter jobs through the management API
Use the engine's management interface to list deadletter jobs, filtering by process definition or process instance where the API allows it. If your deployment exposes REST, query the management endpoints with a service account that has read-only rights. If you work directly in the database, read the deadletter job table for the same filters. Confirm which access path your version and deployment support by checking the Flowable documentation for your release before you rely on a query. The step is complete when you have a list of job IDs with their process instance, execution and creation time.
Read the stored exception message and stack trace
For each job, open the exception message and stack trace stored with it. Record the first exception message, the root cause at the bottom of the trace, and the Flowable class or activity where the failure was raised. Note whether the cause is a connection or timeout error, a null or type error in a delegate or expression, a parse error in a variable, or a constraint violation. The step is complete when each job has a one-line summary of its root cause in your triage record.
Confirm the job belongs to the expected process version
Check the process definition ID and version attached to each job against the version you intended to run. A job created under an older definition can fail against code that has since changed, and a suspended or newly deployed definition can produce failures that look unrelated. Compare the definition key, version and deployment ID with your deployment records. The step is complete when you can state, for each job, whether it runs the expected version and, if not, which version it belongs to.
Classify the failure as outage, bad data or code defect
Match each root cause to one of three categories. A downstream outage shows connection refusals, timeouts or HTTP errors from a service the process calls, and often clusters in time. Bad data shows failures tied to specific instance variables, such as a missing value, an unexpected type or a value outside an expected range, and affects only some instances. A code defect shows a failure in your delegate, script or expression that repeats across instances with the same definition version. The step is complete when each job has one category and the evidence behind it, reviewed by the application owner.
Check whether the failure repeats across instances
Group the deadletter jobs by root cause and by definition version to see whether one fault explains many jobs. A single cause across many instances usually points to an outage or a defect and needs a single fix. Scattered causes that each touch one instance usually need instance-level review. Record the counts per group in the triage record. The step is complete when you know whether you are fixing one problem or several, which decides how the next steps are sequenced.
Record the triage decision before changing anything
Write a triage record for each group that lists the job IDs, the root cause, the category, the definition version, and the proposed action: wait for the dependency, correct the data, or deploy a code fix. Include who approved the action and whether the affected instances are customer-facing. Do not move any job back for execution until this record exists and the owner has signed off. The step is complete when the record is stored with the incident or change ticket and every job has a proposed action.
Fixing the cause before moving jobs back
A deadletter job often fails because a process variable it needs is missing, null, or holds a value of the wrong type. This usually happens when an earlier task wrote a variable under a different name, when a form or API payload changed shape, or when a call activity passed an input the child process did not expect. The fix is normally in data, not code: you correct the variable on the affected execution so the job can run with valid input. Confirm the variable name, type and expected value against the process model and the code that reads it. Changing variables on a live instance is an operational change, so record who made it and why.
- Identify the variable: Read the exception to find the expression or delegate that failed, then list the variables that step reads.
- Check the source: Find which earlier step should have set the value and whether it ran on the same version.
- Correct on the execution: Set the value through the runtime service on the affected instance only, not across the whole definition.
- Avoid silent defaults: A default that hides the problem can route the instance down the wrong path without an error.
- Test on a copy: Repeat the change against a restored copy of the database before touching production.
Before requeueing, confirm that the job's execution now resolves every variable it reads and that the next activity produces the expected result.
When a delegate class or expression throws an exception on every attempt, the defect lives in code or in deployed configuration. A null pointer, a wrong bean name, a class missing from the classpath, or a bad expression are typical causes. Here the fix goes in code and reaches production through your normal release pipeline, not through a manual edit of the job table. Patching a job's data while the code stays broken only delays the next failure. Keep the corrected build and the deadletter ticket linked so the audit trail shows the defect and its fix.
- Reproduce the failure: Run the delegate with the same variables in a unit test or integration test before changing anything.
- Fix and release: Ship the corrected delegate or expression through your standard build and deployment path.
- Check bean wiring: Confirm that Spring or the classpath resolves the class in every environment, including nodes that were not redeployed.
- Confirm the deployed version: Verify that every node runs the new artifact, since a mixed-version cluster can keep failing on old nodes.
- Review related delegates: Search for the same pattern elsewhere, because one defect often appears in several steps.
Before requeueing, check that the fixed artifact is live on every node and that a test run of the delegate completes against realistic variables.
Many deadletter jobs come from a call to a downstream system that was down or rejected requests for longer than the retry window. The process logic may be correct, and the fix is to restore the service, not to change the model. Once the dependency is healthy, the job can usually be requeued as is. If the service changed its contract, such as a new field or a new authentication scheme, the fix moves to the integration code and the deployment path. Confirm that retries are sized to the real outage, because a window shorter than the outage will keep producing deadletter entries.
- Confirm recovery first: Test the endpoint directly from an application node, using the same credentials and network path.
- Check for duplicate side effects: Ask whether the failed call partly ran before the error, since a retry may repeat a payment or record creation.
- Make calls idempotent: Pass a stable request key so the downstream system can ignore duplicates.
- Review retry settings: Compare the retry configuration with the length of past outages.
- Batch the requeue: Move jobs back in controlled groups so a recovering service is not flooded.
Before requeueing, verify that the dependency answers correctly from the engine's network and that duplicate handling is in place for calls that may have partly completed.
A deployment that removes or renames an activity can leave jobs pointing at an activity that no longer exists in the new process definition. The engine cannot resume those jobs against the new model, so they fail. The fix depends on intent. If the activity was removed by mistake, restore it in a corrected definition. If the change was intended, move affected instances to a valid activity with the engine's instance migration support, or cancel the job with a documented business decision. Check the Flowable documentation for the version you run before choosing an approach, because migration behavior varies by release.
- Compare definitions: Diff the old and new BPMN files to list every removed or renamed activity ID.
- Map each instance: Decide, per affected instance, whether it continues, moves, or is cancelled.
- Confirm the target: Check that the activity you move to accepts the current variables.
- Use the migration path deliberately: Test the chosen approach on a copy before running it on production instances.
- Document the decision: Record which instances were moved, cancelled or restored, and who approved it.
Before requeueing, confirm each affected instance points at an activity that exists in its current definition version and that its variables match what that activity expects.
Recovery options and when each fits
This table compares the four ways to recover a deadletter job: fixing data and retrying, deploying a corrected delegate and retrying, moving the job back through management operations, and deleting it. Read each row against the exception you found, and confirm the behavior of each operation in the Flowable version you run before acting on it.
| Option | When it fits | What it risks | What to confirm for your version |
|---|---|---|---|
| Fix data, then retry in place | The exception came from bad or missing process variables or an external record that can be corrected | Correcting data without understanding why it went bad can hide an upstream defect that will recur | Which variables the failing step reads, and whether the retry re-evaluates expressions against the corrected values |
| Deploy a corrected delegate, then retry | The exception is a defect in your Java delegate, script or expression that a code change resolves | A new deployment can change behavior for other running instances, and a bad fix can fail the same jobs again | Whether running instances pick up the new definition or delegate class, and how your pipeline deploys during active traffic |
| Move the job back for execution via management operations | The downstream dependency has recovered and the job's inputs are still valid | Retrying a non-idempotent step can repeat side effects such as payments, notifications or external writes | The management API or Java call that moves deadletter jobs back to the executable table, and whether it resets the retry count |
| Delete the job | The business outcome has already been reached elsewhere, or a compensating action has been completed and documented | Deleting a job can leave a process waiting forever or skip a step your controls expect to run | How the process model behaves when the job is gone, and whether a manual task or audit record must be created first |
| Escalate to the process owner before acting | Financial, regulated or customer-facing steps where the right outcome is a business decision | Delay keeps the process stuck and users continue to see incomplete work | Who owns each process definition, and the approval path for a manual correction |
Alerting on stuck processes before users notice
Deadletter jobs rarely fail loudly. Users notice a stuck approval or a missing notification before anyone checks the engine, so the team needs alerts on the engine's own state. The four signals below cover the deadletter count, job age, timer behavior and instance age, and each one needs a threshold chosen for your workload.
Growing deadletter count
Track how many rows sit in the deadletter job table and alert when the count rises between checks, not only when it crosses a fixed number. A steady count may be acceptable for a batch process, while a sudden jump usually points to one failing delegate or a downstream outage. Read the count directly from the database, since the actuator endpoint reports running work rather than the deadletter table. Start with a short rising-trend window and tune it after your first month of data. Check that the alert names the process definition so the on-call engineer knows where to look.
Jobs older than threshold
Measure the age of the oldest job in the deadletter table and alert when it exceeds a limit your business owners agree to. A job that failed an hour ago may be routine for a nightly settlement process and urgent for a customer onboarding flow. Compare the job creation time against the current time in one time zone, because mixed time zones produce false ages. Pair the age alert with the count alert so a single stuck job in a critical process still gets attention. Record the threshold per process, not once for the whole engine.
Timers that stop firing
Timers and asynchronous jobs that stop executing often produce no deadletter rows at all, so a count-based alert stays silent. Watch for timers whose due date passes without a matching execution, and for a sudden drop in completed timer jobs. Confirm the async executor is active on each node, since the actuator and JMX expose its state. Expired locks after a node crash can also freeze work, so compare lock expiry times against the clock on every node. Where these checks show a gap, use the timer troubleshooting guide before changing retry settings.
Process instance age
Count active process instances grouped by definition and age bucket, then alert when a bucket that should be empty keeps growing. A long-running instance waiting on a human task is expected, but an instance parked in a service task for days usually means a failed job has been waiting for someone to move it. The actuator endpoint at /actuator/flowable exposes running process counts that can feed this view. Ask the business owner to define the expected maximum age for each process, then set the alert from that answer.
Log and audit signals
Engine logs record job execution failures with the exception that caused them, which gives the on-call engineer a second signal when metrics are slow to update. Route errors from the job executor packages to your log platform and alert on repeated exceptions with the same message. If you enable the event logger, the engine events can also show jobs that were moved or retried. Confirm log levels and retention with your security team, since exception text can include business data.
Security and access considerations for job recovery
Recovering deadletter jobs gives operators control over live business processes and exposes the data inside failed steps. The cards below cover who may act, how actions are recorded, what exception text reveals and how management interfaces are restricted. Agree these operating rules before the first manual retry.
Limit who can move or delete jobs
Retrying, moving and deleting deadletter jobs changes live process state, so treat these actions as privileged operations. Grant them to a named operator group, separate from the developers who deploy process definitions, and keep bulk deletes off shared support accounts. The engine leaves authorization decisions to your application, so confirm which API routes, JMX operations and admin screens your application exposes and which roles can reach each one. For how identities and permissions are set up, see the Flowable security and access control topic in this group.
Audit every manual intervention
Each retry, move or deletion should answer who acted, when, on which job and why. Require a ticket reference in every operator request and copy the action into your own log or ticketing system as well as the engine. Check whether your history and event logging capture job-level actions in your Flowable version, and close any gap with an external record before the first production intervention. Auditors and incident reviews will ask for this trail, and a deleted payment job is hard to reconstruct from memory weeks later.
Exception text can carry personal data
A deadletter job keeps the exception that stopped it, and that exception often includes stack traces with variable values, identifiers or SQL parameters from the failing step. Those values can include personal or payment data that should not be widely visible. Limit read access to the deadletter table and to logs that hold stack traces, and redact exception text before pasting it into tickets or chat. Ask whether your logging configuration writes full exception details to shared log platforms, since those platforms often have broader access than the engine database.
Lock down JMX management
JMX can enable the job executor, deploy definitions and run other management operations, so an open JMX port is equivalent to administrative access on the engine. Bind the JMX service URL to an internal interface, require authentication and encrypt the transport, and allow only the operations operators need for recovery. Keep JMX off public network paths and restrict it to the operations and monitoring hosts. Recheck the exposed attributes after each Flowable upgrade, because the set of managed operations can change between releases.
Keep monitoring read-only
The /actuator/flowable endpoint reports process counts, running instances and completed tasks. That data suits dashboards and alerts for stuck work, but it also reveals operational detail about your business. Expose the endpoint on the management port only, behind your network controls, and keep it on a separate route from anything that can change job state. Confirm how your Spring Boot setup exposes actuator endpoints and whether authentication applies to them before you wire stuck-job alerts to that data. The Spring Boot docs describe the starters that expose it.
Setting up a deadletter runbook and alerts
This sequence turns deadletter jobs from a surprise into a managed queue: you know which job types matter, when someone is told, who acts, and how a requeue behaves before it happens in production.
Inventory async and timer job types in use
List every async step and timer in your deployed process and case definitions, grouped by job type, and note which ones call external systems such as payment, identity or messaging services. The engineering lead for each application does this review, using the definitions in the deployment pipeline rather than memory. Done means every job type has a named owner, a business process it supports, and a note on whether a failure stops a customer-facing flow or only a background task. Check the list against the engine's job tables after a test run to confirm nothing is missing.
Agree alert thresholds with business owners
Meet with the owners of each affected business process and agree what counts as a problem: the number of deadletter jobs per definition, the age of the oldest one, and which job types page someone immediately. Engineering proposes values from the inventory, and the business owner confirms how long a stuck instance can wait before a customer, case worker or regulator is affected. The output is a signed-off threshold table with one row per job type. Confirm the table is stored with the runbook, not only in a chat thread.
Build the dashboard and alert rules
Create a query or dashboard panel against the deadletter job table, grouped by process definition and exception type, and wire the thresholds into your monitoring tool. The platform or observability team owns this, and the alert should route to the on-call group for the application. Use the actuator endpoint or metrics your team already exposes where they cover the job state you need, and confirm the exact field names in your Flowable version's documentation. Done means a test deadletter job, created on purpose in a non-production environment, triggers the alert and clears it when resolved.
Write the triage runbook with owners and escalation
Write a short runbook for each alert: how to read the exception stored with the job, which team owns the fix for each class of error (data, code, or downstream service), and when to escalate to the business owner. Include who can authorize a requeue and who can delete or leave a job in place. The application owner writes the first draft and the security or compliance contact reviews access rules for anyone who can change job data. Done means each alert links to a runbook entry and the on-call roster names a person for every escalation level.
Rehearse requeueing a job in a non-production environment
Run a full cycle in staging: force a job to fail, let it reach the deadletter table, read its exception, correct the cause, and move the job back for execution. Follow the runbook exactly, using the engine's management API or console that your team has chosen, and time-box the rehearsal against the threshold the business owner agreed. Record any step that was unclear or needed access nobody had. Done means the rehearsal finishes, the process instance continues to its next step, and the runbook has been corrected from what the team learned.
Record every manual intervention
Each time someone reads, edits, requeues or deletes a deadletter job in production, they log the job ID, process definition, instance, the exception, the action taken, the approver and the time. The application owner reviews the log at a regular interval set by the team. Keep the log in a system with access controls and retention that match your audit and data protection rules, and confirm with your compliance lead where it must be stored. Done means a sample of recent interventions can be traced end to end from the alert to the resolution.
Questions teams ask before handling deadletter jobs
How much effort does recovering Flowable deadletter jobs take?
Effort depends on how many jobs are stuck, how many distinct causes sit behind them, and whether the fix is data, configuration or code. A single missing variable on one process definition is a short job. Dozens of jobs failing for different reasons across several process definitions is a triage exercise first, then a set of separate fixes, each needing testing before jobs are moved back for execution. No duration figure is supplied for this work, so ask for an estimate that lists the failure categories you expect, the number of process definitions involved and whether each fix needs a code release. Also ask who approves replaying jobs that have side effects, such as payments or outbound messages.
Can our in-house team handle deadletter jobs without outside help?
Often yes, if the team knows the Flowable engine well and has access to the database, the deployed process definitions and the application code that runs in delegates. The harder part is usually not the SQL or the API call that moves a job back, but deciding whether replaying it is safe. Teams that lack a Flowable developer, a database administrator or a business owner for the affected process tend to stall at triage. Assess your bench against those three roles before deciding. An outside team can supply the triage method and runbook, while your staff keep ownership of production decisions.
What access and people does a deadletter recovery engagement need from us?
Plan for read access to the engine's job and exception tables in production, a copy of the production schema in a test environment, the deployed process definitions and the source for their delegates and listeners, and logs from the time the failures started. You also need a named business owner for each affected process who can say whether a replayed step is still correct, and an operations contact who can pause the async executor or schedule a replay window. Ask the provider to list each access item, who grants it, and how production data is handled in test environments.
How long does a first deadletter runbook take to set up?
The time depends mainly on job volume and the number of process definitions in scope. A small estate with a few definitions and a short list of known failure types can produce a usable runbook quickly, while a large estate with many definitions, several Flowable versions and mixed external calls needs a longer inventory and sign-off cycle. No duration is supplied for this work. Ask for a written scope that covers the inventory of definitions, the classification of exception types, the query and replay steps, the approval rules for replays and a test plan, and agree the timeline against that list.
What drives the cost of a deadletter recovery project?
The main cost drivers are the number of distinct failure causes, the number of process definitions and external systems involved, whether fixes touch application code that needs review and release, and how much testing the replay requires. One-time costs include the initial triage, the runbook and any fixes. Recurring costs appear when you keep the alerting, monitoring and replay procedure in service as processes change. Request a quote that separates triage, fixes, runbook, alerting setup and any retained support, so you can see which items recur.
When is it worth scoping outside help for stuck Flowable processes?
Outside help is worth scoping when stuck processes affect payments, customer records or regulated decisions and your team cannot confidently say which replays are safe. It is also worth it when failures recur after a release, when several Flowable versions run side by side, or when no one on staff has time to build the alerting. An illustrative trigger: a failing delegate that calls a downstream service and retries the same side effect on each attempt. Ask for a written scope with defined deliverables before work starts, so the boundary between your decisions and the provider's work is clear.
How do we catch stuck processes before users report them?
Alert on the size and age of the deadletter table, not only on application errors, because a job that exhausts its retries stops without throwing anything a user sees. Pair that alert with the job executor state and the actuator endpoint at /actuator/flowable for process counts, so a sudden drop in progress shows up early. Confirm which metrics your monitoring stack can read from the engine's database and logs, and agree who receives each alert and what the first response is.
How DeployOpen works on this
DeployOpen plans, implements, hardens, documents and supports self-hosted Flowable engines inside your own infrastructure. For deadletter jobs, that means agreeing in writing what gets monitored, how failed jobs are triaged and how fixes are verified before any job is moved back for execution.
How we scope
- Inventory which process and case definitions use async steps, timers and external calls that can exhaust retries
- Agree the deadletter alerting thresholds, owners and escalation path before the engagement starts
- Confirm which databases, deployment targets and Flowable versions are in the written scope
How we build
- Build queries and reports against the deadletter table to group failures by exception and definition
- Write runbooks for reading exceptions, fixing data or code and re-queuing jobs in a controlled order
- Test recovery steps in a non-production environment that matches your production database and engine version
How we hand over
- Document alert rules, dashboards, triage steps and the decision criteria for retrying or abandoning a job
- Walk your operations team through a supervised recovery so they run the next one themselves
- Record configuration, scripts and ownership in your own repositories and wiki
How we support
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts
- Ongoing support, upgrades and continuous testing are available as a retainer
- Ask any partner for the support hours, response expectations and escalation contacts in the written quote
