Flowable logging with MDC for traceable engines

How Flowable logging with mapped diagnostic context traces one process instance

A stuck or failed process instance rarely fails in one place. A single instance can pass through engine calls, a service task delegate, an async job picked up by a different node, and application code that writes its own log lines. By default those lines share no common key, so an engineer chasing one instance has to guess which entries belong to it. SLF4J mapped diagnostic context (MDC) solves this by attaching key-value pairs to the logging thread so that every line written while that thread works carries the same fields. For Flowable, the fields a team usually wants on every line are the process definition ID, the process instance ID and the execution ID, with the business key added where the application uses one. The work that matters is putting those values into MDC at the points where the engine runs your code, and clearing them afterward so that pooled threads do not leak one instance's context into another.

The time to design this is before an incident, because MDC cannot be added retroactively to logs that were never written with it. Decide which identifiers go on every line, which log level applies to the engine packages and which to your delegates, and how the log pattern prints the fields. Then test the design with a deliberately failing instance in a non-production environment and confirm that one search returns the full path, including the async job that resumed it. Confirm the exact MDC key names, the logger package names and how context propagates to async executor threads against the Flowable version you run. The Flowable Spring Boot documentation and the Flowable advanced documentation describe the engine, Spring Boot and async executor behavior, but the supplied material does not establish the MDC key names or logger packages, so check them in your own environment rather than assuming them.

Plan for the cases where context is lost. Jobs executed by the async executor run on engine-managed threads, so context set in a web request will not follow them unless you set it again in the job path. Retries and jobs moved to the deadletter table are executed later, possibly by another node, which means the identifiers must be stable across retries and visible in the stored job record as well as in logs. Give the operations team a short runbook that maps a process instance ID to its log search, and review it whenever you upgrade Flowable, since logger names and internal call paths can change between releases.

Which logging approach fits which Flowable deployment

This table compares four ways to add process, instance and execution context to Flowable logs, and shows where each one applies across Spring Boot starters, plain Java SE or servlet embedding, and the Flowable REST application. Read each row for the deployment you run, then weigh the main benefit against the code or operations cost before choosing a combination.

Approach	Where it applies	Main benefit	Main cost
Application-level MDC servlet filter or interceptor	Servlet containers and Spring web applications that call Flowable services in the request thread	Sets request and correlation IDs once for every log line in the request, including engine calls made inside it	Does not cover async jobs or timers, which run on executor threads outside the request, so those logs need a separate approach
MDC set inside engine event listeners or delegates	Plain Java SE, servlet and Spring embedding, where you own the event listener or delegate classes	Adds process definition, instance and execution IDs from the event context to logs written by your own code	Each delegate or listener must set and clear MDC correctly, and engine internals log without these fields unless wrapped
Per-package log levels for Flowable engine packages	All deployments, configured through the logging framework your runtime uses	Raises detail for engine packages during an incident without flooding application logs at debug level	Verbose engine logs add volume and storage cost, and debug output can include variable values that need review for data protection
Log shipper or collector reading container or file output	Containers, Kubernetes and VM deployments that already forward logs to a central store	Keeps Flowable logs with other platform logs and lets teams query by fields that the shipper parses from the output	Fields only appear in search if the logging pattern writes them, so shipper parsing must match the pattern used by each engine
Flowable REST application's own logs	Deployments that run the Flowable REST application as a separate service over HTTP	Shows API request and response activity from the service boundary, separate from the calling application	Engine and API logs are separate from caller logs, so correlating them depends on passing a shared request ID through HTTP headers
Spring Boot starter logging configuration	Spring Boot applications using the Flowable starters	Uses familiar Spring logging properties to set levels and patterns for Flowable packages in one place	Confirm the logger names and pattern keys against the starter version you deploy, since they change across major releases

Which identifiers belong in MDC for Flowable processes

Put the process definition ID, process instance ID and execution ID into the MDC first, because together they let you move from a single log line to the whole path a process took. The process instance ID is the anchor: it ties together every task, timer, job and variable change for one case or order, and it is the value support staff will paste into a search. The execution ID narrows the view to the branch of the process that was active, which matters when parallel gateways or subprocesses produce interleaved log lines for the same instance. The process definition ID records which version of the model ran, so when a deployment changes behavior you can separate failures from the old model from those of the new one. Confirm in the version you run which of these identifiers the engine exposes at each event point, such as job execution, task completion and delegate invocation, since the values available inside a delegate can differ from those available in an event listener. The Flowable docs for your release are the place to check this before you write the MDC setter code.

Business keys and tenant identifiers are useful, but they need more discipline. A business key such as an order number or customer reference lets a service desk find the instance without knowing Flowable internals, and a tenant identifier separates data in multi-tenant deployments. Both can also be high-cardinality, meaning nearly every value is unique, and that raises index size and query cost in log platforms that index every field. Each unique value added as an indexed label or field multiplies storage and slows aggregation, while the same value kept as searchable free text costs less. Decide with your platform team which fields are indexed, which are stored only, and whether personal data such as customer names must stay out of the MDC altogether, a question that also touches the data protection controls auditors test. Document that decision in the logging standard before the first production deployment, so each application uses the same field names.

Setting up MDC and engine log levels step by step

This sequence adds Flowable process, instance and execution identifiers to every log line, sets engine log levels deliberately, and proves with a known test instance that one instance can be traced end to end.

Agree the field list and naming convention
Operations, the application team and the platform owner agree which diagnostic fields every engine and application log line will carry, such as process definition key, process instance ID, execution ID, business key and job ID, along with the exact field names used in the log pattern and the log store. Write the list down with one owner who approves changes, because a field renamed later breaks dashboards and saved searches. This step is done when the list is signed off and the same names appear in the logging configuration template that the team will use.
Add a filter that sets and clears MDC
The application team adds a servlet filter for HTTP requests and a wrapper or listener for async jobs and timers that writes the agreed fields into the SLF4J mapped diagnostic context at the start of the unit of work and removes them in a finally block. Clearing the context matters because thread pools reuse threads, so stale values would otherwise appear on unrelated lines. The check is a unit test that runs two sequential requests on one thread and confirms the second request sees no values from the first.
Attach IDs at engine event points
Using the Flowable engine's event listener or the points where the application starts, completes or fails a process instance, the team writes the process instance ID, process definition key and execution ID into the context as they become known, and updates the context when a sub-process or call activity creates a new instance. Confirm the exact event types and lifecycle hooks against the Flowable documentation for the version in use, since the available hooks are version-specific. The check is a log line from a test run that shows the instance ID appearing on engine messages that occur after the instance is created.
Set log levels for Flowable engine packages
The platform team adds explicit logger entries for the Flowable engine packages in the logging configuration, starting with INFO for the engine and DEBUG only for the packages being investigated, such as the job executor or the process engine's command execution. Keep debug output scoped and time-limited, because verbose engine logging adds volume to the log store and can expose variable values. The check is a startup run that prints the effective level for each configured logger, confirming that the Flowable packages are not inheriting a root level that is too quiet or too noisy.
Include MDC fields in the log pattern
The platform team updates the log pattern in the logging framework so that every appender writes the agreed MDC keys, including console output in containers and the file or shipped format sent to the central log store. Missing keys should print as an empty value rather than failing the pattern. The check is a local run that writes one line from a request and one from a job, and both lines show the field names from the agreed list in the same positions.
Deploy to a test environment
The team deploys the filter, listeners, logging configuration and log pattern to a test environment that matches production's logging path, including the log shipper and the index or query layer. Confirm that the deployment uses the same configuration artifact promoted from the development branch, so that the test results describe what production will run. The check is that the test environment's log store shows the new fields as searchable attributes on engine and application lines.
Trigger a process with a known business key
The tester starts a process instance with a business key that is unique to the test, such as an identifiable test order number, and lets it run through at least one service task, one asynchronous job and one user task. Record the business key, the process instance ID and the time window of the run so the results can be compared against the log store. The check is that the instance ID returned by the start call matches the ID that the engine logs for the same run.
Confirm every line for the instance carries the same fields
Search the log store for the instance ID recorded in the previous step, and confirm that the results include lines from the HTTP request, the engine, the job executor and the application code. Then search for the same time window without the instance ID filter and confirm that no lines from other instances appear in the results for that instance's fields. Keep the query and its result as the acceptance evidence for the change. The sequence is complete when both searches agree and any missing lines are traced to a path that lacks the filter or listener.

Why MDC often breaks in async jobs and timers

MDC values set during an HTTP request usually do not reach the code that runs a Flowable async job or timer. The Flowable async executor runs timers and asynchronous jobs from its own thread pool, and it coordinates nodes through lock owners and lock expiration times stored in the database. A job is therefore picked up by whichever thread and whichever node wins the lock, not necessarily the thread that started the process instance. The MDC map is thread-local, so the processInstanceId and request correlation values your API filter placed on the request thread are absent when the job executes. A job recovered from an expired lock after a node crash is an even clearer case, because the original request, and the JVM that held its context, no longer exists.

The practical fix is to store the correlation values in the data the job carries, then restore them in the job's own logic. Process variables, a dedicated correlation variable set at start, or a value written into the job's configuration can hold a request ID or tenant ID. A delegate, listener or job-related hook can then copy those values into MDC at entry and clear them on exit, so each execution sets its own context instead of inheriting one. Treat this as illustrative until verified: the exact hook points, and whether the values survive retries and timer re-creation, depend on the supported Flowable version and your executor settings. Confirm them against the Flowable advanced documentation and by reproducing a failed job, a retry and a recovered expired lock in a test environment.

Keeping sensitive process data out of log lines

Flowable engines write process state, variables and identifiers into log output whenever a delegate, listener or script logs them. The cards below cover where sensitive data usually leaks into logs, and the controls that keep the MDC fields useful for tracing without turning log stores into a second copy of your process data.

Process variables and form data in logs

Logging a whole variable map or a submitted form from a delegate or listener copies customer names, account numbers and free-text answers into every log line. Engine debug output can also reveal variable values when the engine packages run at debug or trace. Restrict logging to an explicit allow-list of named fields, and review each delegate's log statements before go-live. Check whether any logger prints the variable scope object.

Personal data in business keys

Business keys are often set to a customer number, national ID or email address, and the engine copies them into process instance records and, depending on your MDC setup, into log fields. A value that is personal data in one system becomes searchable in every log store that receives it. Use an opaque reference generated for the instance and keep the mapping in the application database. Confirm which business key values reach the MDC mapping.

Tokens and credentials logged by delegates

Service tasks and HTTP calls often pass bearer tokens, API keys or connection strings through variables or expressions, and a stack trace or request dump can print them. Once logged, these credentials persist in log archives long after rotation. Mask Authorization headers and secret-named variables in the logging layer, and keep credentials in a secrets store that the delegate reads at runtime rather than in process variables.

Log retention outliving the data policy

Log stores commonly keep data for longer than the retention period set for the process records they describe. A debug-level trace kept for months can hold data that your data protection policy requires you to delete after a shorter period. Set shorter retention for logs from engine packages at debug or trace, document the retention for each log category, and confirm deletion with your log platform before relying on it.

Log viewers with broader access than the engine

Flowable's own access controls govern who can see a process instance, but the log store may let any engineer or support analyst read every line, including variables that the engine would hide from them. Give log viewers separate roles scoped by application or environment, and audit who queries production logs. Verify that the log platform's roles match the authorization model used by the process applications.

Correlating engine logs with application logs by runtime

With the Flowable Spring Boot starters, the engines run in the same JVM as your application, so the MDC filter normally belongs in your own web layer. Register a servlet filter (for example, a OncePerRequestFilter) that puts a correlation ID into the MDC for each request and clears it at the end. The engine itself does not know your request, so add the process definition, instance and execution IDs where your code has them, such as inside delegates. Levels usually come from application.yaml, and the pattern that prints MDC values comes from your logging configuration. Follow the Flowable Spring Boot docs for the starter names and the /actuator/flowable endpoint.

Filter location: a component in your web module, registered so it runs before any Flowable REST controllers you expose.
Log configuration: engine levels under logging.level in application.yaml, and an %X{...} placeholder in the log pattern so MDC values appear.
Async work: timers and async jobs run on the async executor's thread pool, outside the request thread, so copy the identifiers into the MDC in the code those jobs call.
Clusters: every node needs the same pattern, and the node name should be in it, because a job may be picked up by a different node than the one that created it.
Versions: Flowable 8.0.0 moved to Spring Boot 4 and Spring Framework 7 with no backward compatibility for Spring Boot 3, so confirm the starter line matches your Boot version before you change the filter.

Check that a log line from a timer-fired job carries the same process instance ID as the request that started the process, on every node.

When you embed the engine as a JAR in a Java SE program or in a WAR deployed to Tomcat or Jetty, the MDC setup depends on whether a servlet pipeline exists. In a servlet container, add the filter in web.xml or programmatically during startup, so it wraps every request. In plain Java SE, there is no pipeline, so wrap the entry points that start or continue process work in a helper that sets and clears the MDC. Logging configuration usually sits in the application's own logback.xml or Log4j2 file, but the container's shared libraries can supply a competing binding, so check which framework actually writes the output. The Flowable getting started guide lists the embedding options.

Filter location: web.xml or a ServletContextListener in the WAR, or a wrapper around engine calls in Java SE.
Log configuration: one logging file on the classpath, with the MDC placeholder in the pattern, and no duplicate SLF4J bindings in the container's shared libraries.
Thread handling: MDC is stored per thread, so clear it in a finally block, and copy values into the MDC inside any executor you start yourself.
Clusters: each Tomcat or Jetty node writes its own log files, so ship them to one store and keep the node name in the pattern. Each node also runs its own async executor against the shared database.
Versions: Flowable 7.0.0 moved to Java 17, Spring Boot 3 and Jakarta EE, so a filter written against the older javax servlet API must be migrated. Confirm that your Tomcat or Jetty release supports the Jakarta Servlet API used by your Flowable line.

Confirm that a stack trace from a failed job shows the same correlation ID as the request that triggered it, because that is the first thing an operator will search for.

Limits to confirm before relying on MDC for investigations

MDC fields only help an investigation if they are present, consistent and readable on every node when an incident happens. The cards below list what to confirm in your own Flowable version, logging stack and cluster before your team depends on them, and DeployOpen does not set fixed numbers for these limits, so measure each one in a staging environment that matches production.

Which MDC keys Flowable sets

Confirm which MDC keys your chosen Flowable version populates itself, such as process definition, process instance and execution identifiers, and which ones your application must add. Keys are not guaranteed to be stable across major versions, so check the keys against the release notes for the exact version you run. Write the list down and use it in log parsing rules and dashboards. Flowable 8.0.0 was released on February 27, 2026, and its upgrade changes the surrounding stack, so verify again after any upgrade.

MDC propagation across threads

MDC is stored per thread, so values set on a request thread are lost when work moves to the async executor thread pool, a worker thread or a new executor. Test each path where Flowable jobs, timers and external calls run, and confirm the log framework copies or propagates the context into pooled threads. Without this, the lines that matter most during a failed job will carry no process identifiers.

Debug log volume in production

Raising the level of engine packages to debug can produce far more output than info, and the volume grows with process throughput and the number of variables logged. Measure the bytes and lines per hour at the level you plan to run during an incident, and confirm your log pipeline, storage and retention can absorb that spike. Set a plan for turning the level back down, and confirm who can change it without a redeploy.

Overhead of extra log fields

Each added field increases the size of every log line, and high-cardinality values such as instance identifiers can strain indexing in tools that index every field. Compare query speed and ingestion cost with and without the extra fields on a representative load test. Also confirm that the logging framework's context handling does not add measurable latency to job execution under peak concurrency.

Merging logs across cluster nodes

In a clustered deployment, the async executor can move a job from one node to another after a lock expires, so one process instance can produce log lines on several nodes. Confirm that each node writes a node identifier, that clocks are synchronized, and that your log shipper keeps the original timestamps. Test a merged timeline by following one instance through a forced node failure.

Secrets and personal data in context

Business keys, variable values or user identifiers placed in MDC can end up in shared log stores with wider access than the process data itself. Confirm which values your application puts into context, who can read the log store, and how long lines are retained. Align the field list with your data protection review before it goes live.

What buyers ask about Flowable log correlation and support

What drives the effort of Flowable log correlation work?

The effort follows the size and shape of the engine estate. The main drivers are the number of process definitions and case models, the number of service tasks, delegates and script tasks that write logs, how heavily the platform uses async jobs and timers, how many log sinks and existing dashboards must be changed, and the data protection rules that limit which variables and business keys may appear in log lines. A team with a few definitions and one log platform has a much smaller job than one with many deployed versions across several services. Ask the vendor to list each of these drivers against your environment in the written scope.

Can our own team add MDC correlation to Flowable without a partner?

Yes, a team can do this alone if it has a Java engineer who knows Spring Boot or the plain Java engine setup, and access to the application logs in a non-production environment. The core work is configuring the SLF4J MDC fields for process definition, instance and execution IDs, setting log levels for the engine packages, and checking that the fields survive asynchronous job execution and thread hand-offs. Teams that lack someone who has tested logging behavior under async jobs often find the gaps only in production, so a short review by an outside engineer can be useful even when the build stays in-house.

How long does a Flowable logging engagement take?

The duration depends on scope, so you should ask for a written plan with phases, dependencies and milestones rather than a single figure. Timing moves with the number of services that emit Flowable logs, how many log platforms must be configured, whether async jobs and timers need specific test cases, and how quickly your environments and reviewers respond. A fixed-scope agreement records the deliverables and dates in writing before work starts, which makes the schedule easier to compare across proposals.

What does DeployOpen need from our team to start?

Plan on providing a non-production environment that runs the same Flowable version and deployment model as production, read access to the log platform used by the engine and the surrounding applications, a set of sample process instances that cover the common paths and at least one failing path, and a named contact who can approve log fields and levels. A test plan that states which correlation fields must be found in a search, and within what steps, keeps the verification objective. Confirm with your security team which variables and identifiers may leave the application before samples are shared.

What does a fixed-scope Flowable logging engagement cover?

A fixed-scope engagement covers the deliverables agreed in writing before work starts. For logging, these typically include a review of the current logging setup, a defined set of MDC fields and their population points, a log level plan for the engine packages, configuration changes in the application and in the log platform, and test evidence that a single business case can be traced across engine and application logs. It does not include changes to business process design unless the scope says so. Ask for the list of exclusions as carefully as the list of deliverables.

Which data protection rules affect what we can log from Flowable?

Business keys, variable values and user identifiers often count as personal or regulated data, so they should not be written to logs by default. Before adding fields to MDC, agree with your privacy and security owners which identifiers are allowed, how long log retention lasts, who can read the logs, and whether a pseudonymous reference can replace a customer identifier. These rules change the design more than the engine settings do, so they belong in the early scope rather than a late review.

Does DeployOpen guarantee that our logs will pass an audit?

DeployOpen does not issue audit reports, attestations or certifications, and this work does not promise any audit result. DeployOpen can prepare the logging configuration, the evidence of how it was tested and the documentation of decisions, which your independent auditor can then review. Ask your auditor early which log fields and retention periods they expect to see for the controls in scope.

What support is available after the logging work is delivered?

Support after delivery is handled through a separate retainer, which can cover ongoing review of log output after Flowable upgrades, changes to process definitions and new log platform rules. The retainer scope, response expectations and review cadence are agreed in writing, so you can compare them with running the same checks in-house. Make sure the handover includes the MDC field list, the log level settings per package and the test cases used to verify correlation.

How DeployOpen works on this

DeployOpen plans and implements Flowable logging with mapped diagnostic context inside your own infrastructure, from the logging design to the correlation fields your teams query during an incident. Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts, and ongoing support is available as a retainer.

How we scope

Inventory the Flowable engines, Spring or Java SE hosts and application log destinations that need correlated output.
Define the MDC fields for process definition, process instance and execution IDs, and which ones are safe to log.
Agree in writing which logger packages get which levels and which correlation checks the engagement will test.

How we build

Configure the logging framework so engine packages and application packages share one correlation scheme.
Test that process and execution identifiers appear on engine and application log lines for a sample instance.
Verify that async executor threads and delegate calls keep context, and that log volume stays within the levels you approved.

How we hand over

Document the MDC field names, logger configuration and the query patterns your operations team should use.
Walk through the runbook for raising a log level temporarily and restoring it after an investigation.
Confirm with your own platform and security owners where log data is stored, retained and who can read it.

How we support

Review logging changes alongside Flowable version upgrades and changes to your Spring Boot or logging libraries.
Help investigate incidents where correlation gaps slow down diagnosis, and recommend configuration fixes.
Offer ongoing support and periodic reviews as a retainer, scoped in writing with your team.

Flowable logging with mapped diagnostic context