Flowable and Apache Kafka integration with event registry

How Flowable and Kafka integration with the event registry works

The supported path for driving Flowable processes and cases from Kafka is the event registry. Instead of writing custom consumer code that calls the engine, you declare what arrives and where it goes: an event definition describes the payload fields (string, integer, double and boolean), and a channel definition connects that event to a broker destination. Correlation parameters then tell the engine which running process or case instance an incoming event belongs to. Kafka is one of three broker options, alongside RabbitMQ and JMS, so the choice of broker is a platform decision your messaging team should make before the models are designed. The engine itself is licensed under Apache 2.0, so the event registry carries no licence cost of its own, but the Kafka cluster, its hosting and its operation are separate costs you will need to plan.

Event definitions and channel definitions are the two artifacts your team will version and review. The event definition is the contract between the messaging team and the process team: field names, types and the correlation parameters must match what producers actually publish. The channel definition ties that contract to a Kafka topic and decides whether the channel starts a new instance or triggers an existing subscription. Agree the topic naming, partition key and message schema with the producing teams before modeling starts, because a change to the payload later means changing both the producer and the event definition. Confirm with your Kafka administrators which topics and consumer groups the Flowable application may use, since Kafka access control is configured on the cluster side.

Correlation is where most integration defects appear. The correlation parameter values in an incoming message are matched to the event subscription of one case or process instance, so a missing or ambiguous key can start a duplicate instance or drop a message. Test correlation with realistic duplicate and out-of-order messages, and decide in advance what happens when no subscription matches. Outbound events from send event tasks need the same discipline: define which payload fields leave the engine, who consumes them, and how consumers handle redelivery. Before go-live, confirm the exact Flowable version and the Kafka client settings your environment supports, since both move on their own release schedules.

What you need before connecting Kafka to Flowable

A Kafka-driven Flowable integration depends on four things lining up: the Flowable engine modules and their Java and Spring baseline, a Kafka release that your brokers and clients can run, and the network and credentials that connect them. Confirm each one before you write a channel definition, because a mismatch here is far harder to fix once event flows are in production.

Flowable version and engine modules

Decide which Flowable release and which engines you need: BPMN, CMMN, DMN and the event registry. The event registry is part of the version 7 and later engine line, so confirm it is present in the release you pick. Flowable 8.0.0 was released on February 27, 2026, and its upgrade moves to Spring Framework 7 and Spring Boot 4. Confirm the Spring Boot 4 requirement against your stack before you commit to 8.0.0, since it carries no backward compatibility for Spring Boot 3.

Java and Spring baseline

Check that your application runtime matches the Flowable release. Version 7 moved to Java 17, Spring Boot 3 and Jakarta EE, and the Flowable 8.0.0 notes describe Jackson 3 as the default JSON library, with a Jackson 2 compatibility layer. If your event payloads are serialized by existing Jackson 2 code, test the compatibility layer before upgrading. JUnit 3 and 4 support has also been removed, which affects your test suite.

Kafka release line

Choose one of the supported Kafka releases: 4.3.1, 4.2.2 or 4.1.2. Kafka 4.0 and later run only in KRaft mode, so a ZooKeeper-based cluster must migrate to KRaft before upgrading. Clusters older than 3.3.x should move to 3.9.x first. Brokers, Kafka Connect and tools need Java 17, while clients and Kafka Streams applications need at least Java 11, so confirm the Java version of each Flowable host that acts as a producer or consumer.

Network paths and listeners

Map which Flowable nodes must reach which Kafka brokers, and on which listener ports. Flowable nodes often run in a different network zone from the broker cluster, so confirm routing and firewall rules for each bootstrap address. Kafka allows clusters to mix secured and unsecured listeners, so agree with the messaging team which listener Flowable uses. Advertised listener addresses must be reachable from every client, or consumers will connect and then fail.

TLS and SASL credentials

Agree the authentication method before any code is written. Kafka supports SSL for encryption and SASL for authentication, with GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER as documented mechanisms. Obtain the truststore, the client credentials and the rotation process from your messaging team, and store them outside the model definitions. Ask whether the Flowable service account needs read access to certain topics only, and whether it may publish to others.

Topic ACLs and authorization

Confirm the authorization model before the first test. In KRaft clusters, the StandardAuthorizer stores ACLs in the cluster metadata log, and any resource without an ACL is limited to super users unless allow.everyone.if.no.acl.found is set to true. Without an explicit ACL for the Flowable principal, consumers will fail silently or be rejected. Request ACLs for each inbound topic (read, with group access) and each outbound topic (write) at the start, so the integration does not stall during testing.

Configure the event registry and Kafka channel in sequence

This sequence takes a Kafka topic from request to working Flowable wiring, so that a test message reaches the engine, matches the intended process or case instance, and can trigger an outbound event from a model. Each step ends with a check you can run before moving on.

Agree the topic and consumer group with the messaging team
Before any Flowable configuration, the Kafka owners and the Flowable team agree on the topic name, partition count, retention, the consumer group that Flowable will join, and which principal may produce to and consume from the topic. Access is controlled through authorization rules, and the Kafka authorization documentation notes that in KRaft clusters resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true, so confirm the ACLs exist before testing. Done when the topic exists in a test cluster, the consumer group is named in writing, and a test producer can write to it with the agreed principal. Confirm the exact Kafka client and broker versions with your platform team, since they determine the client libraries you bundle.
Define the event definition payload fields
Create an event definition in the event registry that lists the fields the business needs from each message. Flowable supports string, integer, double and boolean payload fields, so map each field in the Kafka message (for example, an order identifier as a string and an amount as a double) to one of those types and name the fields the same way on both sides. Agree with the producing team which fields are mandatory and what happens when a field is missing or has the wrong type. Done when the event definition deploys without errors and a sample JSON payload matches every field in the definition.
Create the channel definition for the Kafka topic
Add a channel definition that binds the event definition to the Kafka topic and sets whether the channel is inbound (consuming messages that start processes or trigger boundary events) or outbound (sent by send event tasks). The channel points at the topic and consumer group agreed in the first step, so a mismatch here is the most common reason a message never arrives. Deploy the channel alongside the event definition and check the engine logs for a clean startup. Done when the engine reports the channel as deployed and the consumer group appears as joined on the broker.
Declare correlation parameters on the event definition
Correlation parameters tell the engine which running case or process instance an incoming event belongs to. Flowable matches an event to a subscription when the correlation parameter values in the message equal the values stored with that instance, so choose a stable business key such as an order or case identifier and make sure the process or case sets the same value when it subscribes. Document the parameter names and their types in the interface contract with the producer. Done when a test message carrying a known identifier is matched to the subscription that was created with that identifier, and a message with an unknown identifier produces no match.
Wire the inbound start or subscription in the model
In the BPMN or CMMN model, attach the inbound event to the place where it should act: a start event that creates a new process instance, or a boundary or intermediate event that waits for a message on a running instance. Map the event payload fields to process or case variables so that downstream tasks can read them. Deploy the model and confirm the definition version is the one you expect. Done when a test message that starts a new instance creates exactly one instance with the mapped variables populated, and a second message with the same identifier is handled as your design specifies.
Add a send event task for outbound messages
Where the model must notify another system, add a send event task that references an outbound channel and maps process variables to the event fields. Decide whether the event should be sent when the task is reached or only after the surrounding transaction commits, because a message sent before a rollback can reach consumers for work that never persisted. Confirm the outbound channel's principal has write access to the topic. Done when a test run of the model produces one message on the topic with the expected field values, and the consuming team confirms they received it.
Verify the full round trip with test messages
Run an end-to-end test with a producer you control: send a valid message that starts an instance, send a message that matches a waiting subscription, send a malformed message, and send a message with an unknown correlation value. For each, record what the engine did, including the instance it matched, the variables it set, and any error in the logs. Keep these tests in the project so they run again after every model or channel change. Done when each case behaves as the design says and the results are saved with the deployment record.
Confirm the broker version and retention before go-live
Before going live, check the Kafka version your platform runs against the supported releases on the Kafka downloads page and the upgrade notes for your target release, and confirm the client library you use is compatible with that broker. Confirm retention and consumer offset behavior with the messaging team, so that a restart or replay does not start duplicate instances. Done when the messaging team and the Flowable team have signed off the versions, retention and replay behavior in writing, and the same test suite passes on the production-like environment.

How correlation parameters route a Kafka event to the right instance

Correlation parameters decide which running case or process instance receives a Kafka event. In the Flowable event registry, each event definition declares its payload fields and its correlation parameters, and the values of those parameters are matched against the event subscription of a specific instance. The practical design task is therefore to pick the business identifier that both sides can derive from the event, such as an order ID or a claim number, and to name it the same way in the event definition, the channel definition and the producing service. If the correlation value changes during the life of the instance, the event will not reach it. Check this with the teams that own the source systems before the model is built, not after the first missed message.

Kafka adds a second layer to the design through record keys and partitions. Kafka only orders records within a single partition, and a record's key determines its partition in the default partitioner, so a producer that keys events by the same business identifier used for correlation keeps related events in one partition. That is the ordering Kafka's partitioning provides, and it should not be extended in the design to cover events spread across topics or keyed differently. Duplicates are a separate concern: Kafka delivery to consumers is at least once in common configurations, so the process side should tolerate a repeated event for an instance that has already moved past the waiting point. Confirm with the messaging team which producer idempotence and consumer offset settings are in use, and document how an event that arrives with no matching subscription is handled.

Agree the naming and ordering rules with the messaging team in writing before build starts. Settle the topic naming convention, the key format and its encoding, whether the key is the correlation value or a derived string, and which events must share a topic so that they land in one partition. Ask the team to state which events may arrive out of order across producers and how the model should respond, for example by waiting for a later event or by ignoring a stale one. Decide who owns the schema of the payload fields and how changes to them are versioned. These decisions shape the event definitions more than any engine setting, and they are cheaper to change in a design review than in a running system.

Mapping Kafka message fields to event payloads and correlation parameters

This table maps each part of a Kafka record (key, headers, value and topic) to the Flowable event definition field, correlation use and the owner who should confirm it. Read it as a checklist to agree with the messaging team before any event definition or channel is built. The main takeaway is that correlation values should come from fields the producer guarantees are present, stable and unique to the case or process instance.

Kafka source	Flowable event field	Correlation use	Owner to confirm
Record key	Payload string field, usually a business identifier	Strong candidate for a correlation parameter if it is the same for every event about one instance	Producing team and messaging team, including partitioning impact
Record value field, business identifier (for example an order ID)	String payload field declared in the event definition	Correlation parameter that matches the event subscription of one process or case instance	Producing service owner
Record value field, status or amount	Integer, double or boolean payload field	Usually not a correlation value; used as a condition or variable on the instance	Producing service owner and process owner
Message header, event type or schema version	Field the channel definition reads or a string payload field	Not a correlation value; used to filter or route to the correct event definition	Messaging team and platform owner
Message header, correlation ID set by the producer	String payload field if the producer copies it into the value	Correlation only if values are unique per instance and reused across retries	Messaging team, confirm header names and retry behavior
Topic name and partition	Not part of the event payload	No direct correlation; affects ordering of events for one key	Messaging team and Kafka platform owner
Timestamp or sequence number	Optional string or integer payload field	Usually not for correlation; helps detect duplicates or out-of-order delivery	Messaging team and process owner
Null key or missing field	Field absent from the payload	Events without a correlation value may not reach any instance, so agree a rejection or dead-letter path	Messaging team and operations owner

Choosing between Kafka, RabbitMQ and JMS for process events

Choose Kafka when your organization already runs Kafka for other systems and the platform team can offer topics to process applications. Kafka keeps events in a retained log, so consumers can read the same stream independently and re-read earlier records. That fits cases where several systems need the same business events, where a new consumer must catch up on history, or where operations teams already monitor consumer lag. The cost is that process applications join a shared platform with its own capacity, retention and access rules.

Multi-consumer fan-out: Flowable can start or correlate processes from a topic while analytics, fraud or audit consumers read the same events.
Replay: Retained records let you reprocess events after a fix, subject to the retention period set for each topic.
Shared platform rules: Topic naming, partition counts and ACLs are set by the platform team, not by the process team.
Ordering scope: Ordering holds within a partition, so the partition key must match the correlation key you plan to use.
Operations fit: Consumer group lag and broker metrics are usually already watched by the team that runs Kafka.

Confirm with the Kafka owners which topics you may use, the retention and partition keys they will set, and which consumer group naming and ACL model applies to Flowable.

Securing the Kafka connection and the event registry

A Kafka-driven Flowable deployment has two security surfaces: the connection between the event registry and the brokers, and the access rules Kafka applies to the topics that carry process events. Each card below covers one decision in that path, from encryption and client identity to ACLs and the authorization checks Flowable leaves to your application.

Encrypt broker traffic with TLS

Kafka encrypts traffic between clients, brokers and tools with SSL/TLS. Security is optional in Kafka, so a cluster can run unsecured or mix secured and unsecured listeners. Agree with the messaging team which listeners the Flowable event channels use, and confirm that any plaintext listener is disabled for production topics. Ask who issues and rotates the broker and client certificates.

Choose a SASL mechanism for clients

Kafka authenticates clients, brokers and tools with SSL or SASL. The documented SASL mechanisms are GSSAPI (Kerberos), PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Pick one that matches your identity estate, and confirm with the Kafka team whether the Flowable engine's Kafka client configuration supports the mechanism you choose. PLAIN sends credentials in a form that depends entirely on TLS.

Give the Flowable client its own identity

Create a dedicated Kafka principal for the process application so that its reads and writes can be traced and revoked independently. Avoid sharing the credentials of a broker administrator or another application. Store the client secret in your secrets manager and inject it at startup, not in the process definition or the Git repository that holds the models.

Restrict topics with KRaft ACLs

In KRaft clusters the StandardAuthorizer stores ACLs in the cluster metadata log. Resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Keep that setting false for production, grant the Flowable principal read or write access only to the event and channel topics it uses, and confirm that consumer group permissions are included.

Enforce authorization in the application

Flowable's event registry routes a message to a process or case instance using correlation parameters, but it does not decide whether the sender should be allowed to trigger that instance. Broker ACLs control who can publish to a topic, not which business object a message may affect. Decide where those checks live, such as a validation step in the channel handling or the application service, and document them for the security review.

Validate payloads before correlation

Event definitions declare typed payload fields, which gives the registry a structure to check against. Treat any value used as a correlation parameter as untrusted input until it is validated, because a wrong or forged identifier can attach an event to another customer's case. Ask the team to test malformed and oversized messages on a non-production topic, and to log rejected events for investigation.

Why events are not starting or correlating processes

This table lists the failures teams hit most often when Kafka messages do not start a process or reach the right running instance, with the usual cause and fix for each. Work through the rows in order, since a broken channel definition or Kafka ACL hides every failure below it. Confirm the exact error text in your own Flowable and Kafka logs before changing configuration.

Failure	Likely cause	Fix
No matching correlation value	The event payload field used for correlation does not carry the same value as the process or case variable, or the value is empty, differently cased or formatted differently.	Compare the event definition's correlation parameter with a sample message from the topic. Agree the key format with the producing team and test with one known instance.
Payload type mismatch	A field declared as integer, double or boolean in the event definition receives a string or a differently typed value from the producer, so the event fails to map.	Align the event definition types with the producer's schema. Send a test message with each declared type and check the event log for the rejected field.
Consumer group offset issues	A new consumer group starts at the latest offset and skips messages already on the topic, or an existing group has committed offsets past the events you expected to process.	Decide the starting position before go-live. Inspect committed offsets for the group and reset them only after the team agrees which messages to replay, since replays can start duplicate work.
ACL denials	KRaft clusters restrict resources without ACLs to super users unless allow.everyone.if.no.acl.found is true, so the Flowable client principal may be unable to read the topic or join its group.	Grant read on the topic and the consumer group to the Flowable principal, and write on any outbound topic. Confirm the rules with the Kafka team rather than enabling open access.
TLS trust chain errors	The client does not trust the broker certificate chain, the truststore path or password is wrong, or the certificate hostname does not match the listener address.	Load the internal CA into the Flowable service truststore and check the broker listener matches the advertised hostname. Test the connection with a Kafka client tool before debugging the engine.
Duplicate starts after retries	A failed handler or lost acknowledgement causes the message to be redelivered, and the inbound channel starts a second process instance for the same business event.	Use a business key or correlation value that is unique per event and check for an existing running instance before starting. Confirm how the Flowable version you run handles redelivery and design the process to be idempotent.
Misconfigured channel definitions	The channel points to the wrong topic name, uses the wrong event definition key, or is deployed to a different tenant or environment than the one the listener expects.	Compare the channel definition with the topic name and event key in the deployed model. Confirm the deployment version that is active and log the channel key at startup.

Questions buyers ask before connecting Flowable to Kafka

What drives the cost of connecting Flowable to Kafka?

The cost depends mostly on how many event flows you need, how complex the correlation logic is, and how much of the messaging setup already exists. Each inbound channel that starts a process or triggers a boundary event, and each outbound channel from a send event task, adds design, build and test work. Kafka itself is licensed under Apache-2.0 with no paid edition from the project, so the spend is on the people and infrastructure around it. Ask any provider for a written scope that lists each channel, each event definition, the correlation parameters, the environments to test in, and who owns the Kafka cluster and topic setup.

How long does a Flowable and Kafka integration usually take?

Duration follows the number of channels, the state of the Kafka environment and how quickly the messaging team can review payloads and topic decisions. Teams that already run Kafka in a stable state usually spend most of their time on event payload design, correlation rules and failure testing rather than on connection setup. No duration figure is supplied for this work, so ask the provider to tie each milestone in the written scope to a specific channel or flow and to name the dependencies on your messaging and platform teams.

What does the client team need to provide?

The work needs a named owner for the Kafka cluster, access to create or review topics, and agreement on the message schema for each event. Your team should also supply sample messages from production or a realistic test source, the correlation key that ties each event to a running case or process instance, and the Flowable engine version and deployment model. Confirm early which side owns authentication to the brokers, because Kafka supports SSL and SASL mechanisms and the choice affects both the Flowable configuration and the cluster setup.

Can an internal team build this without outside help?

Yes, an internal team with Java, Flowable and Kafka experience can build this, and the event registry is defined through event and channel definitions rather than custom protocol code. The harder parts are usually correlation design, idempotent handling of repeated messages, and agreeing payload contracts across teams. If your engineers have not run Kafka consumers in production or do not own the broker operations, a partner can cover the gaps, and the written scope should name which steps your team keeps.

Which Kafka version should we target?

Target a supported Kafka release, and confirm the choice with the Kafka project before you start. The Kafka downloads page lists 4.3.1, 4.2.2 and 4.1.2 as supported releases, and earlier versions are archived. Kafka 4.0 supports only KRaft mode and requires Java 17 for brokers, so a ZooKeeper-based cluster must migrate to KRaft first. Clusters older than 3.3.x are recommended to move to 3.9.x before 4.x. Check which Java and client versions your Flowable runtime uses, since Kafka clients require at least Java 11.

What security controls should we agree on before the build?

Agree on transport encryption, client authentication and topic-level access before any event flows go live. Kafka supports SSL/TLS for traffic and SASL mechanisms including GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER. Security is optional in Kafka, so confirm that no unsecured listener is exposed to the Flowable services. In KRaft clusters, the StandardAuthorizer stores ACLs in the metadata log, and resources without ACLs are restricted to super users unless allow.everyone.if.no.acl.found is set to true. Confirm the ACL model with your security team before the integration uses those topics.

What should a written scope for this integration include?

A written scope should list every event and channel definition, its payload fields and types, the correlation parameters, and the process or case it starts or reaches. It should name the topics, partitions and consumer groups in use, the authentication method, the test environments, and the failure handling for duplicate or late messages. Add the acceptance tests, the monitoring you expect, the deployment steps, and the responsibilities for each team. Ask the provider to confirm the Flowable and Kafka versions being targeted and to state what is excluded, such as broker operations or changes to upstream producers.

How DeployOpen scopes and delivers Flowable and Kafka integrations

DeployOpen plans, builds, documents and supports Flowable and Apache Kafka integration with the event registry inside your own infrastructure. Each engagement starts with a written, fixed scope that names the event and channel definitions, correlation parameters, broker settings and handover material agreed with your messaging and process teams before work begins.

How we scope

Inventory which Kafka topics, RabbitMQ queues or JMS destinations will start or correlate Flowable processes and cases
Agree correlation parameters per model, with the business keys that identify the right instance for each event
Confirm with the messaging team the broker version, authentication method, topic ownership and access control model

How we build

Define event and channel definitions with payload fields, then wire inbound channels to start events or boundary events
Configure outbound send event tasks so model state changes publish events in the format your consumers expect
Test message replay, duplicate delivery and out-of-order events against representative correlation scenarios in a staging environment

How we hand over

Deliver event and channel definitions, correlation mappings and broker configuration as versioned files in your repository
Document the message contract, failure paths, dead-letter handling and the runbook for restarting consumers
Walk your platform and messaging teams through the test results and the open decisions that remain with them

How we support

Offer ongoing support, upgrades and continuous testing for the integration as a retainer, scoped separately from the build
Plan upgrades of Flowable and Kafka client libraries against the versions your broker and runtime support
Review message volumes, consumer lag and failed correlations with your operations team when the retainer covers them

Flowable and Apache Kafka integration with the event registry