How Fineract business events reach Kafka
Core banking teams often need downstream systems to react to client and loan events without polling the REST API. Apache Fineract can publish external business events, such as ClientCreated, to Apache Kafka when this feature is enabled. The method is Fineract's built-in external event support, not a separate connector that sits beside the core. The feature is disabled by default, so the first build step is a deliberate decision to switch it on and to choose which events leave the system.
The Fineract README reports that Kafka support was tested with PLAINTEXT brokers and with AWS MSK using IAM authentication. Nothing beyond those two configurations is confirmed by the project, so a TLS-secured, SASL-authenticated or self-managed cluster on your own infrastructure is a configuration to verify rather than assume. Confirm with your own test cluster which listener security, authentication mechanism and broker version you will run, and check the Fineract documentation for the event settings that match your version before you plan the rollout.
Plan the broker connection as part of the event design, not as an afterthought. Decide who owns the topics, which network path Fineract uses to reach the brokers, and how credentials are stored and rotated. Because the event framework writes events within the business transaction, a broker outage is a question of how Fineract retries and recovers, so test that behavior on the actual cluster under the load you expect before go-live.
What you need before enabling Fineract events
Event publishing touches four systems: the Fineract release, its database, the Kafka cluster and the network between them. Each card below is a decision or check to settle with your team before anyone changes a Fineract configuration key. Confirm every key name and default in the current Fineract documentation before you rely on it.
Fineract release line
Start from 1.15.0, the current stable release on the Fineract website, and confirm that your deployment target matches it. Quarterly releases mean the version you pick today will need an upgrade plan within months, so record which release your event configuration was tested against. Check the release notes for any change to event publishing before you upgrade Fineract or Kafka.
PostgreSQL as the database
The Fineract README names PostgreSQL as the recommended database, and MariaDB and MySQL are deprecated under FSIP-9. Plan your PostgreSQL version, high availability and backups as part of the same build, since business events are saved within the business transaction. Confirm with your database team how the event records are sized and retained.
Kafka cluster on a supported release
Use a KRaft-mode Kafka cluster on a supported release: 4.3.1, 4.2.2 or 4.1.2. Fineract's tested configurations were PLAINTEXT brokers and AWS MSK with IAM authentication, so confirm your broker setup against those before you assume it works. Ask who owns topic creation, partition counts and retention for the Fineract event topics.
Network paths and TLS or SASL
Confirm the network route from the Fineract write instance to every Kafka broker, including firewall rules, DNS and any private endpoints. Decide whether listeners use TLS, SASL or both, and which SASL mechanism your brokers support, then confirm the matching client settings with your security team. Do not assume the test configurations cover your authentication choice.
Tenant-aware configuration
Fineract is multi-tenant, and API requests carry the Fineract-Platform-TenantId header, with tenant connections read from a tenants database. Decide whether events from each tenant go to separate topics or share one, and make sure consumers can tell tenants apart. Confirm with the Fineract documentation how event publishing is enabled per tenant before you plan topic naming.
Instance roles and topic access
Only the write instance runs Liquibase migrations, so decide which instance publishes events and confirm it is the one that holds the Kafka credentials. Give that instance only the Kafka permissions it needs, and confirm in the Kafka authorization documentation how ACLs apply when no ACL exists for a resource. Document who approves changes to these grants.
Configuring event publishing in sequence
This sequence takes a Fineract instance from an agreed event scope to a verified test event that a consumer reads from a Kafka topic, with each step closing on a check before the next one starts.
Agree the event scope
Before any configuration, the bank's product owner, integration lead and data governance contact agree which business events leave Fineract, such as ClientCreated, and which consuming systems subscribe to each one. Fineract event publishing is disabled by default, so this list defines what you will enable, what topics you will need and which fields downstream systems depend on. The step is done when the scope is written down, signed off by the owners of each consuming system, and stored with the integration design.
Enable external events in a non-production environment
The platform engineer turns on external event publishing in a non-production Fineract instance, using the configuration that the Fineract documentation describes for enabling Kafka output. Keep the rest of the environment unchanged so that any unexpected behavior can be traced to this one setting. The step is done when the instance starts cleanly, logs show the event producer initializing, and no events are yet reaching a production consumer.
Connect Fineract to the broker with chosen authentication
The platform engineer points Fineract at the non-production Kafka bootstrap servers and configures the authentication the cluster requires. Kafka supports SSL and several SASL mechanisms, so confirm with the cluster owner which one applies and whether it is secured. Fineract's own tests covered PLAINTEXT brokers and AWS MSK with IAM authentication, so confirm any other mechanism against the Fineract documentation before relying on it. The step is done when Fineract connects, and the broker logs show an authenticated producer session.
Create topics with planned partition counts
The Kafka platform owner creates the topics from the agreed scope, with partition counts and retention set in the planning phase rather than left at defaults. Partition count affects ordering scope and consumer parallelism, so record which key determines the partition for each event type. The step is done when the topic description shows the planned partition count and retention, and the cluster accepts a test write from the Fineract account with the agreed permissions only.
Confirm Avro schema registration
Fineract's event framework uses Avro schemas, so the integration lead confirms how each event schema is registered and which compatibility rule governs later changes. Check the schema for the event type against the Fineract documentation, and confirm that the consuming team can deserialize it with the same schema version. The step is done when the schema for ClientCreated is available to both producer and consumer, and a deserialization test on a sample payload succeeds without manual edits.
Emit a test ClientCreated event
In the non-production tenant, the QA lead creates one test client through the Fineract REST API, using a clearly marked synthetic record. This produces a ClientCreated event that Fineract saves with the business transaction and then sends to the topic. The step is done when the event appears on the topic with the expected key, headers and payload fields, and the test client's identifiers match the record in Fineract.
Verify the consumer reads the record and the offset advances
A consumer in its own consumer group reads the test topic and must return the same fields that were written, including the tenant context the event carries. Check that the committed offset for that group moves from the starting value to the new position after processing. Repeat the test once to confirm that a redelivered event is handled without a duplicate effect. The step is done when the record matches, the offset advances as expected, and the results are recorded before any production enablement is considered.
Ordering and at-least-once delivery explained
Fineract's event framework saves each event in the same business transaction that changes the data, then sends it to Kafka in order with at-least-once delivery. Because the event is written with the business change, a committed loan or client update should not be lost silently. The trade-off is that the same event can arrive more than once, for example when a publish is retried after a timeout whose outcome is unknown. Your consumers must be designed for that from the start, not treated as an edge case.
Kafka only orders records within a single partition of a topic. Fineract's ordering is therefore meaningful per topic and partition, and the key used to route each event decides which events share a partition. Confirm in the Fineract documentation which key is used for each event type, whether ordering holds across partitions or after a broker failure, and how retries interact with ordering. Do not assume a global sequence across all business events. Test these cases in your own environment with broker failures and producer retries before relying on them.
A consumer that tolerates redelivery records the identifiers of events it has already applied, and checks that record before each side effect such as posting an accounting entry or calling a downstream system. The processing and the record of it should commit together in the same database transaction where possible. Confirm which identifier Fineract places on each event (the event schema defines the fields, and the Avro definitions in the project repository are the place to check them) and whether it is stable across retries. If it is not stable, you need a deterministic key of your own, agreed with the team that owns the ledger.
Mapping Fineract events to consumer data
This table maps a Fineract business event such as ClientCreated to the Avro fields, Kafka keys, topic naming and consumer target tables a team needs to plan. Read each row as a design decision to settle before consumers are written, and confirm the field names against the published Avro schema for your Fineract version.
| Mapping element | Planned value or approach | Confirm before building |
|---|---|---|
| Event type | ClientCreated, one event type per business action such as a client, loan or savings change | Exact event names and the list of enabled events in your Fineract build |
| Avro schema | One schema per event type, with fields read by name rather than by position | Field names, nullability and types in the published Avro schema for your release |
| Topic name | One topic per event type or per domain, named by your naming standard | Whether Fineract or your configuration sets topic names, and how they can be overridden |
| Kafka message key | A business identifier such as the client identifier, so events for one client stay in order on one partition | Which field Fineract uses as the key, if any, and whether it is set per event |
| Tenant identifier | Stored as a column in every target table, so one platform can serve several tenants safely | Where the tenant identifier appears in the event payload or headers in your deployment |
| Client identifier | Primary business key for customer-level tables and the join key for loan and savings events | Whether the identifier is stable across events and unique within a tenant |
| Event timestamp | Stored as the time the business transaction occurred, kept separate from the time the consumer received the message | Which timestamp field the event carries, its time zone and its precision |
| Consumer target table | A staging table keyed by tenant, client and event identifiers, then a current-state table derived from it | Table design is set by your data team, so confirm column types against the schema |
| Event identifier | A unique event or aggregate identifier stored with each row, used to skip duplicates on redelivery | Whether the event carries a unique identifier that stays the same on retries |
Designing idempotent consumers by target system
A duplicate posting in a ledger changes balances and must be reversed by hand, so this consumer must apply each business event at most once. At-least-once delivery means a consumer restart or partition rebalance can redeliver an event, so the consumer needs a stable identity for each event and a record of what it has already posted. Confirm the event identifier fields in the Avro schema for your Fineract version before you design the key, as described in the Fineract platform documentation.
- Idempotency key: the event's unique identifier from the payload, combined with the posting type, so a replay cannot create a second journal entry.
- Deduplication store: a processed-events table in the same database and transaction as the journal entries, so the marker and the posting commit together or not at all.
- Retry handling: retry transient failures such as database unavailability with backoff, and commit the Kafka offset only after the transaction succeeds.
- Poison messages: send events that fail validation, such as an account mapping that does not exist, to a dead-letter topic with the original partition and offset, and hold the affected postings for review.
Check that replaying a sample of events into a test environment produces no second journal entry and that ledger totals match the balances Fineract reports for the same period.
Reporting stores tolerate delay better than ledgers, but duplicates still distort counts, sums and balances in every report built on them. The design question is whether the store can overwrite a row by key, and that depends on the store and the table engine you choose. Some stores provide keyed upserts, others only append, so confirm the sink's write semantics before you commit to a deduplication approach.
- Idempotency key: the event identifier together with the entity identifier and event version, so a later change to the same loan or client is stored as a separate version and not lost.
- Deduplication store: where the sink supports keyed upserts or merges, use the key as the deduplication mechanism. Where it does not, load into a staging table and deduplicate on the key before the reporting tables refresh.
- Retry handling: retry whole batches with the same keys, so a batch that was written before a failure is not counted twice. Confirm how the sink handles bulk inserts that partly succeed.
- Poison messages: move records that fail schema or type checks to a quarantine table with the raw payload and offset, and show the quarantine count on the report so gaps stay visible.
Check that a replay into staging produces identical row counts and totals, and that the quarantine table is reviewed on a set schedule.
A message sent to a customer cannot be recalled, so the risk in this consumer is a duplicate notification or a duplicate CRM record. Retries make this harder, because a timeout does not show whether the provider accepted the request. Whether the provider or CRM accepts a client-supplied idempotency key varies by product, so confirm this with the vendor before you rely on it.
- Idempotency key: the event identifier together with the notification type and recipient, so one event produces one message per channel.
- Deduplication store: a sent-notifications table written after the provider accepts a request, storing the provider's message identifier when one is returned. If the provider offers an idempotency header, use it and confirm how long it retains keys.
- Retry handling: retry only on timeouts and server errors. A timeout leaves the outcome unknown, so check the sent-notifications table before each retry.
- Poison messages: hold events with missing contact details or with a consent flag that is not set in the current record for operator review. Do not pass them to the provider.
- CRM updates: upsert on an external identifier that the CRM supports for that object, and confirm which objects support it.
Check the duplicate-send count in a provider sandbox while the consumer is restarted during a burst of events, and confirm that the count stays at zero.
Using Kafka for COB remote partitioning
Close of business (COB) processing in Fineract is batch work, and the instance model determines where that work can run. The cards cover the documented instance roles, the migration constraint, the open question of whether Kafka can carry partitioned batch work, and the duplicate and stalled-work risks to test before a design is approved.
Read, write and batch instances
Fineract's documentation describes three instance types set through environment variables: read, write and batch. For high load, it recommends one write instance, one batch instance and several read instances on database read replicas. Write instances handle changes, batch instances run scheduled work and read instances serve queries. Confirm the exact variable names and role behavior for your target version before drawing the topology, since any Kafka consumer would be a new workload alongside these three.
Liquibase runs on write instances
According to the Fineract documentation, Liquibase database migrations run only on write instances. On Kubernetes, they can run separately using the liquibase-only Spring profile. If you add batch or consumer pods, they must not start a second migration path, and the order of schema changes during a release must be written down. Ask which profile and startup order the project recommends for your version, and test an upgrade against a staging copy of your database.
Whether Kafka can carry partitioning
The supplied documentation covers Fineract's Kafka events and its instance model, but does not establish that Kafka can carry COB remote partitioning. Treat this as an open question. Read the batch and COB documentation for the version you run, search the apache/fineract source for how partitioned jobs are dispatched and tracked, and confirm whether a supported path exists. Without that confirmation, Kafka can still serve event consumption, but it should not be assumed to dispatch batch partitions.
Duplicate batch work
Fineract's event framework delivers events at least once, so a consumer can receive the same message again after a retry or a rebalance. If a redelivered message restarts partition work, the same accounts could be processed twice in one business day. Give each batch task a stable key, such as business date and partition ID, and record completion in the same transaction as the result. Then force a redelivery in a non-production environment and inspect the resulting balances.
Stalled batch work
A partition that no consumer picks up, or a consumer that stops after a rebalance, leaves part of the business day unprocessed. The scheduler may report the job as finished while some partitions never ran. Define a completion check that compares finished partitions with planned partitions for the business date. Alert on consumer lag and on tasks that exceed a time limit, and decide in advance who reruns a partition and how that rerun avoids the duplicate risk. Set thresholds from your own load tests.
Verification before design approval
Before approving a Kafka-based COB design, obtain written evidence for five points: the instance settings and Liquibase profile for your Fineract version, whether the project supports partitioned batch dispatch through Kafka, how redelivery is handled, how completion is measured, and the rollback path if a batch run stops halfway. Ask your implementation partner and the Fineract community to document their answers, and keep test results with your change records so auditors can review batch controls.
Common failures when Fineract events stop flowing
This table lists the failures most often seen when Fineract business events do not reach a Kafka topic or reach it in an unusable form. Read each row as a symptom, its likely cause, and the check to run first. The main takeaway is that most failures sit at the configuration boundary between Fineract, the broker and the consumer, so check those boundaries before debugging application code.
| Failure | Likely cause | Check to run |
|---|---|---|
| No events appear on the topic | External event publishing is disabled by default in Fineract | Confirm the Kafka event setting is enabled on the write instance, then create a test client and confirm a ClientCreated event is written |
| Publisher receives authorization denied | The Fineract client principal has no write ACL on the target topic in a KRaft cluster | Review broker logs for authorization errors and list ACLs for the Fineract principal on the topic |
| Producer fails with unknown topic | Topic was not created in advance and broker auto-creation is turned off | List topics on the cluster and confirm the event topic exists with the planned partition count and replication factor |
| Consumer cannot deserialize messages | Avro schema on the consumer differs from the schema Fineract publishes | Compare the registered schema with the consumer's expected version and test-deserialize a sample record |
| Duplicate records downstream | At-least-once delivery re-sends events after a retry or timeout | Check whether the consumer stores processed event identifiers and whether writes are idempotent |
| Consumer lag keeps growing | Consumer is slower than event volume, or a poison message blocks its partition | Compare committed offsets with log end offsets per partition and inspect the consumer for repeated errors |
| Events arrive out of order for one entity | Events for different entities land in different partitions and the consumer assumes global order | Confirm the message key matches the entity identifier and that the consumer handles versions per entity |
| Events missing after a database rollback | Event rows are saved within the business transaction, so a rolled-back change should emit no event | Confirm the originating business change committed, then check the event record in the Fineract database |
Which regulations shape event handling in a bank
The first question is whether the Fineract event pipeline touches cardholder data at all. PCI DSS applies to entities that store, process or transmit cardholder data or sensitive authentication data, and also to systems that could affect the security of the cardholder data environment. If Fineract events carry card numbers, card tokens, or payment details that flow through a card-related service, the Kafka cluster, Avro schemas, connectors and consumers that carry those events may fall inside the assessed scope. The scope decision comes before the topic design: a pipeline that publishes only account identifiers and balances is scoped differently from one that forwards payment instrument data. Confirm applicability with your compliance team and your Qualified Security Assessor, if you use one, before the first topic is created. Note that the PCI SSC does not enforce compliance itself, so the payment brands and your acquirer or assessor decide who must validate.
If the bank is in DORA scope, the pipeline becomes part of the ICT service landscape whenever it supports a critical or important function. Article 28(3) requires a register of information covering all contractual arrangements for ICT services, so the Kafka platform, any managed or vendor-supplied component, and any implementation partner should appear there with the functions they support. Article 28(8) requires documented, tested exit strategies and transition plans for those services. For an event pipeline, that means being able to rebuild consumers, replay retained events, or move topics to another cluster without losing business records. Article 30(2) requires each ICT services contract to state where services are provided and data processed, to give advance notice of changes, and to cover data return on insolvency or termination. Check these terms with any partner, and confirm with counsel whether your entity falls within the 20 financial entity types covered, since applicability depends on the entity.
Questions teams ask before wiring Fineract to Kafka
What drives the effort and cost of wiring Fineract events to Kafka?
The main drivers are how many event types you publish, how many downstream consumers depend on them, and how much of the Kafka and identity setup already exists. Fineract event publishing is disabled by default, so enabling it, choosing which business events leave the system, and agreeing the Avro schemas are the first decisions. Consumer work, error handling and monitoring add more. Cost is set by the written scope: each item listed (event selection, schema review, consumer build, test environment, runbook) is a line you can price and compare. Ask any quote to separate one-time build work from recurring support, and to state who operates the Kafka cluster.
How long does a Fineract to Kafka integration take?
Duration depends on scope rather than on a fixed number. A narrow integration that publishes a few event types to one consumer, on an existing cluster, takes much less time than one that covers loan and savings events, several consumers, and authenticated brokers in more than one environment. The schedule should be fixed in the written scope, with milestones for schema agreement, test environment validation, consumer verification and production cutover. Ask any partner to show which milestones depend on your teams, such as network approvals, Kafka topic creation or identity provider changes, because those often set the pace.
What must our team provide before the integration starts?
You need to provide several things before build work can begin:
- Kafka access, including bootstrap addresses, topic creation rights, and the network path from the Fineract instances to the brokers
- Credentials for the chosen authentication method, and the identity owner who issues and rotates them
- Agreement on Avro schemas and their compatibility expectations, with a named owner for each schema
- Named consumer owners who will read the topics, handle failures and confirm outputs
- A test environment and access to Fineract's configuration for enabling external events
Can our internal team build this integration without a partner?
Yes, if your team already runs Kafka, understands Avro, and has Fineract operators who can change its configuration and deployment. The work is not unusual in shape, but the Fineract README notes that production deployments can be complex and time-consuming and that the project does not provide a comprehensive production deployment guide. Check three capabilities before deciding: someone who can operate the Kafka cluster, someone who can own Fineract configuration across read, write and batch instances, and someone who can verify consumers under failure. If any of these are missing, a fixed-scope engagement can cover the gap in writing.
How are duplicate events handled in a Fineract Kafka integration?
Fineract's event framework delivers events at least once, so a consumer can receive the same event more than once, including after a retry or restart. The consumer must therefore be idempotent. A common approach is to store a unique event identifier in the consumer's own database and skip events already recorded, in the same transaction as the side effect where possible. Confirm with the Fineract documentation which identifiers each event carries, and test redelivery during design. Avoid assuming Kafka itself removes duplicates in this path, since that depends on the producer and consumer configuration you choose.
What changes when moving from PLAINTEXT to authenticated MSK or self-hosted brokers?
The Fineract README says external events were tested with PLAINTEXT brokers and AWS MSK with IAM authentication. Moving to authenticated brokers adds work on both sides. Fineract needs the client security settings, credentials or IAM roles, and certificates that match your broker configuration. Brokers need listeners that require TLS and a SASL mechanism, and ACLs that allow each principal to write or read the right topics. Kafka's documented SASL mechanisms include GSSAPI, PLAIN, SCRAM and OAUTHBEARER. Confirm the exact client properties with the Kafka security documentation and your broker version, and test each environment separately before production.
Does Fineract event publishing support compliance evidence on its own?
No. Publishing events does not make a deployment compliant with PCI DSS, DORA or other frameworks. The integration can support evidence such as documented access to topics, encrypted transport, retention settings and change records, but the assessment itself is run by auditors or certification bodies. If cardholder data or payment flows are in scope, confirm with your assessor how the Kafka cluster, Fineract instances and consumers fall into the cardholder data environment. DeployOpen prepares systems, controls and evidence for assessments but does not issue audit reports or attestations.
How DeployOpen scopes and delivers this integration
DeployOpen approaches the Fineract-to-Kafka wiring as a scoped engagement inside your own infrastructure, starting with the events you need and the consumers that will act on them. The work covers Fineract event configuration, the Kafka topics and Avro schemas, consumer idempotency and operational runbooks, with each deliverable agreed in writing before build begins.
How we scope
- Inventory which Fineract business events your consumers need, such as ClientCreated, and which must stay internal.
- Confirm your Kafka platform, broker authentication and the Fineract version your environment will run.
- Agree in writing the topic layout, schema ownership, delivery expectations and the systems that will consume events.
How we build
- Enable and configure external event publishing in a non-production Fineract environment, then test each event type end to end.
- Build idempotent consumers that deduplicate on event identifiers, since delivery is at-least-once and duplicates must be expected.
- Test ordering assumptions, broker outages and consumer restarts, and record the results against the agreed acceptance criteria.
How we hand over
- Deliver configuration, schema files, consumer code and topic definitions in your version control, with documented ownership.
- Provide runbooks for replaying events, handling poison messages and reconciling consumer state with Fineract records.
- Walk your platform, security and operations staff through the design so they can run and change the integration without us.
How we support
- Offer ongoing support, upgrades and continuous testing as a retainer, separate from the fixed-scope build.
- Plan Fineract and Kafka upgrades as scoped work, retesting schemas and consumers against each new release.
- Ask your team to confirm retention, access control and compliance evidence needs, since DeployOpen prepares evidence but does not issue audit reports.
