Why Apache Fineract Close of Business batch jobs decide your nightly window
The nightly Close of Business run is the job most likely to decide whether a Fineract deployment fits inside its operating window, and it tends to become a problem gradually. A platform that is live or close to go-live may run the Close of Business Day Spring Batch job against a small book of accounts without trouble. As the book grows, the same job can start to consume the hours that staff, reconciliation teams and downstream systems expect to have. The project's README notes that this job can run for several hours on deployments with millions of accounts, so the question to answer early is not whether it will finish, but when it will finish as volumes rise. Fineract separates its workload by instance type. Read, write and batch instances are set through environment variables, and the Fineract documentation recommends one write instance, one batch instance and multiple read instances on database read replicas for high load. The Close of Business job belongs on the batch instance. Running it on a write instance, or sharing one instance between interactive traffic and batch processing, puts nightly work in direct competition with customer-facing requests. Confirm the instance layout in your own environment before go-live, including which instance executes the job and how that instance is scaled. What makes this harder than it looks is the project's own warning. The README states that a production deployment can be complex, costly and time-consuming, and that the project does not provide a comprehensive production deployment guide. Teams therefore have to work out for themselves how many batch resources to allocate, how the job behaves under real account volumes, how failures are restarted, and how the schedule fits around other maintenance such as database work and Liquibase migrations. None of these answers come pre-written, so they need to be tested against your own data sizes rather than assumed from a demonstration environment. The remaining decisions are where the window is actually won or lost. Remote partitioning across worker nodes, coordinated through ActiveMQ or Kafka, is the documented route for spreading the work, but it adds broker infrastructure, worker sizing and monitoring to operate. Each of those components can become the bottleneck or the point of failure on a night when the job is already late. Ask who will own the batch schedule, what alert fires when the job overruns its expected completion, and what the team does if the run has not finished before the business day starts. The practical starting point is a measured baseline. Ask your implementation partner or internal team to run the Close of Business job against a dataset that matches your projected account count, record elapsed time per stage, and identify where the job spends most of its time. That baseline shows whether the current design will hold through the growth you expect, and it gives you a defensible basis for choosing between a larger batch instance, partitioned workers, or a change to the schedule.
Components and settings involved in the batch run
The Close of Business Day batch job depends on how Fineract is split into instance types, which database and replicas those instances use, and whether a message broker carries work to worker nodes. Each component below is a setting to decide before the first nightly run, and each one changes how long the job takes and what happens when it fails.
Read, write and batch instance types
Fineract selects its role for each running instance through environment variables, so one codebase can serve read traffic, accept writes or execute batch work. The Fineract documentation recommends one write instance, one batch instance and multiple read instances for high load. Check that exactly one instance is configured to run the Close of Business job and that no read-only instance can start it.
PostgreSQL primary database
The Fineract README names PostgreSQL as the recommended database, and the official Docker image ships only its JDBC driver. The batch job writes ledger, loan and savings state to this primary, so its sizing, connection limits and maintenance window directly shape run time. Confirm the version your platform team supports and how backups are scheduled around the nightly run.
Read replicas for reporting and API traffic
Read instances are meant to query PostgreSQL read replicas, which keeps reporting and client API reads away from the primary while the batch writes. Replication lag matters here: a read instance that is behind the primary can show balances that the batch has not yet updated. Ask the replication setup team how lag is measured and what the front end does when lag is high.
Spring Batch Close of Business job
The Close of Business Day job is a Spring Batch process that the README says can run for several hours on deployments with millions of accounts. Its step structure, chunk sizes, restart behavior and job repository tables determine whether a failed night can resume or must restart. Confirm how job execution history is retained and who is alerted when a step fails.
ActiveMQ or Kafka broker for remote partitioning
Remote partitioning spreads the batch across worker nodes that receive their partitions through ActiveMQ or Kafka. The broker is therefore a runtime dependency of the batch, not only an event channel, and an unavailable broker can stall the whole run. Decide which broker the operations team already runs, how it is secured, and how queued partitions are monitored and retried.
Tenants database and tenant identifiers
Fineract is multi-tenant, and each API request carries a Fineract-Platform-TenantId header that selects a connection configured in the tenants database. The batch must run against every tenant that the deployment hosts, so a missing or misconfigured tenant entry can leave one institution's accounts unprocessed. Confirm the tenant list, the schema for each tenant and the order in which tenants are processed.
How remote partitioning across workers with ActiveMQ or Kafka works
The Close of Business Day Spring Batch job can split its work into partitions that run on separate worker nodes. According to the Fineract README, those nodes are coordinated through either ActiveMQ or Kafka. Before your team plans a nightly run around this option, it helps to understand the roles involved, what each node must share with the others, and where the messaging layer sits in the critical path. Treat this as the model to confirm against the README and the version you intend to run, since the details of message handling belong to the Spring Batch remote partitioning pattern and to the Fineract configuration for your release.
In this model, one node acts as the coordinator, which is the Spring Batch job execution that divides the account population into partitions and dispatches each one as a message. Worker nodes receive those messages, execute their assigned partition against the database, and report status back through the broker. The coordinator then waits for all partitions to finish before the job completes. Ask your team to confirm which node in your topology is designated as the coordinator, how it is started, and what happens to the job if the coordinator restarts while workers are still processing.
Several things must be identical across every node that takes part. Application configuration, including the Spring profiles, batch settings and broker connection details, must match so that all nodes interpret messages and partition ranges the same way. Tenant connections must also match: each node has to resolve the same tenants database and reach the same tenant schemas, or a partition may silently process the wrong data or fail. Database access needs the same rights and the same target PostgreSQL primary, with network paths and credentials verified from every worker. Configuration drift between nodes is one of the more common causes of failed or inconsistent nightly runs, so build a check that compares these settings before each run.
The messaging layer becomes a dependency of the nightly run. If the ActiveMQ or Kafka cluster is unavailable, partitions cannot be dispatched or acknowledged, and the Close of Business job may stall or fail. Plan for broker capacity, authentication, topic or queue retention, monitoring of consumer lag or queue depth, and a documented procedure for restarting a failed run. Confirm with your platform team who owns the broker, what its availability target is, and how it is patched, because a broker maintenance window can now overlap with your batch window. Test failure scenarios, including a lost worker and a broker restart, in a non-production environment before go-live.
Choosing the batch topology: single node, partitioned with ActiveMQ, or partitioned with Kafka
This table compares three ways to run the Close of Business Day batch job: one node, remote partitions coordinated through ActiveMQ, and remote partitions coordinated through Kafka. Read each row as a question to answer for your environment, since the operational cost of the broker and the network paths between nodes often decide the topology more than job speed does. The main takeaway is that partitioning adds moving parts, so it pays off only when a single node cannot finish the nightly run in your close window, and a broker you already operate usually beats a new one.
| Decision factor | Single batch node | Partitioned with ActiveMQ | Partitioned with Kafka |
|---|---|---|---|
| When it fits | Nightly run finishes inside the close window with headroom at your current account count and growth plan | Run no longer fits the window and your team already runs ActiveMQ or another JMS broker | Run no longer fits the window and your team already runs Kafka for other workloads |
| Broker to run and monitor | None added for batch | Only if one must be added; otherwise the existing JMS broker carries partition messages | Only if one must be added; otherwise the existing Kafka cluster carries partition messages |
| Skills your team needs | Fineract operations and PostgreSQL tuning | JMS broker administration plus Fineract operations | Kafka administration (topics, consumer groups, retention) plus Fineract operations |
| Failure modes to plan for | A single long job that fails late must be restarted and may overrun the window | Lost or stuck partition messages, broker unavailability, and workers stopping mid-partition | Consumer group rebalances, topic misconfiguration, and broker unavailability during a run |
| Network paths between nodes | None beyond the database and the batch node | Batch manager and each worker need reachable broker and database endpoints inside the same network zone | Batch manager and each worker need reachable broker endpoints and database access, often across more hosts |
| Interaction with event publishing | Event publishing is disabled by default; if you enable it, confirm the batch run does not flood downstream consumers | Event publishing and partition coordination can share a broker, so confirm topic and queue separation before enabling events | Event publishing and partition coordination can share a cluster, so confirm topic naming, retention and consumer permissions before enabling events |
| Reuse existing broker or add one | Not applicable | Reuse if the JMS broker has capacity, monitoring and an owner; adding one only for batch adds a system to patch and back up | Reuse if the cluster has capacity, monitoring and an owner; adding one only for batch adds a system to patch and back up |
| What to confirm with your team | Measured run time at your account volume and whether a single node meets your close deadline | Broker version support, HA design, and which team owns partition queue failures | Cluster version support, HA design, and which team owns topic and consumer group failures |
Build sequence for a partitioned Close of Business setup
This sequence moves the Close of Business Day batch job from a single node to a partitioned run across worker nodes, with a measured baseline and a reconciled result at each stage. The client keeps the same end-of-day totals and can show that the change did not alter them.
Baseline the single-node run time
Restore a copy of production data to a staging environment sized like production, then run the Close of Business Day job on a single node with normal load on the read path. Record the start and end times, the per-step timings from the Spring Batch execution history, and the database resource use during the run. Your team and the implementation partner should agree on this baseline in writing. The step is done when you have a repeatable single-node run and a saved set of end-of-day totals to compare against later.
Separate batch instances from read and write
Configure one instance as the batch instance and keep the write and read instances separate, using the instance-type environment variables documented for your Fineract release. Confirm that only the write instance runs Liquibase migrations, and that the read instances point at database read replicas where your design uses them. The batch instance should be the only process that starts the Close of Business job. The step is done when a test deployment starts each instance type and the job can be launched only from the batch instance.
Provision broker and worker nodes
Provision the message broker, either ActiveMQ or Kafka, and the worker nodes that will execute partitions of the job. Keep the worker nodes in the same network zone as the database so partition traffic stays on private addresses. The platform team confirms broker authentication, TLS and network rules before any job runs. The step is done when each worker node starts, connects to the broker, and reports as available to the coordinator in the staging environment.
Configure worker tenant and database access
Configure each worker to resolve tenant connections from the tenants database and to reach the operational database with its own least-privilege credentials. Verify that every worker can read and write only the tenant schemas it is assigned, and that no worker holds a default development login. Run a connection test for each tenant on each worker. The step is done when every worker passes the tenant connection test and an access review confirms that its credentials match the approved list.
Run the partitioned job in staging
Launch the Close of Business Day job from the batch instance in staging with remote partitioning enabled across the workers. Watch the partition status, the broker queues and the database locks while the job runs, and record any failed or restarted partitions. Confirm that a failed partition can be restarted without rerunning completed partitions. The step is done when the job completes with all partitions in a completed state and the run log has no unresolved errors.
Verify against the single-node run
Compare the partitioned run with the single-node baseline from the first step. Match end-of-day totals per tenant, including balances, accrued interest, fees and posted journal entries, and check that account-level outcomes are identical. Compare the total run time and the per-step timings to see where the partitioned setup helps and where it does not. The step is done when totals match for every tenant and the timing results are written up for the go or no-go decision.
What drives the effort and cost of batch scaling and how to contain it
This table lists the factors that lengthen the Close of Business run and raise the work needed to partition it, with a containment step for each. Read it as a checklist for scoping: the first column names the driver, and the other columns explain why it matters and how to keep it under control.
| Driver | Why it changes effort or cost | How to keep it down |
|---|---|---|
| Account and loan volume | More accounts mean more rows for each nightly step to process, so the run gets longer and partitioning and worker sizing need more attention. | Baseline the run against a realistic account count early, then size partitions and workers from measured duration instead of assumptions. |
| Number of institutions on one platform | Each tenant adds its own batch workload to the same database and worker pool, so one tenant's run can slow another's window. | Agree per-tenant run windows and load limits up front, and test the combined nightly load with all tenants active. |
| Broker choice and existing skills | ActiveMQ or Kafka each add a broker to install, secure, monitor and patch, and the team must be able to operate it under load. | Reuse a broker your organization already runs and supports, and confirm which broker your team can operate before choosing one. |
| Environment count | Every extra environment (development, test, performance, production) needs its own workers, broker, database and batch configuration, and each must be kept in step. | Keep environments on the same configuration and deployment method, and limit the number of full-scale environments to what testing requires. |
| Data volume for baseline testing | Realistic test data is needed to measure run time and find bottlenecks, and copying or generating large datasets takes time and storage. | Generate synthetic data sized to production patterns, reuse it across test cycles, and record timings so each change can be compared. |
| Network and security between nodes | Workers, the broker and the database exchange traffic that must be encrypted and allowed through firewalls, and misconfigured links can stall partitions. | Define the required connections in advance, use encrypted links between nodes, and test failure and reconnection paths before go-live. |
| Customization of batch logic | Custom steps added to the Close of Business job change its duration, its failure modes and what must be retested after each Fineract upgrade. | Keep custom batch steps in separate modules, document their inputs and runtime, and include them in every regression and performance test. |
How the batch setup differs by deployment model
On a single node, one Fineract instance serves API traffic, runs Liquibase migrations and executes the Close of Business Day job in the same process. The batch job has no separate instance to hand work to, so it shares CPU, memory and database connections with users and integrations. Remote partitioning across worker nodes is only relevant if you add workers, so the main planning question is how one process will handle the job alongside daytime load.
- Migrations: Liquibase runs on startup, so each upgrade is a restart that includes a schema change. Plan a tested backup and a rollback path before every release.
- Scheduling: the job runs inside the same instance, so choose the run window using measured API load from a production-sized test environment.
- Capacity: size the heap and the database connection pool for the batch job and peak API traffic together.
- Failure handling: if the instance stops during a run, confirm how your version of the job resumes from its last completed step.
- Monitoring: job duration and API latency come from the same process, so a single set of dashboards can show both.
Check whether one process can finish the job inside the window your operations team can allow, and ask the implementation partner to measure that with realistic account volumes.
The Fineract documentation describes read, write and batch instance types set through environment variables. It recommends one write instance, one batch instance and multiple read instances on database read replicas for high load. Separating these roles keeps batch work from competing with user-facing reads and writes, but it adds coordination around migrations, database connections and failover.
- Write instance: this is the only instance that runs Liquibase migrations, so it must be upgraded and started before the others.
- Batch instance: it runs the Close of Business job. Confirm in the documentation that the batch instance writes to the primary database and not to a replica.
- Read instances: they serve read-heavy API calls and reporting from replicas. Measure replication lag, because a lagging replica can return balances that are behind the primary.
- Failover: define what happens to a running batch job if the database primary fails over, and test that scenario before go-live.
- Routing: the load balancer should send each endpoint only to the instance role that is allowed to serve it, so no read path reaches the batch instance.
Ask your team to document which instance each endpoint and job uses, then confirm that the routing configuration matches that list before go-live.
On Kubernetes, the liquibase-only Spring profile lets schema migrations run as a separate step from the application pods. This keeps migrations out of the startup path of every replica. The Close of Business job can then run as its own workload, with partition workers started for the duration of each run, so batch resource use stays separate from API pods.
- Migration job: run a one-off job with the liquibase-only profile before rolling out new application pods, and make the rollout wait until that job completes.
- Batch workload: schedule the Close of Business job as a separate workload with its own resource limits so it does not starve API pods.
- Worker pool: for remote partitioning, size the worker pool and confirm that each pod receives the correct broker address and credentials.
- Retries: set retry behavior so a rescheduled pod does not start a second run over accounts that were already processed. Confirm restart semantics for the Spring Batch version you ship.
- Configuration: supply database, broker and tenant settings through Kubernetes secrets and config maps, and check that each workload uses the profile that matches its role.
Check that the scheduler, the migration job and the rollout order cannot start a batch run against a schema that is still changing.
In an isolated network there is no route to an external broker, public package repository or managed database service. The broker that carries partition messages must sit inside the same isolated network as the batch workers. Every container image, driver and library the job needs must be imported through a controlled path before the run is planned.
- Broker placement: host ActiveMQ or Kafka inside the enclave, with its own patching, monitoring and backup process, since no vendor service is reachable.
- Artifacts: mirror container images and build dependencies internally. The official Docker image ships only the PostgreSQL JDBC driver, so confirm how any other driver will be added and mirrored.
- External events: publishing of business events is disabled by default. Enable it only for consumers inside the network, and document each consumer.
- Time and logs: job timing and audit records depend on internal clock synchronization and log forwarding, both of which must work without internet access.
- Update cycle: every change to images or broker versions passes through the import path, so schedule time for that process in each upgrade plan.
Confirm with your security team which broker versions and artifacts they will admit into the enclave, because that list sets how quickly fixes can be applied.
Securing the batch workers, broker and credentials
The Close of Business job reaches the database, the message broker and the API surface, so each of those paths needs its own controls. The cards below cover the credentials, transport, broker access, database accounts, tenant boundaries and secret handling that a team should settle before the first real batch run.
Replace the default development login
Fineract's README documents a default development login, mifos with a matching password, and states that it must be changed for any real deployment. Before a batch run touches real data, replace it with named credentials, confirm that no environment still accepts the default, and check that the batch and worker processes authenticate with their own identities rather than shared staff logins. Record who owns each credential and how it rotates.
Enforce HTTPS on every HTTP path
The Fineract documentation strongly encourages HTTPS only and documents SSL settings through environment variables such as FINERACT_SERVER_SSL_ENABLED. Apply this to every HTTP path that carries batch-related traffic, including the API, health and management endpoints used by orchestration, and any internal call between nodes. Confirm the setting on each instance type, because a read or batch instance left on plain HTTP can expose credentials even when the write instance is secured.
Authenticate and encrypt broker traffic
Remote partitioning coordinates worker nodes through ActiveMQ or Kafka, so the broker carries job instructions and status between the coordinator and workers. Decide who may connect to the broker, which credentials each worker uses, and whether the transport is encrypted between nodes. Confirm the authentication and encryption options with your broker vendor and the version you run, since the batch documentation does not specify them.
Least-privilege database accounts for workers
Workers read account state and write batch results, but they do not need schema change rights. Liquibase migrations run only on write instances, so give batch and read workers database roles that exclude DDL and unrelated tables. Review the grants against the tables the Close of Business job actually touches, and keep migration credentials separate from runtime credentials so a compromised worker cannot alter the schema.
Tenant isolation for batch access
Fineract is multi-tenant, with API requests carrying a Fineract-Platform-TenantId header and tenant connections configured from a tenants database. A batch run that spans tenants must not let one tenant's job read or write another tenant's data. Confirm how the batch job selects its tenant context, test that a worker configured for one tenant cannot reach another, and document which operators may trigger batch runs for each tenant.
Keep secrets out of images and job definitions
Database passwords, broker credentials and SSL keystore material should not be baked into container images or written into job definitions and version-controlled configuration. Inject them at runtime from a secrets store the platform team controls, and scan images and manifests for embedded values before promotion. Also confirm that logs from failed batch steps do not echo connection strings or credentials.
How to test the batch run and prove it fails safely
The test plan for the Close of Business Day Spring Batch job should answer one question before go-live: if a worker fails at any point during the nightly run, does the institution end up with exactly the postings it expects, no more and no less. Testing the normal path is the easy part. The proof that matters is a full run on data sized like production, with its outputs compared against a reference, followed by deliberate failures that show how the job recovers. Ask the implementation partner to write these cases into the test plan with expected results agreed in advance, so the outcome is not judged after the fact.
Start with a full nightly run on a copy of production-sized data, not a small sample. The Fineract README notes that the job can run for several hours on deployments with millions of accounts, and a sample database will hide the slow queries, lock contention and memory pressure that appear at that volume. Run the same dataset once on a single node as a reference, then run it again with remote partitioning across the worker nodes you plan to use, coordinated through ActiveMQ or Kafka. Compare the outputs: loan and savings balances, accrued interest, fees, journal entries and the counts of processed accounts. Any difference should be explained before the test passes. Confirm the exact comparison queries with your partner and record them in the test plan.
Next, force a worker to fail partway through the run. Stop one worker process or its container while partitions are in progress, then confirm how the job resumes. The restart behavior should come from the Spring Batch job repository, which records step and partition execution state, so check that the job is restarted with the same job parameters and that completed partitions are not reprocessed. Confirm the restart behavior against the specific Fineract version you deploy, because batch configuration and broker settings differ between releases. Also test a broker interruption, since a lost message can leave a partition unclaimed.
The most important check is that a partial run never posts twice. Use the failure scenarios above, then compare account-level postings after the restart with the reference run. Every account should have one set of Close of Business postings for the business date, and every journal entry should trace to a single batch execution. Duplicate interest accruals or fees are the failures that auditors and finance teams will find first. If the deployed version cannot prove this from its job state and data, treat that as a design finding to resolve before go-live, not a note for later.
Finally, test that the job is blocked during maintenance windows. Schedule a window, attempt to start the Close of Business job during it, and confirm it refuses to run or waits, and that no partition starts. Repeat the test with a maintenance window that begins while the job is already running. Record the results for each case, along with who is authorized to start and stop the job. Ask your partner which Fineract settings and operational runbook steps control this behavior in your version, and keep the evidence with your change records.
What to monitor, back up, upgrade and document after go-live
This table lists the signals, safeguards and records that keep Close of Business batch jobs dependable once Fineract is live. Read each row as a question your team must be able to answer during a close window, with the evidence to answer it. The main takeaway is that batch reliability depends as much on runbooks, backup points and upgrade re-runs as on the job design itself.
| Area | What to track or prepare | Why it matters for Close of Business |
|---|---|---|
| Job duration per run and per tenant | Record start time, end time and elapsed time for each run and each tenant, and trend them against account volume | The README notes the job can run for several hours at millions of accounts, so a tenant that drifts longer shows up here first |
| Step failures and restarts | Log each Spring Batch step's status, failure reason and restart count, and confirm restarts resume from the last committed point | A failed step left unclear can cause duplicate or skipped postings, so restart behavior must be tested, not assumed |
| Broker lag or queue depth | Watch ActiveMQ or Kafka queue depth and consumer lag for the partition messages used by remote partitioning | Growing depth means worker nodes are falling behind and the close window will stretch |
| Worker health | Track worker node availability, memory, CPU and connection pool use during the run | Idle or crashed workers reduce parallelism silently unless each node reports its state |
| Database replica lag | Measure PostgreSQL replica lag on the instances that serve reads during the window, and confirm batch reads and writes hit the intended instances | Stale reads during close can produce balances or reports that do not match the ledger |
| Alerts tied to the close window | Define alert thresholds that open only during the scheduled window and route to a named on-call owner with a runbook link | Generic alerts at all hours create noise, while missing window-specific alerts delay recovery |
| Backups before the run | Take and verify a database backup point before each run and before any Liquibase migration, and record the restore procedure | A verified point gives a known state to return to if a run must be reversed |
| Re-running after quarterly upgrades | After each quarterly Fineract release, run the Close of Business job in a non-production copy with production-like volume and compare results and timings | Upgrades can change job behavior or data, so a regression run should be a standing gate, not an optional check |
| Runbooks for restart and manual intervention | Document who may restart a failed run, which steps are safe to re-run, when to halt, and how to escalate | Under time pressure, undocumented manual steps are where errors and duplicate postings most often start |
When to partition and when to fix the single-node run first
Fix the single-node run first, and partition only when measured bottlenecks show that the run still cannot finish inside its window after tuning. The Close of Business Day Spring Batch job can run for several hours on deployments with millions of accounts, and the Fineract README describes remote partitioning across worker nodes coordinated through ActiveMQ or Kafka as a way to spread that work. Partitioning is a real option, but it is the most expensive change in this list, so it should come after cheaper causes have been ruled out.
Start by profiling the job on a production-sized copy of the data. Ask the team to capture where time goes in each step: database query time, lock waits, index use on the tables the job reads and writes, and time spent waiting on replica lag if the batch instance reads from a replica. Check that the batch instance runs on its own, as the Fineract documentation describes separate read, write and batch instance types, and that it is not competing with API traffic for the same database connections or CPU. Confirm which PostgreSQL version is in use and whether the query plans for the largest loops are sensible, since PostgreSQL is the recommended database for Fineract.
Instance sizing and database placement are usually the next cheap levers. Give the batch instance enough memory and CPU for the thread pool it is configured to use, and make sure the database host has the I/O headroom the job needs during the window. If the batch job reads from a replica, confirm that the replica is not so far behind that the job sees stale balances, because that is a correctness risk as well as a performance one. Each of these changes can be made, measured and rolled back without new infrastructure to operate.
Partitioning should be justified by a written test result. Record the baseline run time, the bottleneck found, the change made, and the new run time on the same data volume. If the window still does not fit after those fixes, partitioning becomes a candidate, and the team should estimate what it adds: a message broker (ActiveMQ or Kafka), worker nodes that must be provisioned and patched, and new failure modes such as a worker dying mid-partition, duplicate or missing work after a broker outage, and restarts that must resume cleanly. Ask the implementation partner to show how partitions are retried and how the job reports partial failure before accepting the design.
Treat the decision as reversible only if you keep it measured. Keep the baseline run and the timing results with the change record, and rerun the job after each Fineract release, because upgrades can change query patterns. Confirm the exact partitioning settings and broker support against the Fineract documentation and the version you deploy, rather than relying on the README summary alone. A single well-tuned node that finishes inside the window is a better outcome than a partitioned design that nobody can operate.
Questions teams ask before scaling nightly batch jobs
What drives the cost of scaling the nightly batch job?
Data volume, partition design and the number of worker nodes are the main cost drivers, along with the effort to operate the broker and workers. The Fineract README warns that a production deployment can be complex, costly and time-consuming, and notes that organizations may pay a vendor for deployment and maintenance.
Main cost components
- Discovery: timing the current run on production-sized data and finding the slow steps.
- Design: partitioning rules, worker sizing and broker setup with ActiveMQ or Kafka.
- Build and test: configuration, monitoring and rehearsal runs.
- Infrastructure: worker nodes, broker capacity and database resources.
Discovery, design and testing are one-time work. Infrastructure, monitoring and support recur. Ask for a quote that prices each component separately.
How long does it usually take to scale the batch job?
The duration depends on how far the current run is from the target and how much testing the change needs, so no general figure applies. The README says the job can run for several hours on deployments with millions of accounts, which is why baseline measurement comes first.
The work usually follows these phases: baseline timing on production-sized data, partition design, worker and broker build, load testing, then a staged rollout with a fallback plan. Each phase depends on data access, change windows and staff availability. Ask for a schedule with milestones and the test runs each milestone requires.
What data and environments must we provide for batch tuning?
You must provide a test environment that matches production in data volume, database engine and database configuration, because partitioning behavior changes with account counts. The README recommends PostgreSQL, so the test database should use the same engine and version family as production.
You also need:
- Production-sized data, or a copy with equivalent volume and shape, under your data protection rules.
- Staff who know the broker, either ActiveMQ or Kafka, including how to inspect queues and consumer lag.
- Access to job logs, database metrics and existing monitoring.
- A defined window for timed test runs.
Confirm these with your own platform owners before the quote is finalized.
Can our in-house team scale the batch job without a partner?
Yes, if your team already has Spring Batch experience, PostgreSQL tuning skills, and operational experience with ActiveMQ or Kafka, and can run repeated load tests. The Fineract documentation separates batch instances from read and write instances, so the work sits in one part of the platform, which makes an internal project more contained.
The main risk is time spent on trial and error. Without a timed baseline and a rehearsal process, teams often change several settings at once and cannot tell which change helped. A partner can add experience, but the decision should rest on your staff's skills and availability, and on who will run the system after go-live.
What should a fixed-scope quote for batch scaling include?
A fixed-scope quote should state what is in scope, what is excluded, and what will be delivered. DeployOpen agrees scope, deliverables and timelines in writing before work starts, so the quote should be traceable to that document.
Ask the quote to cover:
- Baseline measurement on production-sized data, and the target run time agreed with you.
- Partition and worker design, including the broker choice.
- Monitoring and alert definitions.
- The number of rehearsal runs and the acceptance criteria for each.
- Environment requirements you must provide, and any assumptions behind the estimate.
- Handover documentation and any ongoing support options, listed separately.
What risks should we check before enabling remote partitioning?
Check failure handling, duplicate processing, partition imbalance and database contention before enabling remote partitioning across workers. The README describes coordination through ActiveMQ or Kafka, so a broker outage or a stalled worker can affect the whole run.
Confirm these points against your Fineract version and configuration:
- How failed partitions are retried and whether restarts resume safely.
- Whether partitions are balanced by data volume, not just by count.
- How the database responds when many workers read at once, especially if read replicas serve other traffic.
- Whether external business events are enabled, since the README says they are disabled by default and they add broker load when turned on.
Test each failure case in the test environment before the production rollout.
How do we monitor the batch job after scaling changes?
Monitor job start and end times, step durations, failed or retried partitions, broker queue depth and database locks, and set alert thresholds from your measured baseline. Without a baseline, an alert threshold has no reference point, so record timings before any change.
Keep a run history so you can compare each night with the previous week. Watch for partitions that finish late, since one slow partition often sets the total run time. Confirm which metrics your Fineract version and deployment expose, and which ones you must collect through your own tooling. Review the dashboards with the operations team before go-live so that on-call staff know what a normal run looks like.
Does the Fineract README mean we need a vendor for production?
The README does not require a vendor, but it states that the project does not provide a comprehensive production deployment guide and that organizations may pay a vendor for deployment and maintenance. That means the production knowledge must come from your team, a partner or both.
Use the README warning as a skills check. If your team can design, test, run and document the batch setup, including failure recovery and upgrades, you may not need outside help for every part. If any of those areas is thin, a partner can cover that gap under a written scope. Either way, the operating responsibility stays with the bank or institution that runs the platform.
How DeployOpen scopes, builds and hands over Close of Business scaling
DeployOpen starts with a written scope that maps which Close of Business Day Spring Batch steps run on your Apache Fineract deployment, how long they take against your account volumes, and whether remote partitioning is needed. The work is then built, tested and documented inside your own infrastructure, so your team can run and extend the batch layer after handover.
How we scope
- Inventory every Close of Business step your products and accounting configuration trigger, with current run times from your environment.
- Confirm account volumes, batch window expectations and whether the job must finish before a fixed daily cutoff.
- Agree in writing whether the scope covers partitioning with ActiveMQ or Kafka, monitoring, or both, since engagements are fixed-scope.
How we build
- Separate the batch instance from read and write instances, using the environment variable settings Fineract documents for each type.
- Configure remote partitioning across worker nodes and test broker behavior under failure, restart and partial worker loss.
- Set up monitoring for job duration, step failures and partition lag, then rehearse the nightly run on production-sized copies of data.
How we hand over
- Deliver runbooks covering job scheduling, restart procedures, partition sizing changes and broker configuration for your operations team.
- Document the monitoring dashboards, alert thresholds and escalation paths agreed during the build.
- Walk your engineers through a supervised nightly run and a failed-run recovery exercise before handover is signed off.
How we support
- Support is available as a retainer covering Close of Business job reviews as account volumes and product configuration change.
- Retainer work can include regression testing of batch jobs against each new Fineract release before your upgrade window.
- Confirm response times, covered environments and upgrade assistance in the retainer terms, since these are set per agreement.
