Why loans get stuck after a failed Close of Business in Apache Fineract
The right fix for a loan that returns HTTP 409 or stays locked after a failed Close of Business (COB) depends on one question: is the lock still held by a running job, was it left behind by a job that failed, or is it attached to a recorded error? Each case calls for a different response, and the repair itself must be controlled, logged and approved, because the work touches the balances and schedules of accounts that the bank has already closed for the day. Start by confirming which of these states applies before anyone calls an endpoint or changes a record.
During COB, Fineract's batch processing runs business steps across loan accounts to accrue interest, apply overdue and delinquency logic and prepare the next business date. A lock on a loan is the mechanism that stops customer and back-office transactions from changing that account while a step is reading or writing it. The lock protects balance integrity, but it also means a loan can remain unavailable after the job that set it has stopped. Operations teams should treat the lock as a record with a type, a status and an owner, and read it before deciding whether it is stale.
Exact lock names, endpoints, status values and the repair flag must be confirmed against the Fineract 1.x documentation and the source code for the release you run, since these details vary between releases and are easy to misremember under pressure. Ask your implementation team to document the release tag, the endpoints used to inspect and clear locks, and the role required to run them, and to rehearse the procedure in a non-production tenant before a production incident. Your own change and approval process should cover every repair transaction, with the business owner signing off on each affected loan.
Which lock state points to which repair path
This table compares the four lock states operators are likely to meet after a failed Close of Business run, with what each usually means, what to check first, the repair path that fits and the main risk of acting too early. Exact state names and how each appears in API responses must be confirmed in the Fineract documentation and source for the release in use before any repair is run.
| Lock state | Usually means | Check first | Repair path | Main risk of acting early |
|---|---|---|---|---|
| Soft lock | A batch or scheduled process is still running or has not released the loan yet | Whether the COB job or the related job is still active in job history | Wait for the run to finish, then retry the request | Clearing a lock while the job still writes to the loan can corrupt balances |
| Hard lock | A run stopped part way and the loan was not released by the normal path | Job run history, server logs and the timestamp of the lock | Confirm no run is active, then apply the documented unlock or COB retry for that loan | Forcing a second run over a partly processed loan can post interest or fees twice |
| Lock with an error | A business step failed and recorded an error against the loan | The error text, the failing business step and the loan's transaction history | Fix the cause (data, product or holiday setting), then rerun the step for that loan | Rerunning before the cause is fixed repeats the same failure and adds noise to the ledger |
| Orphaned lock | An interrupted or killed process left a lock with no active owner | That no job is running on any worker node, including partitioned workers | Use the orphaned lock cleanup procedure in the release documentation, then record the action | Cleanup while a worker still runs can release a lock that protects in-flight postings |
How to diagnose a locked loan before you repair it
This sequence turns a 409 response or a loan that will not move past Close of Business into a documented diagnosis, so the operator knows which lock they are dealing with and whether any repair is safe to attempt.
Capture the failing response and loan ID
The operator who received the error records the full HTTP 409 response body, the request method and path, the tenant identifier sent in the Fineract-Platform-TenantId header, the loan ID, and the time of the request. Keep the raw response rather than a paraphrase, because the error text often names the lock or the operation that conflicted. The step is complete when another person could reproduce the request and find the same loan from the notes alone. If the tenant or loan ID is uncertain, stop here and confirm it before going further.
Check whether a COB or batch job is running
Before touching the loan, confirm whether the Close of Business job or another batch job is still executing on the tenant. Find the running state through the job and run history views your deployment exposes, and confirm the start time against the business date. Because the Close of Business job can run for several hours on large deployments, a lock during an active run is often expected behavior, not damage. If a job is still running, wait and recheck the loan rather than intervening. Confirm the exact method for viewing job state in the platform documentation and in your environment before relying on it.
Review job history and logs for the failure
If no job is running, read the run history for the last Close of Business attempt and note whether it completed, stopped or recorded an error. Then search the application logs from that run window for the loan ID, the tenant and the exception name. Log locations and retention depend on how Fineract was deployed (container runtime, Kubernetes pod logs, or a file path on a host), so confirm where your instance writes them. The step is complete when you can say whether the failure happened on this loan, on a shared step, or on a different loan entirely.
Confirm the lock type and any attached error
Using the loan's lock information in the platform, identify whether the lock is a soft lock or a hard lock, and whether it carries an error record. Record the lock type, the operation it protects, its creation time and any message attached to it. A lock with an error needs a different response from a lock that is simply waiting for a job to clear it. If you cannot see lock details through the interface or API available to you, document that gap and confirm the access path with your platform administrator before proceeding.
Decide the repair path from the evidence
Choose one repair path based on the findings: wait and retry when a job is still running, rerun Close of Business for the affected loan when the failure is understood, or use a repair transaction with the bypass setting only after the cause is fixed and approved. Orphaned lock cleanup should follow only when no job is running and the lock has no live operation behind it. Write the chosen path, the reason and the approver in the incident record before making any change. The step is complete when the decision can be defended to an auditor from the captured evidence.
Which repair path fits which situation
Use this path when the Close of Business job, or a related batch, is still executing and the lock belongs to it. A held lock is expected while the job works, and a manual change made during a live run can collide with postings the job is writing to the same loans. The first task is to prove the job is alive, not to clear anything.
- Confirm the job is live: check the job run history and application logs on every node, including any remote partition workers coordinated through ActiveMQ or Kafka.
- Schedule a re-check: the README notes that COB can run for several hours on deployments with millions of accounts, so set a re-check time and do not intervene early.
- Do not set the bypass flag: the bypass for repair transactions is meant for loans a finished run left unprocessed. Using it against a live run risks conflicting postings.
- Do not run orphaned lock cleanup: cleanup applies only to locks with no owning process. Prove that no process owns the lock before considering it.
- Record the observed state: log the job identifier, start time, node names and the time of each check in the incident ticket. Approval at this stage belongs to the operations lead for COB, and no write should be approved yet.
Before leaving this tab, confirm that the job has either finished or stopped, and that a verified backup and a named approver exist before any write.
Use this path when the job has ended or its process crashed, the lock remains, and no error is recorded against the affected loans. The lock is now orphaned, and the loans it blocks need a controlled restart of processing. Because the cause is unknown, the repair must start from a verified backup and a restore point that predates the crash.
- Prove no owner remains: confirm no COB process is running on any application node or worker, and that the job history shows an aborted or stopped run.
- Scope the cleanup: list the locks by loan identifier and confirm each one belongs to the stopped run. Apply orphaned lock cleanup only to that list.
- Use inline COB for selected loans: after cleanup, run COB for the chosen loans only, so the rest of the batch is not reprocessed unnecessarily. Check the results loan by loan.
- Check the ledger: compare postings for the affected loans before and after the repair with the accounting owner, because a crash can leave partial journal entries.
- Record the change: the cleanup and each inline run should appear in the audit trail with the actor and loan list. Confirm with your logs and vendor documentation what is captured per loan. Approval should come from the platform owner and the COB owner, with finance sign-off if postings change.
Before applying the cleanup to production, rehearse the same steps on a restored copy and confirm that the loans process as expected.
Use this path when the lock is paired with an error that blocks one loan or a group of loans, and retries do not clear it. The error is the evidence, so the cause must be identified before any lock is touched. Clearing the lock first can hide the original failure and let the same data fault recur on the next run.
- Read the error and history: record the exact error text, the loan transaction history and any linked group or centre records. Decide whether the fault is in loan data, such as a schedule or charge, or in the environment, such as database timeouts.
- Fix the root cause first: correct the data or environment, then rerun the failing step on a non-production copy to confirm it succeeds.
- Scope the bypass flag: use the bypass for repair transactions only where the correction requires it. The written scope should list each loan, the exact transaction and the expected result. For group loans, include every affected member.
- Obtain a second review: the change should be reviewed by someone other than the person who prepared it, and security should review any use of the bypass flag.
- Confirm the audit trail: the bypass use and each repair transaction should appear in the audit trail with the actor, time and reason. Verify that retention meets your policy.
Approve the repair with the loan operations manager and the finance or risk owner. Check that the failing step now completes without error and that the lock is released before closing the ticket.
Safeguards to set before anyone clears a lock
A lock on a loan is a state the platform set for a reason, so clearing it is a change to a live core banking record. The controls below are meant to be agreed and written down before the first incident, so that operators under pressure follow the same steps every time.
Written runbook per lock type
Keep a separate runbook for each lock state you see in your environment, such as a soft lock, a lock with an error, or a lock left behind after a failed run. Each runbook should name the symptom, the query or screen used to confirm it, the permitted repair, and the step that returns the loan to a known state. Before anyone acts, check that the runbook matches the lock type you actually observed, because a fix for one lock type can damage a loan locked for another reason.
Bypass flag limited to trained operators
The bypass flag allows repair transactions that the normal rules would block, so access to it should be a named, short list of trained operators rather than a general back-office role. Record who holds the permission, review the list on a fixed schedule, and remove access when someone changes role. Confirm in your own environment how the flag is granted and whether it is visible in audit records.
Second approver for repair transactions
Require a second person to review and approve any repair transaction before it posts. Check whether your Fineract configuration and your operating model support maker-checker for the specific transaction types involved, and if they do not, document the compensating review. The approver should confirm the loan, the amount, the accounting effect and the reason, not just that the request arrived.
Backup or point-in-time marker before repair
Before any manual change, take a database backup or record a point-in-time marker in PostgreSQL so that the state before repair can be restored and compared. Confirm that the backup covers the tenant schema being repaired, not only the tenants database. Note the marker in the incident record so that anyone reviewing the repair can find the exact state it started from.
Log of every manual action
Keep a single log for all manual lock actions that records the loan ID, the tenant, the operator, the time, the lock type, the reason and the exact transaction or command used. Entries should be written before the change where possible and closed afterward with the result. Review the log at the end of each COB cycle so that repeated locks on the same loan or product are spotted early.
Business date check before inline COB
Confirm the current business date for the tenant and the date of the last completed Close of Business before running inline COB for selected loans. Running it against the wrong date can post interest and accruals into a period that has already been closed, which is difficult to unwind. Record the date check in the log entry for that loan, and do not start inline COB while the scheduled batch job is still running.
What usually causes failed COB runs and leftover locks
Most failed Close of Business runs and leftover loan locks trace back to a short list of operational causes, and the fix starts with ruling those out before anyone touches a lock. The Fineract README notes that the Close of Business Day Spring Batch job can run for several hours on deployments with millions of accounts. A run that long is exposed to restarts, pool exhaustion and human intervention for much of its duration. The usual suspects are long runs on large portfolios, worker pods interrupted during a rolling restart, database connection exhaustion or timeouts while the job is working, an error in one business step, and manual changes made while the batch is still active.
The COB job can also be partitioned across worker nodes coordinated through ActiveMQ or Kafka. That design means a failure on one node can leave some loans locked and others already processed, so the state of the portfolio after a failure is mixed rather than uniform. Before running any repair, confirm which nodes ran which partitions, whether any worker was restarted or evicted, and whether the broker and database stayed reachable throughout. Your Kubernetes events, the Fineract logs and database connection metrics for the run window are the evidence to collect first.
Check capacity and connection settings before blaming the repair tools. Compare the number of connections the write and batch instances can open with what the database accepts, and confirm that timeouts were not the reason the job stopped. Confirm with your DBA and platform team which values were in force during the run, because a configuration that worked on a smaller portfolio may not hold at current volume. Finish root cause analysis and agree the corrective change before a second run is started, since rerunning a batch on an unexplained failure can lock or double-process more loans.
What to confirm with the vendor and in your environment
The repair steps for locked loans and failed COB depend on details that vary by release and deployment. None of the items below should be assumed from this page, and each one should be confirmed in writing or tested in your own environment before anyone runs a repair against production data.
Exact lock states and API responses
Confirm which lock types exist in the release you run, what each one means, and which HTTP status and error body each returns to a client. A 409 on one endpoint may not mean the same thing on another. Ask for the lock states as the running version emits them, and capture real responses from a restored copy so your runbook matches what your team will actually see.
Bypass flag name, scope and default
Confirm the exact name of the flag that lets repair transactions pass a lock, which requests and roles it applies to, and whether it is disabled by default. Find out who can set it and whether its use is logged. Treat any bypass as a privileged change that needs maker-checker style approval and a record of why it was used.
Orphaned lock cleanup method
Establish whether orphaned locks can be cleared through a supported API operation or only by changing rows in the database. A direct database change bypasses application logic, audit trails and any cache state, so it needs a documented procedure, a backup taken immediately before, and sign-off from the product owner. Ask the vendor which tables hold lock records in your release.
Inline COB behavior for selected loans
Confirm whether inline Close of Business can run for a chosen set of loans, which business steps it executes, and how it reacts when a loan in the set fails. Check whether it affects accounting entries, interest accrual and the business date in the same way the scheduled job does. Ask what happens to the full batch when a single loan is processed inline.
Interaction with the scheduler and job settings
Confirm how repair operations interact with the Quartz-scheduled COB job, including whether a manual run can start while the scheduled job is active, and which global configuration switches enable or disable jobs during repair. Agree a change window in which the scheduler is paused, and confirm how to resume it safely without a second COB run colliding with the first.
Repair steps tested on a restored copy
Confirm that every repair step has been run against a restore of current production data, not only against a small test dataset. Record the run time, the number of loans affected and the resulting ledger balances, then compare them with the expected figures. Ask your team to verify that the restore itself is recent and complete before using it as the test basis.
Questions operators and buyers ask about stuck loans and failed COB
How long does it take to fix stuck loans after a failed COB?
The duration depends on how well the cause is understood, not on a fixed number. A single loan locked by a known error is usually a short investigation, while a failed run that left many loans in mixed states needs a full inventory first. The main drivers are the size of the portfolio, the number of affected loans, whether the failure is reproducible in a test environment, and whether the job history and logs are complete. Ask any partner to state the assumptions behind its estimate and to separate diagnosis from repair in the written scope.
Can our own team fix stuck loans without a partner?
Yes, many teams can, provided they have trained operators, a written runbook, and a restore path they have rehearsed. The real risk is not the API calls themselves but acting before the cause is diagnosed. A repair that clears a lock without understanding why the loan failed can leave ledger entries, schedules or accruals inconsistent, and those errors surface later at period close. Teams without database and Fineract job experience should confirm they can read job run history and check the database state before they start, and should consider an outside review of the first incident.
What does a partner's scope usually include for this work?
A typical scope covers four parts, agreed in writing before work starts:
- Diagnosis: reviewing logs, job run history, lock records and database state to identify the cause and the affected population.
- A runbook: documented steps for each lock type, the bypass flag for repair transactions, orphaned lock cleanup and inline COB for selected loans, with approval points.
- Controlled repairs: executing agreed fixes in a test environment first, then in production under the client's change process.
- Post-incident review: recording root cause, the changes made and preventive actions such as capacity or scheduling adjustments.
The exact boundaries, including whether preventive work is included, should appear in the statement of work.
What do we need to provide for a stuck loan engagement?
The work depends on evidence from your environment, so the client needs to provide access and people. Typical requirements are read access to application and scheduler logs, the Quartz and COB job run history, and database views needed to inspect loan and lock state. Partners also need recent database backups for restore testing, a named approver for each repair category, and a test environment that matches production's Fineract version, configuration and tenant setup. Confirm with your partner how access is granted, recorded and removed, and whether production data can be copied to the test environment under your data protection rules.
What drives the cost of a quote for COB troubleshooting?
A quote follows from the written scope, so the main cost drivers are the scope itself and the work it implies. Ask how many incidents are included, whether the engagement covers one failure or a standing response capability, how complex the environment is (multiple tenants, read replicas, custom plugins, cloud or air-gapped hosting), and whether preventive capacity work is included. One-time work includes diagnosis and the first runbook. Recurring costs appear when you want retained support or rehearsal of the restore path. Request a quote that separates these items so you can compare it with the cost of your own team's time.
Can repairs to loans locked during COB be undone?
Some can, and some cannot, which is why the repair plan should be decided before any change. Fineract records financial transactions, so a corrective transaction is normally a new entry rather than an edit, and the reversal path needs to be agreed in advance. Lock records and job state can often be restored from backups taken before the change, but only if those backups are recent and tested. Before a repair, confirm which steps are reversible, who approves irreversible ones, and how the database will be restored if a repair goes wrong. The project's documentation for the platform is the starting point for these checks.
How DeployOpen scopes and delivers Fineract incident recovery
DeployOpen treats a stuck loan or failed Close of Business run as a controlled incident on a self-hosted Fineract platform inside your own infrastructure. We start by establishing which locks, job runs and data states exist, then agree in writing what the recovery covers before any repair transaction is made.
How we scope
- Review lock types, failed job runs and affected loan accounts in your Fineract environment before proposing any fix.
- Define in the written scope which loans, tenants and COB dates are covered by the recovery work.
- Agree who approves use of the bypass flag and who records each repair transaction and its justification.
- Engagements are fixed-scope, with deliverables and timelines agreed in writing before work starts.
How we build
- Prepare a runbook for soft locks, hard locks, locks with errors and orphaned lock cleanup specific to your deployment.
- Test each repair step against a restored copy of your database before running it on production data.
- Confirm inline COB for selected loans behaves as expected on your version and data model before use.
- Document the database state captured before and after each change so that results can be reviewed independently.
How we hand over
- Deliver a written record of every lock cleared, repair posted and job rerun, with the approver named for each.
- Provide a diagnostic checklist your operations team can run when a 409 response or stuck lock recurs.
- Walk your platform and operations staff through the runbook and the conditions under which the bypass flag is used.
- Document configuration and scheduler settings changed during recovery so they can be reviewed by your auditors.
How we support
- Offer ongoing support and continuous testing as a retainer if you want DeployOpen on call for future COB incidents.
- Review scheduler job history and COB logs after each upgrade or configuration change to catch recurring lock issues early.
- Retest the runbook when your Fineract release changes, so repair steps match the code you run.
- Ask for a quote based on the written scope, since DeployOpen does not publish a price list.
