Running Apache Fineract Liquibase migrations as a separate Kubernetes job

Running Liquibase once in a dedicated job before the rollout prevents migration races between scaled Fineract pods. This page covers the sequence, rollback points and the single-threaded limit.

On this page

Why an Apache Fineract Liquibase migration job comes before the rollout

Run the liquibase-only Spring profile as a one-off job before the new Fineract rollout, and keep migrations disabled on every read, write and batch instance. The Fineract documentation provides this profile for exactly this purpose: it runs only the Liquibase migrations and exits, so the schema changes happen once, in a known order, instead of racing across replicas that all start at the same moment. Liquibase database migrations run only on write instances, and on Kubernetes they can run separately with the liquibase-only profile, so the job is the supported path and the write deployment should not be the place where schema changes happen. The upgrade cadence is set by the project, not by your change calendar. Official Fineract releases are created quarterly, at the end of each quarter, under the ASF release policy, and end-of-life releases are archived. The Fineract website lists 1.15.0 as the current stable release, so a platform team running an older 1.x line should confirm which release it is moving to and whether that line still receives fixes before scheduling the work. This page assumes PostgreSQL, which the README names as the recommended database. If your tenant databases still run MySQL or MariaDB, plan that move as a separate project first, because the job below is written for the PostgreSQL path.

Treat the migration job as a change with its own owner, runbook and rollback point. Before the job runs, confirm with your DBA that a verified backup exists, that the database role used by the job can apply DDL, and that no batch or Close of Business Day run is in progress, since the batch instance is a separate process that also depends on the schema. Check the release notes for the target version and the Mifos X web app version you will deploy alongside it. Your team should confirm the exact FINERACT_LIQUIBASE_ENABLED value on each deployment and the job's completion status in your cluster tooling, not just its exit code in a log. The sections that follow cover the manifests, the rollout order and the limits of Liquibase's single-threaded execution.

What changes in the release and what each change means for the upgrade

This table lists the platform changes that affect a Liquibase job split on Kubernetes, with the impact of each and the action your team needs to take. Read it before you edit any deployment manifests, because the first two rows decide which instances may touch the schema.

ChangeImpactAction needed
liquibase-only Spring profileRuns only the Liquibase migrations and exits, so schema changes happen in a single short-lived process instead of inside app instances.Build a Kubernetes Job that sets the liquibase-only profile and confirm the job exits successfully before the rollout starts.
FINERACT_LIQUIBASE_ENABLED on regular instancesRead, write and batch instances should not run migrations on their own startup, which removes the race between replicas.Set FINERACT_LIQUIBASE_ENABLED to disabled on read, write and batch deployments, and check each environment variable set.
Liquibase runs only on write instancesMigrations are tied to the write instance type, so a batch or read instance is not a valid place to apply schema changes.Confirm which deployment carries the write instance type, and make sure the migration job is aligned with that configuration.
Kubernetes job pattern before rolloutThe schema must be updated and verified before new application pods start, so the job becomes a gate in the release sequence.Order the pipeline as migration job, job success check, then application rollout, and stop the pipeline if the job fails.
Fineract CN deprecation and the 1.x lineFineract CN was deprecated in May 2023 and archived in late 2023 without an official release, so 1.x is the maintained platform for this work.Confirm your target is a 1.x release, and remove any dependency on Fineract CN artifacts from your build and deployment plans.
Default development loginThe README default login of mifos with password is intended for development only and must be changed in any real deployment.Replace the default credentials before the first environment is exposed, and verify that the default login is rejected in each environment.

What to prepare before the migration job runs

Five items need to be ready before anyone runs the liquibase-only job against a production database. Together they give the team a known restore point, a complete list of databases to migrate, a clear view of what changes between releases, a rehearsal environment, and a front end that matches the backend.

Verified database backup and restore

Take a fresh backup of every tenant database and the tenants database, then restore it to a separate instance and confirm Fineract starts against it. A backup that has never been restored is a hope, not a rollback plan. Record the time the restore took and the exact Fineract image tag that was running when the backup was taken, because that tag is the version you return to if the migration fails.

Inventory of tenant databases

Fineract is multi-tenant, and tenant connections are read from the tenants database. Query that database to list every tenant, its schema or host, and its connection settings before the job starts. Migrations that cover only the default tenant leave other tenants on an older schema, and the write instances will fail when they reach those tenants.

Image and environment variable diff

Compare the current and target Fineract images, and the environment variables each deployment sets, line by line. Check for new, renamed or deprecated settings, including the FINERACT_LIQUIBASE_ENABLED flag and the Spring profile used by the migration job. Ask the platform team to confirm which values come from the release notes and which come from your own overrides.

Staging copy of production data

Restore a recent production snapshot into a staging environment that runs the same database platform and Kubernetes manifests as production. Run the liquibase-only job there first, then start read, write and batch instances and run your regression tests for products, accounting and the close of business day. Staging only proves something if its data and configuration are close to production.

Matching Mifos X web app version

Confirm which Mifos X web application version pairs with the Fineract release you are moving to, and deploy it in the same change window as the backend. A front end that calls endpoints the new backend no longer serves will fail for staff even when the migration itself succeeded. Check any custom UI changes against the new version before the cutover.

How to sequence the Liquibase job, app rollout and checks

This sequence makes sure one Liquibase run completes and is verified before any application pod starts the new Fineract image, so scaled instances never race to change the schema.

  1. Freeze writes or schedule the change window

    The platform owner and the business operations lead agree a window in which no new loan, savings or accounting postings are made, and confirm that no Close of Business Day run is in progress. Check the batch job status in the Fineract API and the scheduler before going further. Record the window, the current image tag and the current Mifos web app version in the change ticket. Done means the operations lead has signed off that the system is quiet, and you have a written record of the starting state.

  2. Take a verified database backup

    Before any schema change, take a PostgreSQL backup of the tenant databases and the tenants database, and confirm the backup is restorable by testing a restore into a separate environment. Store a copy of the Kubernetes manifests and the currently deployed image digests. This snapshot is your first rollback point: if the migration fails, you restore it and redeploy the previous images. Done means the restore test has completed and the backup identifier is recorded.

  3. Disable Liquibase on regular instances

    Set FINERACT_LIQUIDBASE_ENABLED to disabled on the read, write and batch deployments, so that none of them attempts a schema change when it starts. Apply the change to the current running release first so that a pod restart during the window cannot trigger a migration. Confirm the variable is set in every deployment's live spec, not only in the template files. Done means each deployment reports the setting and no pod has logged a Liquibase run since the change.

  4. Run the liquibase-only job to completion

    Create a Kubernetes Job that uses the new image with the liquibase-only Spring profile, pointed at the same database credentials as the application. Let it run until the pod reaches the Completed state. Do not start the application rollout until the Job finishes, because any app pod on the new image could otherwise start while the schema is in a partial state. Done means the Job shows a successful completion with exit code zero.

  5. Check job exit status and logs

    Read the Job status and the pod logs from start to finish. Confirm the exit code is zero, that Liquibase reports each changelog as applied or already applied, and that no lock, checksum or SQL error appears. A failure here means the schema may be partly changed, so stop, keep the old image running, and restore from the backup if the database cannot be reconciled. Done means the logs are archived with the change ticket and show a clean run.

  6. Verify schema state against the release

    Query the database for the Liquibase changelog table and confirm the latest changeset recorded matches the one expected for the target release. Check that the DATABASECHANGELOGLOCK table has no lock held by a finished run. Compare key tables and row counts against the pre-change snapshot to confirm no data was lost. Done means the changelog state matches the target release and the counts reconcile, which is the second rollback decision point.

  7. Roll out write, batch and read instances

    Update the write instance first, wait until it passes its health checks and serves API calls, then update the batch instance, and finally the read instances. Keep the Liquibase setting disabled on all of them. Confirm each pod runs the new image and reports healthy before moving to the next deployment. Done means every deployment shows the new image digest and all pods are ready. If a write instance fails its health checks, roll it back to the previous image while the backed-up schema is still reversible by restore.

  8. Run post-upgrade smoke tests and close the window

    Run the regression checks agreed for products, accounting and the batch run, and confirm the Mifos web app version matches the release. Only after these pass, restore normal operations and record the final image tags, the backup identifier and the Job logs in the change record. Done means the smoke tests pass, operations confirms normal service, and the rollback point is retained for the agreed period.

Why Liquibase is a single-threaded bottleneck on large schemas

The most important thing to understand about the migration job is that adding replicas or nodes does not shorten it. Liquibase applies one changelog sequence in order, so the job works through changesets one after another regardless of how much cluster capacity surrounds it. Scaling the Kubernetes cluster helps the application tier, but it does nothing for the migration. On a large schema with many accounts and years of history, the duration is governed by the database work in each changeset, especially index builds, table rewrites and data backfills. Ask your DBA and the Fineract project to estimate those changesets for your target version against a copy of production data, rather than relying on a figure from a smaller test database. Do not set a delivery date for the upgrade window until that rehearsal has run.

The second risk is partial application. Liquibase records each changeset as it completes, but a changeset that fails midway may leave the schema in an intermediate state, and some DDL statements on PostgreSQL run inside a transaction while others do not. That means a failed job does not necessarily return the database to where it started. Set the job's activeDeadlineSeconds and backoffLimit deliberately. A generous retry count that re-runs a job against a half-migrated schema can compound the damage, so consider backoffLimit: 0 and let an operator inspect the DATABASECHANGELOG table before any re-run. Before go-live, confirm with the Fineract project how each changeset in your upgrade path behaves on failure, and record the answer in your runbook.

Rollback options when the migration job or rollout fails

A failed Liquibase job or a bad rollout leaves you with a schema that may or may not match the image you are running. The four options below differ in how much they restore, how quickly they recover, and what they cost in data written after the change. Whether any of them is safe depends on the specific changesets, so confirm feasibility against the release notes and a staging rehearsal before the production window.

Restore the pre-migration database backup

Take a verified backup of the tenant databases and the tenants database immediately before the liquibase-only job starts, and record the Liquibase changelog state. If the migration fails partway, restoring this backup returns the schema and data to the previous state, but every write made after the backup is lost. Ask whether writes can be paused during the window and confirm that a restore has been timed in staging, including the time to replay any transactions your change process requires.

Keep the old image deployable

Keep the previous Fineract image tag and its configuration deployable so the old application can start against a restored schema. This option only works if the changesets in the new release left the old schema in a state the old code accepts, which you must verify rather than assume. Keep the previous image in your registry for the full rollback window and rehearse starting it against a restored copy of production data.

Roll back only the app tier

When the schema change is backward compatible, you can revert the read, write and batch deployments to the previous image and leave the migrated schema in place. This avoids a database restore but depends on the new schema still working with the old code, which is a per-release judgment. Check the release notes for dropped or renamed columns and altered constraints, and test the old image against the migrated database in staging.

Stop the rollout at the write instance

Because Liquibase runs only on write instances, a failed migration can be contained by holding the rollout before any write or batch instance receives the new image. Read and batch deployments stay on the old version until the job succeeds. Confirm the rollout order in your Kubernetes manifests, and make sure no automation, such as a GitOps sync, advances the other deployments while the job is still failing.

Questions teams ask before running Fineract migrations on Kubernetes

How much effort does a separate Liquibase job for Fineract take?

The effort depends on how many environments you run, how much of the Kubernetes deployment is already templated, and how well your current upgrade runbook matches a split of migrations and rollout. Most of the work is in the manifests: a job that runs with the liquibase-only Spring profile, a rollout that sets FINERACT_LIQUIBASE_ENABLED to disabled on read, write and batch instances, and an ordering gate that stops the rollout if the job fails. Testing the sequence in staging with a copy of production data takes more time than writing the manifests. Ask any partner for a written scope that lists each environment, each manifest change and each rehearsal, since those items drive the quote more than the job itself.

Can our own platform team run this migration without outside help?

A team that already operates Kubernetes workloads, manages Helm or Kustomize manifests and has run PostgreSQL schema changes can usually handle the job and the rollout ordering with its own staff. The areas that most often need an outside check are the rollback plan for a schema change that cannot be reversed by redeploying an older image, and the verification of Fineract's read, write and batch configuration across replicas. Ask yourself whether someone on the team has already rehearsed a failed migration in a non-production environment. If not, a fixed-scope review or a rehearsal engagement can cover that gap without taking over operations.

What does our team need to provide for the upgrade?

You need non-production environments that match production closely enough to test the migration, a recent and verified database backup, a change window long enough for a failed migration to be investigated, and people who can approve a rollback decision in advance. On the technical side, you need someone who can operate the Kubernetes cluster and its job scheduling, someone who can administer the PostgreSQL instance, and someone who owns the Fineract configuration for each instance type. Confirm access to the database with an account that can run schema changes, and confirm which secrets the job reads, since Kubernetes stores Secrets unencrypted in etcd unless you configure encryption at rest.

How long will the Liquibase job take in production?

Nobody can give a reliable duration before the job runs against a copy of your database. Liquibase runs its changesets in sequence on a single thread, so the run time grows with the number of pending changesets, the size of the tables they touch and the locks they take on busy tables. The gap between your current release and the target release, your table sizes and your database hardware all change the answer. Measure the job in staging with a recent production copy, record the time for each changeset that touches a large table, and use that record to schedule the production window. Ask a partner to include this rehearsal in the written scope.

Why does the migration job have to run before the new pods start?

Only write instances run Liquibase migrations, and if several application instances try to migrate at the same time, they can race on the schema and leave the database half-changed. Setting FINERACT_LIQUIBASE_ENABLED to disabled on read, write and batch instances stops that race, and a dedicated job running the liquibase-only profile becomes the single place where schema changes happen. The rollout then starts only after the job completes successfully. Engineers should configure the rollout to wait on the job's success status, not on a timer, so a failed migration blocks the new application version.

Is rollback always possible after a Fineract schema migration?

No. Redeploying the previous application image is possible only if the earlier code still works with the migrated schema, and Liquibase changesets that drop columns, change types or transform data often cannot be reversed by a simple script. Plan rollback around a verified database backup taken immediately before the job, a documented restore procedure you have rehearsed, and an agreed decision point after which you accept the forward fix. Confirm with the Fineract release notes for your target version which changesets are irreversible, and treat any restore as a planned event with its own change window.

What does a fixed-scope engagement for this work include?

A written scope for this work usually covers a review of your current deployment and Fineract instance layout, the design of the liquibase-only job and the rollout ordering, the manifest changes, rehearsals in the environments you name, a backup and rollback procedure, and a runbook for the production window. It should state what is excluded, such as application testing of products and accounting, the Mifos web app upgrade or the database platform change. DeployOpen plans, implements and documents this work inside your own infrastructure, and it does not issue audit reports or certifications for the result. Ask for the deliverables and acceptance criteria in writing before any work begins.

What makes a Fineract migration job take longer

This table lists the factors that lengthen a Liquibase migration job for Fineract on Kubernetes, how each one changes effort, and what to confirm before scheduling the upgrade. Read it as a checklist for sizing the window and the test cycle, not as a timing estimate. The main takeaway is that schema work is usually the smaller share of effort; restore testing, custom changelogs and regression checks tend to set the overall duration.

DriverEffect on effortWhat to check
Schema size and number of tenantsEach tenant database is migrated separately, so more tenants mean more migration runs and more verificationCount tenant databases, confirm which tenants are on the target release, and time a migration run against a restored copy
Custom code or custom changelogsCustom Liquibase changelogs or edits to core sources must be reconciled with the new release and can block the jobInventory custom changelogs and forked code, and confirm each one against the target release's changes
Environments and instance typesDev, test, staging and production each need a liquibase-only run, and read, write and batch instances must have the Liquibase flag disabledConfirm the instance configuration in every environment, not only production, and record which pods run Liquibase
Kubernetes version and cluster accessVersion skew rules, cluster upgrades and RBAC for creating jobs can add work before the Fineract job runsConfirm the cluster version, the supported skew for your node pools, and that the team can create and read jobs and logs
PostgreSQL backup and restore processA verified backup is a precondition for rollback, and restore rehearsals on large databases often take the longestConfirm backup type, restore time on a copy of production, and who approves a restore decision
Mifos web app version alignmentThe web app must match the backend release, so front-end testing and deployment run alongside the backend changeCheck the matching web app version, its environment variables, and whether custom UI changes need rebuilding
Regression testing of products, accounting and COBLoan and savings products, ledger postings and the Close of Business job need test cases and sign-off, which often exceed the migration itselfDefine test scope, data sets and owners for products, accounting and COB, and agree the acceptance criteria in writing

How Kubernetes version support affects the upgrade window

The cluster itself must run a supported Kubernetes minor version before the Fineract upgrade window is scheduled, because an unsupported control plane is a risk that sits underneath every application change. Upgrade the kube-apiserver first, one minor version at a time, and only then move kube-controller-manager, kube-scheduler and the kubelets. The Kubernetes version skew policy allows kubelets to be up to three minor versions older than the kube-apiserver, so nodes can lag briefly, but that window is a transition state to close, not a place to run production for months. Read the Kubernetes version skew policy before you plan node pool rotation.

Check the support table before you pick a date. As of September 2026, the Kubernetes project listed 1.37 (end of life October 28, 2027), 1.36 (June 28, 2027) and 1.35 (February 28, 2027) as the three maintained minor releases on the Kubernetes releases page. Confirm that table again on the day you schedule the change, because the project publishes new minor releases roughly every 14 weeks and the list moves with them. If your cluster is on a minor version that is close to end of life, the Liquibase job sequence still works, but you may need a cluster upgrade in the same change window, which doubles the rollback surface.

Keep this dependency at the cluster layer. The Fineract release you are moving to does not depend on a particular Kubernetes minor version, and the Liquibase job runs the same way on any supported cluster. Ask your platform team to record the current API server, kubelet and node versions, the planned target, and the date each component reaches end of life, then decide whether the cluster upgrade goes first in a separate window. Sequencing the two changes apart makes it easier to tell which one caused a failure.

Who runs the migration: internal team or outside partner

Your platform team owns the entire sequence: setting FINERACT_LIQUIBASE_ENABLED to disabled on read, write and batch deployments, creating the liquibase-only Job, gating the application rollout on its success and deciding when to roll back. This option keeps all knowledge inside your organization, but it also means your engineers must understand Fineract's instance types, the Liquibase changelog history and the single-threaded nature of schema changes before the window opens.

  • Manifest changes: your team edits the Helm values or manifests that split write, batch and read instances and removes migration from the app containers.
  • Job and gating: your team builds the Job, its image tag, its database credentials and the pipeline step that blocks the rollout until the Job exits cleanly.
  • Rollback plan: your team writes the decision criteria, database backup and restore steps, and who approves a rollback.
  • Test environment: your team maintains a staging copy of production data to rehearse the upgrade and time it.
  • Ongoing ownership: your team keeps the runbook current for each Fineract release and each Kubernetes upgrade.

Before choosing this option, confirm that the team has someone who has run Fineract upgrades in a non-production cluster and that the change window and approvers are agreed in writing.

How DeployOpen scopes and delivers a Fineract migration job

DeployOpen treats the Liquibase migration job as a controlled release step inside a self-hosted Fineract platform running in infrastructure you control. We scope the job around your current release, database platform and instance layout, then build and test it in a non-production environment before any production upgrade is planned.

How we scope

  • The written scope lists your current Fineract release, target release and database platform before any work starts.
  • We document which instances run as read, write and batch roles, and which ones currently run Liquibase on startup.
  • Engagements are fixed-scope, so deliverables, environments and acceptance tests are agreed in writing before build begins.
  • The scope names who owns the rollback decision, the maintenance window and the evidence kept for each upgrade.

How we build

  • We configure the Kubernetes manifests for the liquibase-only job and disable the migration flag on regular instances.
  • We set ordering so the migration job completes before the application rollout starts, and the rollout stops if it fails.
  • We rehearse the full upgrade on a copy of production data, including a timed run of the migration job.
  • We build a rollback plan that covers database restore, image versions and the handling of any partially applied changes.

How we hand over

  • You receive the manifests, job definitions, runbook and rollback procedure as versioned files in your own repository.
  • The runbook records each check to run after the migration job, such as schema version and instance health.
  • Your team is walked through a dry run so the upgrade can be repeated without DeployOpen present.
  • Test results and the upgrade record are delivered so your auditors and change board can review what was done.

How we support

  • Ongoing support, upgrades and continuous testing are available as a retainer after the handover.
  • A retainer can cover the next Fineract quarterly release, so each migration job is reviewed against the new Liquibase changes.
  • Support can include reviewing Kubernetes version changes that affect the job, scheduling and node placement.
  • Any support scope and its cadence are confirmed in writing, with a quote that follows from the agreed scope.

Talk to our team.

Tell us what you're working on, whether it's a deployment, an audit, a security test or a cyber range. You'll speak with an engineer who can help you scope it.

  • 30-minute call: free, with no obligation.
  • NDA on request: we can sign before you share details.
  • Clear next steps: a scope and plan after the call.