Why GitLab batched background migrations must finish before an upgrade
If your self-managed GitLab instance is several versions behind, every batched background migration must show Finished before you start the next upgrade. This is a hard gate, not a recommendation: the upgrade path runs through required upgrade stops, and each hop assumes the data changes from the previous hop have been fully applied. Batched background migrations rewrite or backfill rows in the background while GitLab runs, so an upgrade started while one is still queued or running can leave the database in a state the next version does not expect.
Since GitLab 17.5, the required stops fall at versions x.2, x.5, x.8 and x.11. The examples in the upgrade paths documentation are 18.2, 18.5, 18.8, 18.11 and 19.2, 19.5, 19.8 and 19.11. A typical plan therefore moves from your current version to the next stop, confirms the migrations are finished, and only then moves to the following stop. Your upgrade plan should list each stop in order and place a migration check between every pair of hops, so the sequence cannot skip one. Confirm the current stop list against the upgrade paths documentation for the exact target version before you schedule any work.
Treat the migration check as a step with an owner and a record, not a glance at a dashboard. Confirm with the team that will run the upgrade whether any migration remained pending from the prior hop, and whether the instance has been stable long enough for background work to complete. The GitLab background migrations documentation sets out the status requirement and the recovery options, and it is the reference your runbook should cite for each hop.
What the background migration gate changes for your upgrade
This table compares what changes before an upgrade when batched background migrations must show Finished, the place you check status, how failed runs are handled and what to avoid. Read each row as a go or no-go check for your upgrade window. The main takeaway is that an upgrade should not start while any batched migration is still pending or failed.
| Change | Impact | Action needed |
|---|---|---|
| Finished status required before upgrade | Upgrading while a batched migration is unfinished can leave data incomplete for the next version | Confirm every batched background migration shows Finished before you schedule the upgrade |
| Status page under Admin > Monitoring > Background migrations | This is the primary place to see the state of each batched migration in the instance | Record the status of each migration in your change ticket before the maintenance window |
| Retry option for failed migrations | A failed migration is not finished, so it blocks the upgrade until it completes | Retry failed migrations from the status page and recheck until none remain failed or pending |
| gitlab:background_migrations:finalize Rake task | Completes migrations that cannot finish through a retry, and is the documented route for finalizing them | Run the task under the GitLab documentation, and confirm the procedure with GitLab Support if a migration will not finish |
| Marking migrations finished in the Rails console | GitLab warns this can cause data loss, so it is not a safe shortcut to clear the gate | Do not mark migrations finished by hand, and escalate to GitLab Support instead |
| Backup before finalizing | Finalizing or forcing migrations changes data, so a recent backup is the recovery point if something goes wrong | Take and verify a backup before running the finalize task or any manual step |
| Status checked between required upgrade stops | Each hop on the upgrade path can add batched migrations that must finish before the next hop | Repeat the status check at every required stop, not only before the first upgrade |
How to check and clear background migrations before each hop
This sequence confirms that every batched background migration reports Finished before you start a GitLab upgrade hop, so the upgrade does not run against unfinished data changes. It also defines where to stop if a migration will not complete.
Take a verified backup before changing anything
Before you touch any migration, create a full backup of the GitLab instance using the backup task and keep it somewhere you can restore from. The administrator running the upgrade owns this step, and the output is a backup file you have confirmed exists and is readable. Record the GitLab version and the target hop next to the backup so the restore point is unambiguous. Check the GitLab backup and restore documentation for the procedure that matches your install method and the secrets files you need to keep alongside it.
Open the background migrations screen
Sign in as an administrator and go to Admin > Monitoring > Background migrations. This screen lists the batched background migrations that GitLab has queued or run, along with their status. Note the current version of the instance and the migrations that are not yet Finished. The output is a written list of every migration that is not in the Finished state, which becomes your working checklist for the rest of the sequence.
Confirm every batched migration shows finished
Check that every batched background migration on the screen shows a Finished status before you begin the upgrade hop. Migrations that are still running, paused or queued must finish on their own before you continue, so allow the instance time to process them. The check is complete when no row remains in a non-Finished state. GitLab requires this condition before each upgrade, so it applies at every required upgrade stop, not only the last one.
Retry failed migrations from the same screen
If a migration shows a failed status, use the retry action on the Background migrations screen to run it again. Retries run the same batched job against the remaining data, so a transient failure, such as a temporary database or resource problem, often clears this way. Record which migrations failed, the error shown, and the result of each retry. The step is complete when the retried migrations report Finished, or when you have a clear record that they did not and you move to the next step.
Finalize with the Rake task when retry is not enough
If a failed migration does not complete after retry, run the gitlab:background_migrations:finalize Rake task for that migration, following the background migrations documentation for the exact arguments and your install method. This task completes the remaining work in the migration and sets its status. Run it only from the documented task, and only after the backup in the first step is confirmed. The output is a migration that reports Finished. Confirm the result on the Background migrations screen before you go further.
Stop and involve GitLab Support if a migration stays unfinished
If a migration remains unfinished after retry and the finalize task, stop the upgrade and do not start the hop. Do not mark the migration Finished in the Rails console: GitLab warns that this can cause data loss, and the workaround should be used only after a backup and with GitLab Support. Record the migration name, status, error messages and the instance version, then open a support request. Your rollback point is the backup from the first step, and the instance remains on its current version until the migration is resolved.
Start the upgrade only after all migrations finish
When the Background migrations screen shows every batched migration as Finished, record that state with a screenshot or export and a note of the time and version. Only then start the upgrade hop, following your upgrade procedure for that stop. Repeat this full sequence before each subsequent required upgrade stop, since new batched migrations may be queued by each hop. The output is a dated record that the migration condition was met before the upgrade began.
Where to check migration status in the Admin area and database
Check the status of every batched background migration in the Admin area before you start the upgrade, because GitLab requires each one to show Finished before you move to the next version. In the Admin area, go to Monitoring, then Background migrations. Each migration appears with a status, and you should treat anything other than Finished as a blocker. A migration that is still running or queued means the upgrade gate is not met, and a failed migration needs action before you continue. Failed entries can be retried from the same page. Record the list and its statuses in your change ticket so the decision to proceed is traceable.
If your team scripts pre-upgrade checks, read the same state from the database instead of clicking through the interface. The status is stored in the background migration tracking tables of the GitLab database, and a query against them lets a pipeline or runbook fail fast when any migration is not finished. Table and column names change between releases, so confirm the exact names, and the values that represent each status, against the GitLab background migrations documentation for the version you are upgrading to. Test the query on a staging copy first, and keep it under version control with the rest of your upgrade runbook.
Do not use the database as a shortcut to force a migration to Finished. Changing status by hand, including from the Rails console, can cause data loss, so any such step belongs with a current backup and GitLab Support. If the Admin area shows a failure you cannot clear with a retry, raise it before scheduling the upgrade window, since the finalize procedure covered in the next section is the supported route.
What to prepare before you check background migrations
Checking migration status is only useful if you can act on what you find. The six items below set up a safe path: a tested backup, a clear view of the upgrade stops ahead, a rehearsed finalization, time to run long jobs, support access for hard decisions, and a written record of status at each hop.
Verified backup and restore test
Take a fresh backup before you touch any batched background migration, and confirm it restores. GitLab warns that finishing a migration by hand in the Rails console can cause data loss, so a backup is your only route back if a finalization goes wrong. Restore the backup to a separate host and check that projects, repositories and the database load. Record the backup time and the GitLab version it came from. Also keep the gitlab-secrets.json file with the backup, because a restore without it leaves encrypted data unreadable.
Instance version and remaining upgrade stops
Record the exact GitLab version your instance runs and list every required upgrade stop between it and your target. Since GitLab 17.5, stops fall at x.2, x.5, x.8 and x.11, for example 18.2, 18.5, 18.8, 18.11 and 19.2. Each stop may queue its own batched background migrations, so the status you check now affects which stop you can safely reach. Write the stops in order next to the migration status you expect at each one. This list tells you how many status checks the whole project involves.
Staging copy for rehearsing finalization
Build a staging instance from a recent production backup, at the same GitLab version. Use it to rehearse the retry and finalize steps before you run them on production. Time each step and note which migrations stay in progress for a long period. A staging copy also shows whether your Rake commands, paths and permissions match what your operators will use. Confirm with your own team how much data the copy needs, since a partial copy can hide the migrations that take longest.
Maintenance window for long migrations
Some batched background migrations work through large tables and can run for a long time. Agree a maintenance window with the business before you start, and plan for the migrations that are still running when it opens. Check the instance load during the window and confirm who can pause the upgrade if a job stalls. Avoid starting a new upgrade hop while a migration from the previous hop is still in progress. Ask your operations team to confirm how they will communicate the window to users.
GitLab Support access for finalization decisions
GitLab's guidance is that finishing a background migration outside normal tooling should happen with GitLab Support. Before you need help, confirm your support entitlement, your contact path and the tier your instance holds. The pricing page lists priority support for Premium, so check which support level your subscription includes before you plan around it. Share the instance version, migration names and status output in the request. Waiting for a support reply during a window is much easier if the request is ready in advance.
Migration status record before and after each hop
Capture the status of every batched background migration before each upgrade hop and again once the hop completes. Export the list from Admin > Monitoring > Background migrations, or query the database if your team prefers that method. Store the output with the backup reference and version number, so anyone reviewing the change can see that each migration reached Finished. A dated record also helps your next operator decide whether a retry has already been attempted. Keep the record with your change tickets so it can serve as evidence later.
How failed migrations are retried or finalized
Start here whenever a batched background migration shows a failed state but the failure looks transient, such as a timeout, a lock wait or a restart during the run. GitLab lets you retry these migrations from the Admin area without touching the database directly, so this is the lowest-risk path. Check the error message first, because a retry only helps if the underlying cause has been resolved.
- Find the migration: Open Admin > Monitoring > Background migrations and filter for the failed entries.
- Read the error: Note the job's error text and confirm the cause (for example, a resource that was busy) has cleared.
- Retry once: Use the retry action in the Admin area, then watch the status until it moves to Finished.
- Take a backup first: Create a current backup before any retry on a production instance, so you can restore if the migration changes data unexpectedly.
- Escalate on repeat failure: If the same migration fails again with the same error, stop and move to the finalize path rather than retrying in a loop.
Before you start the upgrade, confirm every batched migration reads Finished, not merely that the failure count has dropped.
Use this path when a migration still fails after a retry and the Admin area cannot complete it. The gitlab:background_migrations:finalize Rake task runs the remaining work for the migration in a controlled way and records it as complete. It is still a change to your production data, so it follows the same backup discipline as any other maintenance step.
- Confirm the migration name: Copy the exact migration identifier from the Admin area before running any command.
- Back up first: Take and verify a full backup on the instance, using your documented backup procedure, before running the task.
- Run in a maintenance window: Schedule the task when you can monitor the output and roll back if needed.
- Check the status afterward: Reload Admin > Monitoring > Background migrations and confirm the migration reads Finished.
- Involve GitLab Support: If the task errors, the migration stays failed, or you are unsure which migration is blocking the upgrade, open a ticket with GitLab Support before continuing.
Keep the command output and the backup identifier with your change record, since you may need both if the upgrade is reviewed later.
This tab covers the temptation to mark a stuck migration as Finished directly in the Rails console, usually because it looks like a status flag that can be flipped to unblock an upgrade. GitLab warns against this. Setting the status by hand skips the work the migration would have completed, so the data it was meant to convert can be left inconsistent, and the warning names data loss as the risk. Treat this path as a last resort that requires GitLab Support involvement, not a shortcut.
- Do not edit the status by hand: Marking a migration finished in the Rails console bypasses the migration's own work.
- Back up before anything else: If a Support engineer advises any console action, take a verified backup first.
- Involve GitLab Support before acting: Describe the migration, its error and your upgrade target, and follow their instructions.
- Prefer the Rake task when Support agrees: Support may direct you to the finalize task, which is the documented way to complete the work.
- Record the decision: Document who approved the action, the backup used and the final status, so the upgrade record is complete.
If you cannot show a verified backup and a Support decision, do not proceed with the upgrade.
What makes the migration check take longer
This table compares the drivers that stretch the background migration check before an upgrade and shows how each one changes the work. Read each row as a question to answer for your instance before you schedule the upgrade window. The main takeaway is that the number of unfinished migrations and any failures are the factors you can measure first, while backup time depends on your own data size.
| Driver | Effect on effort | What to confirm |
|---|---|---|
| Required upgrade stops still ahead | Each stop has its own set of batched migrations, so every hop adds another check and possibly another wait for the queue to drain | The full list of stops between your current version and the target, using the upgrade path documentation |
| Batched migrations not yet finished | Each unfinished migration must reach Finished, and the time needed depends on how much data each one has to rewrite | The count and names of unfinished migrations under Admin > Monitoring > Background migrations for each stop |
| Failed migrations needing the Rake task | A failure means a retry, and if retries do not succeed, the gitlab:background_migrations:finalize Rake task must be run, adding a planned maintenance step | Whether any migration shows a failed state, and whether the retry from the Admin area clears it |
| Backup time for your data size | A backup must exist before a migration is finalized by hand, so backup duration on your instance adds to the check and to the window | How long a full backup takes on a copy of your production data, and where the backup is stored |
| GitLab Support involvement | Marking a migration finished in the Rails console can cause data loss, so any manual step outside the documented path may need Support review first | Whether your support plan covers this review, the expected response path, and who on your side approves the step |
Questions buyers ask before upgrading GitLab
What does the background migration check cost, and who pays for it?
The check itself is mostly reading status and confirming that nothing is stuck, so the effort is small when migrations are already finished. The cost rises when a migration is still running or has failed, because someone must investigate, retry or finalize it, and that work is billed to whoever holds the scope. If you run it in-house, your engineers carry the time. If you engage DeployOpen, the cost follows the written scope, which should name the instances, the versions crossed and whether failed migrations are included. Ask for the quote to state what is one-time (the readiness review and any remediation) and what recurs (checks at each stop).
How long does it take to confirm every migration reached Finished?
Duration depends on how many migrations sit between your current version and the target, how far behind the instance is, and whether any of them are still running. No duration has been supplied for this check, so confirm the figure with the vendor documentation for your version path and with your own staging results. Plan for the time to wait on long-running jobs as well as the time to read status. Ask any partner to show how they will estimate duration from your instance's actual migration queue.
What does our internal team need to provide for the check?
Your team needs administrator access to the instance, the ability to read the Background migrations page under Admin, and database access if you plan to query status directly. They also need a current backup, a maintenance window agreed with the business, and someone who can decide whether a failed migration should be retried. Confirm that the people who own the backup and restore procedure are available on the day, because finalizing a migration without a backup is the path the documentation warns against.
Can an in-house team complete the background migration check alone?
Yes, for a small instance on a short upgrade path with clean status and a tested backup, an experienced GitLab administrator can read the status, retry migrations and confirm Finished on their own. The harder cases involve failed migrations, large data sets, multi-node installations and any need to run the Rake task. Assess your team against those cases, not against the happy path. If nobody on the team has finalized a migration before, treat the first attempt as a planned risk and confirm the steps with GitLab Support before touching production data.
When should we bring in outside help for this check?
Bring in outside help when the status shows failed or stuck migrations, when the instance spans several required upgrade stops, when the data is large or regulated, or when your team cannot give the check a dedicated window. A partner can also test the sequence on a copy of production before the change window. DeployOpen plans, implements, backs up, upgrades and documents self-hosted platforms in your infrastructure, so an engagement can cover the readiness review, the rehearsal and the upgrade record under one written scope.
What happens if a background migration never reaches Finished?
You should not proceed with the upgrade. GitLab requires every batched background migration to show Finished before the next upgrade, so a migration stuck in another state blocks the path. First, retry it from the Background migrations page and check the logs for the cause. If it still cannot complete, the documented route is the gitlab:background_migrations:finalize Rake task. Marking a migration finished in the Rails console can cause data loss, so do that only after a backup and with GitLab Support involved. Have the recovery plan and the support contact agreed before the window opens.
Does DeployOpen certify that our migrations are complete?
No. DeployOpen prepares systems, controls and evidence for assessments, and it does not issue audit reports, attestations or certifications. What we can provide is a documented check of migration status, the backup and rollback record, and the upgrade notes your auditors or compliance team may ask for. Confirm with your compliance function which evidence they need, and with the vendor documentation what GitLab itself records for each migration.
How DeployOpen plans the gate for your upgrade
DeployOpen starts with an inventory of the instance, because the background migration gate is only as reliable as the list of what is actually running. We record the GitLab version, the install method (Linux package, Helm chart, Operator or Docker), whether the database is bundled or external, and how many nodes serve web, API, Sidekiq and Gitaly traffic. We then check the current state of every batched background migration and note which ones are not yet Finished. The output is a dated register that names each open migration, the stop it blocks, and the owner who will confirm it closed. Ask your team to pull the same list from the Admin area before the first call so the two views can be compared.
Before any production step, we rehearse the check in a staging copy restored from a recent backup. The rehearsal covers the status view under Admin > Monitoring > Background migrations, the retry path for any failed job, and the gitlab:background_migrations:finalize Rake task in a controlled window. We time each step and record which migrations needed intervention, so the production change window is planned from observed behavior rather than assumption. Each required upgrade stop gets its own written runbook entry with the target version, the expected migration state before and after, the rollback point and the evidence to keep. Confirm with your own staging capacity and change calendar how much time the rehearsal needs.
Scope and responsibilities are agreed in writing before work starts. DeployOpen plans, documents and executes the preparation, the rehearsal and the upgrade runbook, and it does not replace GitLab Support for finalization decisions. Any migration that is stuck, or any case where marking it finished is being considered, goes to GitLab Support first, and the write-up should show a current backup and that support was involved. Where your team expects upgrades at every required stop over time, a retainer can keep the inventory, rehearsal and runbooks current between releases. Ask any partner to show how their written scope defines these handoffs.
How DeployOpen scopes, delivers and supports GitLab upgrades
DeployOpen treats batched background migration status as a gate in every GitLab upgrade plan, not a footnote. We check each migration's state on your instance, decide with your team how any failed migration is retried or finalized, and document the result before the upgrade window opens.
How we scope
- Inventory the current GitLab version and every required upgrade stop between it and the target release.
- Record the status of all batched background migrations on your instance before any upgrade step is scheduled.
- Agree in writing who owns the backup, the failed-migration decisions and any GitLab Support involvement.
How we build
- Run a pre-upgrade check of background migration status in the Admin area and, where needed, in the database.
- Retry failed migrations through the Admin interface and re-check status before moving to the next stop.
- Require a verified backup before any finalization step, and keep every status check and decision in the change record.
How we hand over
- Deliver a runbook listing each stop, the expected Finished status for each migration and the rollback point.
- Document who approves finalizing a migration and the exact conditions that must be met first.
- Walk your operations team through the status checks so they can repeat them at the next required stop.
How we support
- Engagements are fixed-scope, with scope, deliverables and timelines agreed in writing before work starts.
- Ongoing upgrade support, upgrades and continuous testing are available as retainers.
- Confirm your support entitlement with GitLab before a finalization step, since GitLab warns against unsupported manual changes.
