Prometheus LTS 3.5 to 3.13 upgrade

The main risk in this step is that changes accumulate across the minor releases between 3.5 and 3.13. Each minor release starts a new six-week cycle, so the path includes several releases, and each one can carry breaking changes and deprecated flags. A removal in one release can interact with a rename or default change in a later one, so reading only the notes for 3.13 can miss behavior that changed along the way. Work through the release notes for every intermediate minor release in order, starting with the first release after 3.5. Record each item in a single register so the review is traceable.

Deprecated items are easy to assume away because a running configuration often keeps starting cleanly. Deprecation notices tell you what will change, but the release notes are where you confirm when a flag or option is removed or its behavior changes. Build an inventory from the environment you actually run: command-line flags in service units or container arguments, every key in the main configuration file, rule file constructs, remote write and remote read settings, scrape job definitions, and any API consumers such as dashboards, automation or Alertmanager integrations. Compare each item against the full range of notes, and confirm the result against your own configuration repository rather than memory.

The 2.x to 3.x changes covered on the sibling page do not all apply to this step. Some of them were already absorbed if the estate moved through 3.x releases earlier, and others never reach a team that starts on 3.5. Do not assume either way. Ask the owners of the current estate which 3.x changes were handled during earlier moves, which were skipped with a workaround, and whether any local patches, custom exporters or hand-edited configuration bypass a change. Each answer should be marked as confirmed in the register, with the person who confirmed it and the date.

Set the scope of the review against the timeline. LTS 3.5 reached end of life on July 31, 2026, and the LTS 3.13 line, released July 1, 2026, is supported until July 31, 2027. Releases after 3.13, including v3.15.0, sit outside this target, so their notes matter only if you plan to move past the LTS line. Before cutover, check the latest patch on the 3.13 line in the project's GitHub releases, since the review should match the build you deploy. Items that need a test should be assigned to an owner and tied to the staging run described on this page.

Test the upgrade by running the same rule groups and dashboard queries against your 3.5 instance and the 3.13 test instance, then comparing what each returns. Start with recording rules. Export the output series from both instances over the same time window and compare sample values, series counts and label sets. A recording rule that produces a different label set on 3.13 will silently break every dashboard panel and alert that selects on those labels, so label-set diffs matter more than small floating-point differences. Keep the comparison scripted and repeatable so the team can rerun it after every change to the test configuration.

Alert rules need a different check because the question is behavior over time, not a single query result. Replay the same input data into both instances, or point both at the same representative targets, and record when each alert enters pending, firing and resolved states. Compare the timing, the label sets attached to each alert, and any annotations that interpolate label values. Pay attention to alerts that depend on staleness, absent series or range functions, since those are the places where a version change is most likely to shift results. Document each intentional difference so that reviewers can separate accepted changes from regressions.

Run a test scrape of representative targets, not only the simplest exporters. Include targets that use each authentication method you rely on, targets reached through service discovery, any targets that push through a gateway, and at least one target with unusual metric names or high cardinality. Confirm that the 3.13 test instance scrapes these targets successfully and that the sample counts match the 3.5 instance within the tolerance you set. Scrape failures on targets that send unusual headers or content types are often the first symptom of a compatibility issue, so review the scrape error metrics as well as the data.

Check Alertmanager routing separately. Point the test Prometheus at the Alertmanager configuration you plan to run, and confirm that each alert reaches the intended receiver, that grouping and inhibition behave as before, and that silences created in one environment match the alerts they should suppress. Prometheus should send to every Alertmanager instance, so verify the full list of endpoints in the test configuration. Ask your team to confirm which Alertmanager version and API it will run against during the upgrade, because routing tests are only meaningful against the version you intend to deploy.

Before choosing any validation tool, check the Prometheus release notes for 3.13 and the intermediate releases between 3.5 and 3.13. Confirm which configuration fields, query behaviors and API changes apply to your rules and dashboards, and whether any linting or rule-checking tool you consider supports those changes. Record the results of each check in a shared document with the owner, the date and the accepted differences, so the sign-off for the production upgrade rests on evidence that someone other than the author can review.

A rollback can return the server to the previous binary and configuration, but it cannot restore data that was never captured or that has already expired. Before the upgrade starts, the team should decide which path it will rely on: a supported downgrade from the new release, or a restore of the pre-upgrade data directory and configuration onto the old release. These paths behave differently, and the first depends on facts this page does not supply. Confirm in the release notes for every release in the path whether downgrade is supported, and whether the newer version changes how it writes time series database files. If the release notes do not state it, treat a downgrade as unsupported and plan around a restore.

Configuration is the easiest part to restore, provided it lives in version control. The main Prometheus configuration file, the rule files it loads, the target definitions and any file-based service discovery inputs should be tagged at the moment of the upgrade, so the exact files that ran on the previous version can be redeployed. Alertmanager configuration and any dashboard or recording rule definitions kept in Grafana or elsewhere need their own tags, because they sit outside the Prometheus data directory. If the team plans to apply the old configuration by reload instead of restart, note that the reload endpoint is only enabled when the --web.enable-lifecycle flag is set, and that flag is disabled by default. The rollback runbook should say exactly how the old configuration will be applied.

The data directory snapshot is the only local copy of stored samples, because local storage is not clustered or replicated. Take it with the Prometheus process stopped, or from a volume or filesystem snapshot taken at a consistent point. Verify it before the upgrade by starting the old binary against a copy in a test environment. A snapshot restores the state at the moment it was taken. Samples collected after that point are lost from the local copy unless they were also sent through remote write to external storage. If the newer version has already written blocks, the old version may fail to read them, so the snapshot taken before the first start of the new version is the one that matters.

Retention sets how far back any restore can reach. The default retention window is 15 days, and it can be configured by time or by size. Samples older than the window are removed as the server runs, so restoring an older snapshot does not bring back history that was already removed before the snapshot was taken. Once the restored server runs again, retention applies to the restored data as well. Size-based limits can remove data sooner than the time limit suggests when ingest is heavy. Ask the team to confirm the retention settings on the current server, then decide how long a rollback window must last. If that window is longer than the retention period, the team must either accept a gap in history or extend retention and storage capacity before the upgrade.

Turn these points into a written rollback decision before the change window opens. Name the point of no return, which is the first start of the new version that writes data, and the criteria for choosing a rollback over a forward fix. Test the restore on a copy of the data directory with the old binary, and record how long the stop, restore and reload steps take during that test. Include Alertmanager, the rule files and any remote write target in the test. Record who can authorize the rollback, and confirm that the saved configuration and snapshot are stored on a system separate from the Prometheus host so they survive a host failure.

An organization on a Prometheus long-term support line should plan one upgrade per support window, and the window is fixed by dates the project publishes. LTS 3.13 was released July 1, 2026 and is supported until July 31, 2027, while LTS 3.5 reached end of life on July 31, 2026. Minor releases work on a much shorter clock: a new minor release cycle begins every six weeks, and a minor release generally stops receiving bug fixes after those six weeks. Teams that treat minor releases as the planning unit end up upgrading far more often than the LTS model requires, and teams that ignore the end-of-life date end up running unsupported code. The planning unit for this rhythm is the LTS line.

Build the calendar backward from the end-of-support date, not forward from whenever the team last looked at the release page. Start with the date the line stops receiving fixes, then place three milestones before it: a staging test run of the upgrade against a copy of current configuration, rules and dashboards, a production change window, and a rollback rehearsal. Set an internal deadline earlier than the vendor date so that a failed test run can be repeated before anything is lost. Confirm each date on the Prometheus release cycle page and in the release notes, because the project publishes them and a planning document should cite them rather than memory. Review the calendar at a fixed interval, such as each quarter, and record the decision in the change log.

Assign one named owner to watch the Prometheus GitHub releases and the release cycle page, and name a backup who receives the same notifications. The owner is responsible for three things: noticing when a new LTS line is announced, recording patch releases on the current LTS line (such as the 3.13 patch series, since LTS patch releases carry bug fixes), and triggering a calendar review when either happens. A team alias on a mailing list is not enough on its own, because alerts tend to be read by no one in particular. The owner should also confirm which version the next LTS line will be before the current line nears its end date, and note whether the chosen path has any intermediate stops.

Schedule the test runs well ahead of end-of-life dates, and make the test environment match production closely enough to mean something. Ask who maintains that environment, how it is refreshed from production configuration, and who signs off the result. Check that the test run covers the parts most likely to change behavior during an upgrade: scrape targets, Alertmanager connections, remote write destinations, recording and alerting rules, and the dashboards that depend on them. Plan for change freezes, staff leave and team turnover, since those are the usual reasons a yearly calendar slips. If the owner changes, transfer the calendar, the subscriptions and the open test runs in the same handover.