Loki 2.x to 3.0 upgrade

Your deployment mode determines how many moving parts the 3.0 upgrade touches, so confirm it before anyone estimates the work. Loki documents three modes. Monolithic runs as a single binary and is documented for up to about 20 GB of logs per day. Simple scalable mode separates read, write and backend targets and is documented as approaching 1 TB per day. Microservices mode is intended for very large clusters, where each component is deployed and scaled on its own. Each mode has its own configuration, scaling settings and order of component changes, so a plan written for one mode rarely transfers to another.

In monolithic mode, the upgrade centers on one process. You change the binary version, update the configuration for the new storage and schema settings, and restart inside a planned window. The work is concentrated and easier to reason about, but rollback is only practical if you have kept the prior configuration and tested the path back to it. Simple scalable and microservices deployments multiply that work. Every target or component needs consistent configuration, and the order of changes matters because writers and readers must agree on the new format. Confirm the exact sequence for your version pair in the official upgrade guide before you rely on any summary, including this one.

If you run simple scalable mode, treat the 3.0 upgrade as the first of two migrations. Loki's documentation states that simple scalable deployment is being deprecated and will be removed with the Loki 4.0 release. Staying on it through 3.0 means the decision you still owe is where you go next: a supported mode chosen for your volume, or microservices. Ask whether the target layout can be designed and tested alongside the 3.0 change so the second move does not repeat the first. Record the decision with an owner and a review date, and confirm the 4.0 timing in the current Loki release notes.

Before estimating, gather three facts: the deployment mode as it appears in your Helm values or process manifests, your average and peak daily ingest volume from ingestion metrics over a representative period, and the number of tenants and stream labels in use. Volume matters because it shows whether monolithic mode still fits and how much compaction and query load the upgrade window must absorb. Ask any partner to base the estimate on these numbers, to list the components in scope, and to state what is excluded, such as log shipper changes, which are separate work.

Agree the test scope and the rollback criteria before the upgrade window opens, not during it. The scope should cover five areas: ingestion from every shipper in use, query results compared before and after the change, retention behavior, tenant isolation, and Grafana dashboards and alert rules. Start with an inventory of shippers. Include every Grafana Alloy pipeline, any Fluent Bit, Fluentd or Docker logging driver source, and any OpenTelemetry client that posts logs over HTTP. Each source needs a known test line that you can search for after the upgrade, so that a missing stream is detected quickly and attributed to a specific source.

For ingestion, send a labeled marker from each shipper before and after the change and confirm that it is queryable with the same labels and a correct timestamp. For queries, choose a fixed set of representative LogQL queries, record result counts and sample lines on the 2.x cluster, and rerun the same queries on 3.0. Expect some differences where label handling or structured metadata changed, and document each one as either accepted or defective. Also watch the write path for rejected logs. Version 3.0 introduces new default limits that can reject logs, so check rejection metrics and error lines across a soak period, not only at the first request.

Retention needs a deliberate check. The compactor applies retention, and retention is disabled by default, so logs are kept indefinitely until retention_enabled is set. Confirm the value in the configuration you will actually run, then verify that expired data is removed only from the streams and tenants you intended, using a test tenant with short, known rules. If auth_enabled is true, send requests with each tenant's X-Scope-OrgID header and confirm that one tenant cannot read another tenant's streams and that requests without the header are refused. Loki has no built-in authentication, so also confirm that the authenticating reverse proxy in front of it still enforces identity on the upgraded deployment, and that Grafana data sources send the correct tenant header.

Set rollback criteria in general terms before the window opens. Typical triggers are unrecoverable loss of ingested logs, sustained rejection of logs caused by new limits that configuration changes cannot clear, and query failures on dashboards and alerts that operations depends on. Name a decision owner and a deadline for making the call. Rollback is clean only while the new schema has not been relied on. Once the cluster has written data that depends on TSDB and schema v13, or on structured metadata, reverting the binary generally does not undo that layout, so the simple reversal window closes at the first write that depends on it. Confirm with the official upgrade guide which rollback steps it supports for your storage backend and deployment mode before you rely on any of them.

If your Loki 2.x deployment runs in simple scalable mode, the Loki documentation states that this mode is being deprecated and will be removed with the Loki 4.0 release. The practical consequence is that the topology you choose for 3.0 is not a decision you can defer indefinitely. Before you schedule the upgrade, decide whether you will stay on simple scalable mode through 3.0 and move later, or change topology as part of the 3.0 work. Record that decision with its reasons so the team can revisit it when the 4.0 guidance is available.

The first path upgrades to 3.0 on the current read, write and backend layout, then moves to monolithic or microservices in a separate change. Each change is smaller and easier to roll back, but you run two maintenance windows, two validation cycles and two sets of client-facing checks. The second path combines the topology change with the 3.0 upgrade. That gives you one window, but it adds variables at the same time: the move to TSDB and schema v13, structured metadata, removed shipper settings, new default limits and a new process layout. If something misbehaves, it becomes harder to tell which change caused it. Pick the path your team can test and roll back with confidence.

The choice depends mainly on three inputs. The first is daily log volume. The Loki deployment modes documentation describes monolithic mode as suited to a single binary handling up to about 20 GB of logs per day, simple scalable mode as approaching 1 TB per day, and microservices for very large clusters. Measure your current ingest over a representative period, including peaks, rather than relying on a monthly average. The second is team operating capacity: microservices means more components to deploy, scale, monitor and upgrade. The third is query load and retention, which affect how much read capacity you need. Ask whoever will run Loki to estimate their weekly operating time for each option.

Confirm the official Loki 4.0 guidance before you fix the target topology. The documentation establishes the deprecation and the removal release, but it does not, in the material this plan relies on, describe the replacement path, the supported modes in 4.0, or the configuration changes that will be required. Check the Loki release notes on GitHub and the Grafana documentation when the guidance is published, and update your decision record. Until then, treat any second migration timeline as unknown. If you ask a partner to scope this work, request written options for each path, with the effort each one assumes for your own environment.